Eliezer Yudkowsky blog posts, 2006-2010

- The Martial Art of Rationality, 22 November 2006 08:00PM

- Why truth? And..., 27 November 2006 01:49AM

- ...What's a bias, again?, 27 November 2006 01:50AM

- The Proper Use of Humility, 01 December 2006 07:55PM

- The Modesty Argument, 10 December 2006 09:42PM

- "I don't know.", 21 December 2006 06:27PM

- A Fable of Science and Politics, 23 December 2006 04:50AM

- Some Claims Are Just Too Extraordinary, 20 January 2007 07:14AM

- Outside the Laboratory, 21 January 2007 03:46AM

- Politics is the Mind-Killer, 18 February 2007 09:23PM

- Just Lose Hope Already, 25 February 2007 12:39AM

- You Are Not Hiring the Top 1%, 02 March 2007 05:34AM

- Policy Debates Should Not Appear One-Sided, 03 March 2007 06:53PM

- Burch's Law, 08 March 2007 02:39AM

- The Scales of Justice, the Notebook of Rationality, 13 March 2007 04:00PM

- Blue or Green on Regulation?, 15 March 2007 06:04PM

- Superstimuli and the Collapse of Western Civilization, 16 March 2007 06:10PM

- Useless Medical Disclaimers, 19 March 2007 04:48PM

- Archimedes's Chronophone, 23 March 2007 05:43PM

- Chronophone Motivations, 24 March 2007 05:23PM

- Self-deception: Hypocrisy or Akrasia?, 26 March 2007 05:03PM

- Tsuyoku Naritai! (I Want To Become Stronger), 27 March 2007 05:49PM

- Tsuyoku vs. the Egalitarian Instinct, 28 March 2007 05:49PM

- "Statistical Bias", 30 March 2007 06:55PM

- Useful Statistical Biases, 01 April 2007 04:51AM

- The Error of Crowds, 01 April 2007 09:50PM

- The Majority Is Always Wrong, 03 April 2007 01:12AM

- Knowing About Biases Can Hurt People, 04 April 2007 06:01PM

- Debiasing as Non-Self-Destruction, 07 April 2007 08:20PM

- "Inductive Bias", 08 April 2007 07:52PM

- Suggested Posts, 09 April 2007 02:32AM

- Futuristic Predictions as Consumable Goods, 10 April 2007 12:18AM

- Marginally Zero-Sum Efforts, 11 April 2007 05:22AM

- Priors as Mathematical Objects, 12 April 2007 03:24AM

- Lotteries: A Waste of Hope, 13 April 2007 05:36AM

- New Improved Lottery, 13 April 2007 11:42PM

- Your Rationality is My Business, 15 April 2007 07:31AM

- Consolidated Nature of Morality Thread, 15 April 2007 11:00PM

- Feeling Rational, 26 April 2007 04:48AM

- Universal Fire, 27 April 2007 09:15PM

- Universal Law, 29 April 2007 06:41AM

- Think Like Reality, 02 May 2007 06:36AM

- Beware the Unsurprised, 03 May 2007 10:45PM

- The Third Alternative, 06 May 2007 11:47PM

- Third Alternatives for Afterlife-ism, 08 May 2007 07:41AM

- Scope Insensitivity, 14 May 2007 02:53AM

- One Life Against the World, 18 May 2007 10:06PM

- Risk-Free Bonds Aren't, 22 June 2007 10:30PM

- Correspondence Bias, 25 June 2007 12:58AM

- Are Your Enemies Innately Evil?, 26 June 2007 09:13PM

- Open Thread, 01 July 2007 07:38PM

- Two More Things to Unlearn from School, 12 July 2007 05:45PM

- Making Beliefs Pay Rent (in Anticipated Experiences), 28 July 2007 10:59PM

- Belief in Belief, 29 July 2007 05:49PM

- Bayesian Judo, 31 July 2007 05:53AM

- Professing and Cheering, 02 August 2007 07:20AM

- Belief as Attire, 02 August 2007 05:13PM

- Religion's Claim to be Non-Disprovable, 04 August 2007 03:21AM

- The Importance of Saying "Oops", 05 August 2007 03:17AM

- Focus Your Uncertainty, 05 August 2007 08:49PM

- The Proper Use of Doubt, 06 August 2007 08:29PM

- The Virtue of Narrowness, 07 August 2007 05:57PM

- You Can Face Reality, 09 August 2007 01:46AM

- The Apocalypse Bet, 09 August 2007 05:23PM

- Your Strength as a Rationalist, 11 August 2007 12:21AM

- I Defy the Data!, 11 August 2007 09:33PM

- Absence of Evidence Is Evidence of Absence, 12 August 2007 08:34PM

- Conservation of Expected Evidence, 13 August 2007 03:55PM

- Update Yourself Incrementally, 14 August 2007 02:56PM

- One Argument Against An Army, 15 August 2007 06:39PM

- Hindsight bias, 16 August 2007 09:58PM

- Hindsight Devalues Science, 17 August 2007 07:39PM

- Scientific Evidence, Legal Evidence, Rational Evidence, 19 August 2007 05:36AM

- Is Molecular Nanotechnology "Scientific"?, 20 August 2007 04:11AM

- Fake Explanations, 20 August 2007 09:13PM

- Guessing the Teacher's Password, 22 August 2007 03:40AM

- Science as Attire, 23 August 2007 05:10AM

- Fake Causality, 23 August 2007 06:12PM

- Semantic Stopsigns, 24 August 2007 07:29PM

- Mysterious Answers to Mysterious Questions, 25 August 2007 10:27PM

- The Futility of Emergence, 26 August 2007 10:10PM

- Positive Bias: Look Into the Dark, 28 August 2007 03:55AM

- Say Not "Complexity", 29 August 2007 04:22AM

- My Wild and Reckless Youth, 30 August 2007 01:52AM

- Failing to Learn from History, 30 August 2007 08:22PM

- Making History Available, 31 August 2007 07:52PM

- Stranger Than History, 01 September 2007 06:57PM

- Explain/Worship/Ignore?, 02 September 2007 08:01PM

- "Science" as Curiosity-Stopper, 03 September 2007 08:04PM

- Absurdity Heuristic, Absurdity Bias, 05 September 2007 03:20AM

- Availability, 06 September 2007 06:55AM

- Why is the Future So Absurd?, 07 September 2007 08:42AM

- Anchoring and Adjustment, 07 September 2007 09:33PM

- The Crackpot Offer, 08 September 2007 02:32PM

- Radical Honesty, 10 September 2007 06:09AM

- We Don't Really Want Your Participation, 10 September 2007 07:53PM

- Applause Lights, 11 September 2007 06:31PM

- Rationality and the English Language, 12 September 2007 10:55PM

- Human Evil and Muddled Thinking, 13 September 2007 11:43PM

- Doublethink (Choosing to be Biased), 14 September 2007 08:05PM

- Why I'm Blooking, 15 September 2007 05:49PM

- Planning Fallacy, 17 September 2007 07:06AM

- Kahneman's Planning Anecdote, 17 September 2007 04:39PM

- Conjunction Fallacy, 19 September 2007 01:54AM

- Conjunction Controversy (Or, How They Nail It Down), 20 September 2007 02:41AM

- Burdensome Details, 20 September 2007 11:46PM

- What is Evidence?, 22 September 2007 06:43AM

- The Lens That Sees Its Flaws, 23 September 2007 12:10AM

- How Much Evidence Does It Take?, 24 September 2007 04:06AM

- Einstein's Arrogance, 25 September 2007 01:29AM

- Occam's Razor, 26 September 2007 06:36AM

- 9/26 is Petrov Day, 26 September 2007 04:14PM

- How to Convince Me That 2 + 2 = 3, 27 September 2007 11:00PM

- The Bottom Line, 28 September 2007 05:47PM

- What Evidence Filtered Evidence?, 29 September 2007 11:10PM

- Rationalization, 30 September 2007 07:29PM

- Recommended Rationalist Reading, 01 October 2007 06:36PM

- A Rational Argument, 02 October 2007 06:35PM

- We Change Our Minds Less Often Than We Think, 03 October 2007 06:14PM

- Avoiding Your Belief's Real Weak Points, 05 October 2007 01:59AM

- The Meditation on Curiosity, 06 October 2007 12:26AM

- Singlethink, 06 October 2007 07:24PM

- No One Can Exempt You From Rationality's Laws, 07 October 2007 05:24PM

- A Priori, 08 October 2007 09:02PM

- Priming and Contamination, 10 October 2007 02:23AM

- Do We Believe Everything We're Told?, 10 October 2007 11:52PM

- Cached Thoughts, 11 October 2007 11:46PM

- The "Outside the Box" Box, 12 October 2007 10:50PM

- Original Seeing, 14 October 2007 04:38AM

- How to Seem (and Be) Deep, 14 October 2007 06:13PM

- The Logical Fallacy of Generalization from Fictional Evidence, 16 October 2007 03:57AM

- Hold Off On Proposing Solutions, 17 October 2007 03:16AM

- "Can't Say No" Spending, 18 October 2007 02:08AM

- Congratulations to Paris Hilton, 19 October 2007 12:31AM

- Pascal's Mugging: Tiny Probabilities of Vast Utilities, 19 October 2007 11:37PM

- Illusion of Transparency: Why No One Understands You, 20 October 2007 11:49PM

- Self-Anchoring, 22 October 2007 06:11AM

- Expecting Short Inferential Distances, 22 October 2007 11:42PM

- Explainers Shoot High. Aim Low!, 24 October 2007 01:13AM

- Double Illusion of Transparency, 24 October 2007 11:06PM

- No One Knows What Science Doesn't Know, 25 October 2007 11:47PM

- Why Are Individual IQ Differences OK?, 26 October 2007 09:50PM

- Bay Area Bayesians Unite!, 28 October 2007 12:07AM

- Motivated Stopping and Motivated Continuation, 28 October 2007 11:10PM

- Torture vs. Dust Specks, 30 October 2007 02:50AM

- A Case Study of Motivated Continuation, 31 October 2007 01:27AM

- A Terrifying Halloween Costume, 01 November 2007 02:54AM

- Fake Justification, 01 November 2007 03:57AM

- An Alien God, 02 November 2007 06:57AM

- The Wonder of Evolution, 02 November 2007 08:49PM

- Evolutions Are Stupid (But Work Anyway), 03 November 2007 03:45PM

- Natural Selection's Speed Limit and Complexity Bound, 04 November 2007 04:54PM

- Beware of Stephen J. Gould, 06 November 2007 05:22AM

- The Tragedy of Group Selectionism, 07 November 2007 07:47AM

- Fake Selfishness, 08 November 2007 02:31AM

- Fake Morality, 08 November 2007 09:32PM

- Fake Optimization Criteria, 10 November 2007 12:10AM

- Adaptation-Executers, not Fitness-Maximizers, 11 November 2007 06:39AM

- Evolutionary Psychology, 11 November 2007 08:41PM

- Protein Reinforcement and DNA Consequentialism, 13 November 2007 01:34AM

- Thou Art Godshatter, 13 November 2007 07:38PM

- Terminal Values and Instrumental Values, 15 November 2007 07:56AM

- Evolving to Extinction, 16 November 2007 07:18AM

- No Evolutions for Corporations or Nanodevices, 17 November 2007 02:24AM

- The Simple Math of Everything, 17 November 2007 10:42PM

- Conjuring An Evolution To Serve You, 19 November 2007 05:55AM

- Artificial Addition, 20 November 2007 07:58AM

- Truly Part Of You, 21 November 2007 02:18AM

- Not for the Sake of Happiness (Alone), 22 November 2007 03:19AM

- Leaky Generalizations, 22 November 2007 09:16PM

- The Hidden Complexity of Wishes, 24 November 2007 12:12AM

- Lost Purposes, 25 November 2007 09:01AM

- Purpose and Pragmatism, 26 November 2007 06:51AM

- The Affect Heuristic, 27 November 2007 07:58AM

- Evaluability (And Cheap Holiday Shopping), 28 November 2007 12:37AM

- Unbounded Scales, Huge Jury Awards, & Futurism, 29 November 2007 07:45AM

- The Halo Effect, 30 November 2007 12:58AM

- Superhero Bias, 01 December 2007 03:14AM

- Mere Messiahs, 02 December 2007 12:49AM

- Affective Death Spirals, 02 December 2007 04:44PM

- Resist the Happy Death Spiral, 04 December 2007 01:15AM

- Uncritical Supercriticality, 04 December 2007 04:40PM

- Fake Fake Utility Functions, 06 December 2007 06:30AM

- Fake Utility Functions, 06 December 2007 04:55PM

- Evaporative Cooling of Group Beliefs, 07 December 2007 11:08PM

- When None Dare Urge Restraint, 08 December 2007 11:09PM

- The Robbers Cave Experiment, 10 December 2007 06:18AM

- Misc Meta, 10 December 2007 10:57PM

- Every Cause Wants To Be A Cult, 12 December 2007 03:04AM

- Reversed Stupidity Is Not Intelligence, 12 December 2007 10:14PM

- Argument Screens Off Authority, 14 December 2007 12:05AM

- Hug the Query, 14 December 2007 07:51PM

- Guardians of the Truth, 15 December 2007 06:44PM

- Guardians of the Gene Pool, 16 December 2007 08:08PM

- Guardians of Ayn Rand, 18 December 2007 06:24AM

- The Litany Against Gurus, 18 December 2007 08:11PM

- Politics and Awful Art, 20 December 2007 03:46AM

- Two Cult Koans, 21 December 2007 05:45AM

- False Laughter, 22 December 2007 06:03AM

- Effortless Technique, 23 December 2007 04:22AM

- Zen and the Art of Rationality, 24 December 2007 04:36AM

- The Amazing Virgin Pregnancy, 24 December 2007 02:00PM

- Asch's Conformity Experiment, 26 December 2007 07:03AM

- On Expressing Your Concerns, 27 December 2007 04:04AM

- Lonely Dissent, 28 December 2007 04:23AM

- To Lead, You Must Stand Up, 29 December 2007 06:38AM

- Cultish Countercultishness, 30 December 2007 12:53AM

- My Strange Beliefs, 30 December 2007 12:15PM

- Posting on Politics, 01 January 2008 07:25AM

- The Two-Party Swindle, 01 January 2008 08:38AM

- The American System and Misleading Labels, 02 January 2008 07:38AM

- Stop Voting For Nincompoops, 02 January 2008 06:00PM

- Rational vs. Scientific Ev-Psych, 04 January 2008 07:01AM

- A Failed Just-So Story, 05 January 2008 06:35AM

- But There's Still A Chance, Right?, 06 January 2008 01:56AM

- The Fallacy of Gray, 07 January 2008 06:24AM

- Absolute Authority, 08 January 2008 03:33AM

- Infinite Certainty, 09 January 2008 06:49AM

- 0 And 1 Are Not Probabilities, 10 January 2008 06:58AM

- Beautiful Math, 10 January 2008 10:43PM

- Expecting Beauty, 12 January 2008 03:00AM

- Is Reality Ugly?, 12 January 2008 10:26PM

- Beautiful Probability, 14 January 2008 07:19AM

- Trust in Math, 15 January 2008 04:25AM

- Rationality Quotes 1, 16 January 2008 07:41AM

- Rationality Quotes 2, 16 January 2008 11:47PM

- Rationality Quotes 3, 18 January 2008 02:20AM

- The Allais Paradox, 19 January 2008 03:05AM

- Zut Allais!, 20 January 2008 03:18AM

- Rationality Quotes 4, 20 January 2008 05:00PM

- Allais Malaise, 21 January 2008 12:40AM

- Against Discount Rates, 21 January 2008 10:00AM

- Circular Altruism, 22 January 2008 06:00PM

- Rationality Quotes 5, 23 January 2008 06:00PM

- Rationality Quotes 6, 24 January 2008 06:00PM

- Rationality Quotes 7, 25 January 2008 06:00PM

- Rationality Quotes 8, 26 January 2008 06:00PM

- Rationality Quotes 9, 27 January 2008 06:00PM

- The "Intuitions" Behind "Utilitarianism", 28 January 2008 04:29PM

- Trust in Bayes, 29 January 2008 11:12PM

- Something to Protect, 30 January 2008 05:52PM

- Newcomb's Problem and Regret of Rationality, 31 January 2008 07:36PM

- OB Meetup: Millbrae, Thu 21 Feb, 7pm, 31 January 2008 11:18PM

- The Parable of the Dagger, 01 February 2008 08:53PM

- The Parable of Hemlock, 03 February 2008 02:01AM

- Words as Hidden Inferences, 03 February 2008 11:36PM

- Extensions and Intensions, 04 February 2008 09:34PM

- Buy Now Or Forever Hold Your Peace, 04 February 2008 09:42PM

- Similarity Clusters, 06 February 2008 03:34AM

- Typicality and Asymmetrical Similarity, 06 February 2008 09:20PM

- The Cluster Structure of Thingspace, 08 February 2008 12:07AM

- Disguised Queries, 09 February 2008 12:05AM

- Neural Categories, 10 February 2008 12:33AM

- How An Algorithm Feels From Inside, 11 February 2008 02:35AM

- Disputing Definitions, 12 February 2008 12:15AM

- Feel the Meaning, 13 February 2008 01:01AM

- The Argument from Common Usage, 13 February 2008 04:24PM

- Empty Labels, 14 February 2008 11:50PM

- Classic Sichuan in Millbrae, Thu Feb 21, 7pm, 15 February 2008 12:21AM

- Taboo Your Words, 15 February 2008 10:53PM

- Replace the Symbol with the Substance, 16 February 2008 06:12PM

- Fallacies of Compression, 17 February 2008 06:51PM

- Categorizing Has Consequences, 19 February 2008 01:40AM

- Sneaking in Connotations, 19 February 2008 07:41PM

- Arguing "By Definition", 20 February 2008 11:37PM

- Where to Draw the Boundary?, 21 February 2008 07:14PM

- Entropy, and Short Codes, 23 February 2008 03:16AM

- Mutual Information, and Density in Thingspace, 23 February 2008 07:14PM

- Superexponential Conceptspace, and Simple Words, 24 February 2008 11:59PM

- Leave a Line of Retreat, 25 February 2008 11:57PM

- The Second Law of Thermodynamics, and Engines of Cognition, 27 February 2008 12:48AM

- Perpetual Motion Beliefs, 27 February 2008 08:22PM

- Searching for Bayes-Structure, 28 February 2008 10:01PM

- Conditional Independence, and Naive Bayes, 01 March 2008 01:59AM

- Words as Mental Paintbrush Handles, 01 March 2008 11:58PM

- Rationality Quotes 10, 03 March 2008 05:48AM

- Rationality Quotes 11, 04 March 2008 05:48AM

- Variable Question Fallacies, 05 March 2008 06:22AM

- 37 Ways That Words Can Be Wrong, 06 March 2008 05:09AM

- Gary Gygax Annihilated at 69, 06 March 2008 08:54PM

- Dissolving the Question, 08 March 2008 03:17AM

- Wrong Questions, 08 March 2008 05:11PM

- Righting a Wrong Question, 09 March 2008 01:00PM

- Mind Projection Fallacy, 11 March 2008 12:29AM

- Probability is in the Mind, 12 March 2008 04:08AM

- The Quotation is not the Referent, 13 March 2008 12:53AM

- Penguicon & Blook, 13 March 2008 05:32PM

- Qualitatively Confused, 14 March 2008 05:01PM

- Reductionism, 16 March 2008 06:26AM

- Explaining vs. Explaining Away, 17 March 2008 01:59AM

- Fake Reductionism, 17 March 2008 10:49PM

- Savanna Poets, 18 March 2008 06:42PM

- Joy in the Merely Real, 20 March 2008 06:18AM

- Joy in Discovery, 21 March 2008 01:19AM

- Bind Yourself to Reality, 22 March 2008 05:09AM

- If You Demand Magic, Magic Won't Help, 22 March 2008 06:10PM

- New York OB Meetup (ad-hoc) on Monday, Mar 24, @6pm, 22 March 2008 09:07PM

- The Beauty of Settled Science, 24 March 2008 05:30AM

- Amazing Breakthrough Day: April 1st, 25 March 2008 05:45AM

- Is Humanism A Religion-Substitute?, 26 March 2008 04:18AM

- Scarcity, 27 March 2008 08:07AM

- To Spread Science, Keep It Secret, 28 March 2008 05:47AM

- Initiation Ceremony, 28 March 2008 08:40PM

- Hand vs. Fingers, 30 March 2008 12:36AM

- Angry Atoms, 31 March 2008 12:28AM

- Heat vs. Motion, 01 April 2008 03:55AM

- Brain Breakthrough! It's Made of Neurons!, 01 April 2008 07:00PM

- Reductive Reference, 03 April 2008 01:37AM

- Zombies! Zombies?, 04 April 2008 09:55AM

- Zombie Responses, 05 April 2008 12:42AM

- The Generalized Anti-Zombie Principle, 05 April 2008 11:16PM

- GAZP vs. GLUT, 07 April 2008 01:51AM

- Belief in the Implied Invisible, 08 April 2008 07:40AM

- Quantum Explanations, 09 April 2008 08:15AM

- Configurations and Amplitude, 10 April 2008 07:41AM

- Joint Configurations, 11 April 2008 05:00AM

- Distinct Configurations, 12 April 2008 04:42AM

- Where Philosophy Meets Science, 12 April 2008 09:21PM

- Can You Prove Two Particles Are Identical?, 14 April 2008 07:06AM

- Classical Configuration Spaces, 15 April 2008 08:40AM

- The Quantum Arena, 15 April 2008 07:00PM

- Feynman Paths, 17 April 2008 06:32AM

- No Individual Particles, 18 April 2008 04:40AM

- Identity Isn't In Specific Atoms, 19 April 2008 04:55AM

- Zombies: The Movie, 20 April 2008 05:53AM

- Three Dialogues on Identity, 21 April 2008 06:13AM

- Decoherence, 22 April 2008 06:41AM

- The So-Called Heisenberg Uncertainty Principle, 23 April 2008 06:36AM

- Which Basis Is More Fundamental?, 24 April 2008 04:17AM

- Where Physics Meets Experience, 25 April 2008 04:58AM

- Where Experience Confuses Physicists, 26 April 2008 05:05AM

- On Being Decoherent, 27 April 2008 04:59AM

- The Conscious Sorites Paradox, 28 April 2008 02:58AM

- Decoherence is Pointless, 29 April 2008 06:38AM

- Decoherent Essences, 30 April 2008 06:32AM

- The Born Probabilities, 01 May 2008 05:50AM

- Decoherence as Projection, 02 May 2008 06:32AM

- Entangled Photons, 03 May 2008 07:20AM

- Bell's Theorem: No EPR "Reality", 04 May 2008 04:44AM

- Spooky Action at a Distance: The No-Communication Theorem, 05 May 2008 02:43AM

- Decoherence is Simple, 06 May 2008 07:44AM

- Decoherence is Falsifiable and Testable, 07 May 2008 07:54AM

- Quantum Non-Realism, 08 May 2008 05:27AM

- Collapse Postulates, 09 May 2008 07:49AM

- If Many-Worlds Had Come First, 10 May 2008 07:43AM

- Many Worlds, One Best Guess, 11 May 2008 08:32AM

- The Failures of Eld Science, 12 May 2008 10:32AM

- The Dilemma: Science or Bayes?, 13 May 2008 08:16AM

- Science Doesn't Trust Your Rationality, 14 May 2008 02:13AM

- When Science Can't Help, 15 May 2008 07:24AM

- Science Isn't Strict Enough, 16 May 2008 06:51AM

- Do Scientists Already Know This Stuff?, 17 May 2008 02:25AM

- No Safe Defense, Not Even Science, 18 May 2008 05:19AM

- Changing the Definition of Science, 18 May 2008 06:07PM

- Conference on Global Catastrophic Risks, 19 May 2008 10:13PM

- Faster Than Science, 20 May 2008 12:19AM

- Einstein's Speed, 21 May 2008 02:48AM

- That Alien Message, 22 May 2008 05:55AM

- My Childhood Role Model, 23 May 2008 08:51AM

- Mach's Principle: Anti-Epiphenomenal Physics, 24 May 2008 05:01AM

- A Broken Koan, 24 May 2008 07:04PM

- Relative Configuration Space, 26 May 2008 09:25AM

- Timeless Physics, 27 May 2008 09:09AM

- Timeless Beauty, 28 May 2008 04:32AM

- Timeless Causality, 29 May 2008 06:45AM

- Einstein's Superpowers, 30 May 2008 06:40AM

- Class Project, 31 May 2008 12:23AM

- A Premature Word on AI, 31 May 2008 05:48PM

- The Rhythm of Disagreement, 01 June 2008 08:18PM

- Principles of Disagreement, 02 June 2008 07:04AM

- Timeless Identity, 03 June 2008 08:16AM

- Why Quantum?, 04 June 2008 05:34AM

- Living in Many Worlds, 05 June 2008 02:24AM

- Thou Art Physics, 06 June 2008 06:37AM

- Timeless Control, 07 June 2008 05:16AM

- Bloggingheads: Yudkowsky and Horgan, 07 June 2008 10:09PM

- Against Devil's Advocacy, 09 June 2008 04:15AM

- Eliezer's Post Dependencies; Book Notification; Graphic Designer Wanted, 10 June 2008 01:18AM

- The Quantum Physics Sequence, 11 June 2008 03:42AM

- An Intuitive Explanation of Quantum Mechanics, 12 June 2008 03:45AM

- Quantum Physics Revealed As Non-Mysterious, 12 June 2008 05:20AM

- And the Winner is... Many-Worlds!, 12 June 2008 06:05AM

- Quantum Mechanics and Personal Identity, 12 June 2008 07:13AM

- Causality and Moral Responsibility, 13 June 2008 08:34AM

- Possibility and Could-ness, 14 June 2008 04:38AM

- The Ultimate Source, 15 June 2008 09:01AM

- Passing the Recursive Buck, 16 June 2008 04:50AM

- Grasping Slippery Things, 17 June 2008 02:04AM

- Ghosts in the Machine, 17 June 2008 11:29PM

- LA-602 vs. RHIC Review, 19 June 2008 10:00AM

- Heading Toward Morality, 20 June 2008 08:08AM

- The Outside View's Domain, 21 June 2008 03:17AM

- Surface Analogies and Deep Causes, 22 June 2008 07:51AM

- Optimization and the Singularity, 23 June 2008 05:55AM

- The Psychological Unity of Humankind, 24 June 2008 07:12AM

- The Design Space of Minds-In-General, 25 June 2008 06:37AM

- No Universally Compelling Arguments, 26 June 2008 08:29AM

- 2-Place and 1-Place Words, 27 June 2008 07:39AM

- The Opposite Sex, 28 June 2008 11:28AM

- What Would You Do Without Morality?, 29 June 2008 05:07AM

- The Moral Void, 30 June 2008 08:52AM

- Created Already In Motion, 01 July 2008 06:03AM

- I'd take it, 02 July 2008 07:57AM

- The Bedrock of Fairness, 03 July 2008 06:00AM

- 2 of 10, not 3 total, 04 July 2008 01:10AM

- Moral Complexities, 04 July 2008 06:43AM

- Is Morality Preference?, 05 July 2008 12:55AM

- Is Morality Given?, 06 July 2008 08:12AM

- Will As Thou Wilt, 07 July 2008 10:37AM

- Where Recursive Justification Hits Bottom, 08 July 2008 10:16AM

- The Fear of Common Knowledge, 09 July 2008 09:48AM

- My Kind of Reflection, 10 July 2008 07:21AM

- The Genetic Fallacy, 11 July 2008 05:47AM

- Fundamental Doubts, 12 July 2008 05:21AM

- Rebelling Within Nature, 13 July 2008 12:32PM

- Probability is Subjectively Objective, 14 July 2008 09:16AM

- Lawrence Watt-Evans's Fiction, 15 July 2008 03:00AM

- Posting May Slow, 16 July 2008 03:00AM

- Whither Moral Progress?, 16 July 2008 05:04AM

- The Gift We Give To Tomorrow, 17 July 2008 06:07AM

- Could Anything Be Right?, 18 July 2008 07:19AM

- Existential Angst Factory, 19 July 2008 06:55AM

- Touching the Old, 20 July 2008 09:19AM

- Should We Ban Physics?, 21 July 2008 08:12AM

- Fake Norms, or "Truth" vs. Truth, 22 July 2008 10:23AM

- When (Not) To Use Probabilities, 23 July 2008 10:58AM

- Can Counterfactuals Be True?, 24 July 2008 04:40AM

- Math is Subjunctively Objective, 25 July 2008 11:06AM

- Does Your Morality Care What You Think?, 26 July 2008 12:25AM

- Changing Your Metaethics, 27 July 2008 12:36PM

- Setting Up Metaethics, 28 July 2008 02:25AM

- The Meaning of Right, 29 July 2008 01:28AM

- Interpersonal Morality, 29 July 2008 06:01PM

- Humans in Funny Suits, 30 July 2008 11:54PM

- Detached Lever Fallacy, 31 July 2008 06:57PM

- A Genius for Destruction, 01 August 2008 07:25PM

- The Comedy of Behaviorism, 02 August 2008 08:42PM

- No Logical Positivist I, 04 August 2008 01:06AM

- Anthropomorphic Optimism, 04 August 2008 08:17PM

- Contaminated by Optimism, 06 August 2008 12:26AM

- Hiroshima Day, 06 August 2008 11:15PM

- Morality as Fixed Computation, 08 August 2008 01:00AM

- Inseparably Right; or, Joy in the Merely Good, 09 August 2008 01:00AM

- Sorting Pebbles Into Correct Heaps, 10 August 2008 01:00AM

- Moral Error and Moral Disagreement, 10 August 2008 11:32PM

- Abstracted Idealized Dynamics, 12 August 2008 01:00AM

- "Arbitrary", 12 August 2008 05:55PM

- Is Fairness Arbitrary?, 14 August 2008 01:54AM

- The Bedrock of Morality: Arbitrary?, 14 August 2008 10:00PM

- Hot Air Doesn't Disagree, 16 August 2008 12:42AM

- When Anthropomorphism Became Stupid, 16 August 2008 11:43PM

- The Cartoon Guide to Löb's Theorem, 17 August 2008 08:35PM

- Dumb Deplaning, 18 August 2008 11:49PM

- You Provably Can't Trust Yourself, 19 August 2008 08:35PM

- No License To Be Human, 20 August 2008 11:18PM

- Invisible Frameworks, 22 August 2008 03:36AM

- Mirrors and Paintings, 23 August 2008 12:29AM

- Unnatural Categories, 24 August 2008 01:00AM

- Magical Categories, 24 August 2008 07:51PM

- Three Fallacies of Teleology, 25 August 2008 10:27PM

- Dreams of AI Design, 26 August 2008 11:28PM

- Against Modal Logics, 27 August 2008 10:13PM

- Harder Choices Matter Less, 29 August 2008 02:02AM

- Qualitative Strategies of Friendliness, 30 August 2008 02:12AM

- Dreams of Friendliness, 31 August 2008 01:20AM

- Brief Break, 31 August 2008 04:00PM

- Rationality Quotes 12, 01 September 2008 08:00PM

- Rationality Quotes 13, 02 September 2008 04:00PM

- The True Prisoner's Dilemma, 03 September 2008 09:34PM

- The Truly Iterated Prisoner's Dilemma, 04 September 2008 06:00PM

- Rationality Quotes 14, 05 September 2008 08:16PM

- Rationality Quotes 15, 06 September 2008 06:53PM

- Rationality Quotes 16, 08 September 2008 03:36AM

- Singularity Summit 2008, 08 September 2008 11:26PM

- Points of Departure, 09 September 2008 09:18PM

- Rationality Quotes 17, 11 September 2008 03:29AM

- Excluding the Supernatural, 12 September 2008 12:12AM

- Psychic Powers, 12 September 2008 07:28PM

- Optimization, 13 September 2008 04:00PM

- My Childhood Death Spiral, 15 September 2008 03:42AM

- My Best and Worst Mistake, 16 September 2008 12:43AM

- Raised in Technophilia, 17 September 2008 02:06AM

- A Prodigy of Refutation, 18 September 2008 01:57AM

- The Sheer Folly of Callow Youth, 19 September 2008 01:30AM

- Say It Loud, 19 September 2008 05:34PM

- Ban the Bear, 19 September 2008 06:14PM

- How Many LHC Failures Is Too Many?, 20 September 2008 09:38PM

- Horrible LHC Inconsistency, 22 September 2008 03:12AM

- That Tiny Note of Discord, 23 September 2008 06:02AM

- Fighting a Rearguard Action Against the Truth, 24 September 2008 01:23AM

- My Naturalistic Awakening, 25 September 2008 06:58AM

- The Level Above Mine, 26 September 2008 09:18AM

- Competent Elites, 27 September 2008 12:07AM

- Above-Average AI Scientists, 28 September 2008 11:04AM

- Friedman's "Prediction vs. Explanation", 29 September 2008 06:15AM

- The Magnitude of His Own Folly, 30 September 2008 11:31AM

- Awww, a Zebra, 01 October 2008 01:28AM

- Intrade and the Dow Drop, 01 October 2008 03:12AM

- Trying to Try, 01 October 2008 08:58AM

- Use the Try Harder, Luke, 02 October 2008 10:52AM

- Rationality Quotes 18, 03 October 2008 12:39PM

- Beyond the Reach of God, 04 October 2008 03:42PM

- My Bayesian Enlightenment, 05 October 2008 04:45PM

- Bay Area Meetup for Singularity Summit, 06 October 2008 12:50PM

- On Doing the Impossible, 06 October 2008 03:13PM

- Make an Extraordinary Effort, 07 October 2008 03:15PM

- Shut up and do the impossible!, 08 October 2008 09:24PM

- AIs and Gatekeepers Unite!, 09 October 2008 05:04PM

- Crisis of Faith, 10 October 2008 10:08PM

- The Ritual, 11 October 2008 11:52PM

- Rationality Quotes 19, 12 October 2008 08:10PM

- Why Does Power Corrupt?, 14 October 2008 12:23AM

- Ends Don't Justify Means (Among Humans), 14 October 2008 09:00PM

- Entangled Truths, Contagious Lies, 15 October 2008 11:39PM

- Traditional Capitalist Values, 17 October 2008 01:07AM

- Dark Side Epistemology, 17 October 2008 11:55PM

- Protected From Myself, 19 October 2008 12:09AM

- Ethical Inhibitions, 19 October 2008 08:44PM

- Ethical Injunctions, 20 October 2008 11:00PM

- Prices or Bindings?, 21 October 2008 04:00PM

- Ethics Notes, 21 October 2008 09:57PM

- Which Parts Are "Me"?, 22 October 2008 06:15PM

- Inner Goodness, 23 October 2008 10:19PM

- San Jose Meetup, Sat 10/25 @ 7:30pm, 23 October 2008 10:55PM

- Expected Creative Surprises, 24 October 2008 10:22PM

- Belief in Intelligence, 25 October 2008 03:00PM

- Aiming at the Target, 26 October 2008 04:47PM

- Measuring Optimization Power, 27 October 2008 09:44PM

- Efficient Cross-Domain Optimization, 28 October 2008 04:33PM

- Economic Definition of Intelligence?, 29 October 2008 07:32PM

- Intelligence in Economics, 30 October 2008 09:17PM

- Mundane Magic, 31 October 2008 04:00PM

- BHTV: Jaron Lanier and Yudkowsky, 01 November 2008 05:04PM

- Building Something Smarter, 02 November 2008 05:00PM

- Complexity and Intelligence, 03 November 2008 08:27PM

- Today's Inspirational Tale, 04 November 2008 04:15PM

- Hanging Out My Speaker's Shingle, 05 November 2008 02:00PM

- Back Up and Ask Whether, Not Why, 06 November 2008 07:20PM

- Recognizing Intelligence, 07 November 2008 11:22PM

- Lawful Creativity, 08 November 2008 07:54PM

- Ask OB: Leaving the Fold, 09 November 2008 06:08PM

- Lawful Uncertainty, 10 November 2008 09:06PM

- Worse Than Random, 11 November 2008 07:01PM

- The Weighted Majority Algorithm, 12 November 2008 11:19PM

- Bay Area Meetup: 11/17 8PM Menlo Park, 13 November 2008 05:32AM

- Selling Nonapples, 13 November 2008 08:10PM

- The Nature of Logic, 15 November 2008 06:20AM

- Boston-area Meetup: 11/18/08 9pm MIT/Cambridge, 16 November 2008 03:19AM

- Logical or Connectionist AI?, 17 November 2008 08:03AM

- Whither OB?, 17 November 2008 07:38PM

- Failure By Analogy, 18 November 2008 02:54AM

- Failure By Affective Analogy, 18 November 2008 07:14AM

- The Weak Inside View, 18 November 2008 06:37PM

- The First World Takeover, 19 November 2008 03:00PM

- Whence Your Abstractions?, 20 November 2008 01:07AM

- Observing Optimization, 21 November 2008 05:39AM

- Life's Story Continues, 21 November 2008 11:05PM

- Surprised by Brains, 23 November 2008 07:26AM

- Cascades, Cycles, Insight..., 24 November 2008 09:33AM

- ...Recursion, Magic, 25 November 2008 09:10AM

- The Complete Idiot's Guide to Ad Hominem, 25 November 2008 09:47PM

- Engelbart: Insufficiently Recursive, 26 November 2008 08:31AM

- Total Nano Domination, 27 November 2008 09:54AM

- Thanksgiving Prayer, 28 November 2008 04:17AM

- Chaotic Inversion, 29 November 2008 10:57AM

- Singletons Rule OK, 30 November 2008 04:45PM

- Disappointment in the Future, 01 December 2008 04:45AM

- Recursive Self-Improvement, 01 December 2008 08:49PM

- Hard Takeoff, 02 December 2008 08:44PM

- Permitted Possibilities, & Locality, 03 December 2008 09:20PM

- Underconstrained Abstractions, 04 December 2008 01:58PM

- Sustained Strong Recursion, 05 December 2008 09:03PM

- Is That Your True Rejection?, 06 December 2008 02:26PM

- Artificial Mysterious Intelligence, 07 December 2008 08:05PM

- True Sources of Disagreement, 08 December 2008 03:51PM

- Disjunctions, Antipredictions, Etc., 09 December 2008 03:13PM

- Bay Area Meetup Wed 12/10 @8pm, 10 December 2008 12:13AM

- The Mechanics of Disagreement, 10 December 2008 02:01PM

- What I Think, If Not Why, 11 December 2008 05:41PM

- You Only Live Twice, 12 December 2008 07:14PM

- BHTV: de Grey and Yudkowsky, 13 December 2008 03:28PM

- For The People Who Are Still Alive, 14 December 2008 05:13PM

- Not Taking Over the World, 15 December 2008 10:18PM

- Visualizing Eutopia, 16 December 2008 06:39PM

- Prolegomena to a Theory of Fun, 17 December 2008 11:33PM

- High Challenge, 19 December 2008 12:51AM

- Complex Novelty, 20 December 2008 12:31AM

- Sensual Experience, 21 December 2008 12:56AM

- Living By Your Own Strength, 22 December 2008 12:37AM

- Rationality Quotes 20, 22 December 2008 10:05PM

- Imaginary Positions, 23 December 2008 05:35PM

- Harmful Options, 25 December 2008 02:26AM

- Devil's Offers, 25 December 2008 05:00PM

- Nonperson Predicates, 27 December 2008 01:47AM

- Nonsentient Optimizers, 27 December 2008 02:32AM

- Nonsentient Bloggers, 27 December 2008 04:27PM

- Can't Unbirth a Child, 28 December 2008 05:00PM

- Amputation of Destiny, 29 December 2008 06:00PM

- Dunbar's Function, 31 December 2008 02:26AM

- A New Day, 31 December 2008 06:40PM

- Free to Optimize, 02 January 2009 01:41AM

- The Uses of Fun (Theory), 02 January 2009 08:30PM

- Growing Up is Hard, 04 January 2009 03:55AM

- Changing Emotions, 05 January 2009 12:05AM

- Rationality Quotes 21, 06 January 2009 02:45AM

- Emotional Involvement, 06 January 2009 10:23PM

- Rationality Quotes 22, 07 January 2009 12:00PM

- Serious Stories, 08 January 2009 11:49PM

- Rationality Quotes 23, 10 January 2009 12:12AM

- Continuous Improvement, 11 January 2009 02:09AM

- Eutopia is Scary, 12 January 2009 05:28AM

- Building Weirdtopia, 12 January 2009 08:35PM

- She has joined the Conspiracy, 13 January 2009 07:48PM

- Justified Expectation of Pleasant Surprises, 15 January 2009 07:26AM

- Seduced by Imagination, 16 January 2009 03:10AM

- Getting Nearer, 17 January 2009 09:28AM

- In Praise of Boredom, 18 January 2009 09:03AM

- Sympathetic Minds, 19 January 2009 09:31AM

- Interpersonal Entanglement, 20 January 2009 06:17AM

- Failed Utopia #4-2, 21 January 2009 11:04AM

- Investing for the Long Slump, 22 January 2009 08:56AM

- Higher Purpose, 23 January 2009 09:58AM

- Rationality Quotes 24, 24 January 2009 04:59AM

- The Fun Theory Sequence, 25 January 2009 11:18AM

- BHTV: Yudkowsky / Wilkinson, 26 January 2009 01:10AM

- 31 Laws of Fun, 26 January 2009 10:13AM

- OB Status Update, 27 January 2009 09:34AM

- Rationality Quotes 25, 28 January 2009 08:39AM

- Value is Fragile, 29 January 2009 08:46AM

- Three Worlds Collide (0/8), 30 January 2009 12:07PM

- The Baby-Eating Aliens (1/8), 30 January 2009 12:07PM

- War and/or Peace (2/8), 31 January 2009 08:42AM

- The Super Happy People (3/8), 01 February 2009 08:18AM

- Interlude with the Confessor (4/8), 02 February 2009 09:11AM

- Three Worlds Decide (5/8), 03 February 2009 09:14AM

- Normal Ending: Last Tears (6/8), 04 February 2009 08:45AM

- True Ending: Sacrificial Fire (7/8), 05 February 2009 10:57AM

- Epilogue: Atonement (8/8), 06 February 2009 11:52AM

- The Thing That I Protect, 07 February 2009 07:18PM

- ...And Say No More Of It, 09 February 2009 12:15AM

- (Moral) Truth in Fiction?, 09 February 2009 05:26PM

- Informers and Persuaders, 10 February 2009 08:22PM

- Cynicism in Ev-Psych (and Econ?), 11 February 2009 03:06PM

- The Evolutionary-Cognitive Boundary, 12 February 2009 04:44PM

- An Especially Elegant Evpsych Experiment, 13 February 2009 02:58PM

- Rationality Quotes 26, 14 February 2009 04:14PM

- An African Folktale, 16 February 2009 01:00AM

- Cynical About Cynicism, 17 February 2009 12:49AM

- Good Idealistic Books are Rare, 17 February 2009 06:41PM

- Against Maturity, 18 February 2009 11:34PM

- Pretending to be Wise, 19 February 2009 10:30PM

- Wise Pretensions v.0, 20 February 2009 05:02PM

- Rationality Quotes 27, 22 February 2009 01:55AM

- Fairness vs. Goodness, 22 February 2009 08:22PM

- On Not Having an Advance Abyssal Plan, 23 February 2009 08:20PM

- Formative Youth, 24 February 2009 11:02PM

- Tell Your Rationalist Origin Story, 25 February 2009 05:16PM

- Markets are Anti-Inductive, 26 February 2009 12:55AM

- Tell Your Rationalist Origin Story... at Less Wrong, 27 February 2009 04:06AM

- The Most Important Thing You Learned, 27 February 2009 08:15PM

- The Most Frequently Useful Thing, 28 February 2009 06:43PM

- The Most Frequently Useful Thing, 28 February 2009 07:54PM

- That You'd Tell All Your Friends, 01 March 2009 12:04PM

- Unteachable Excellence, 02 March 2009 03:33PM

- Teaching the Unteachable, 03 March 2009 11:14PM

- No, Really, I've Deceived Myself, 04 March 2009 11:29PM

- Belief in Self-Deception, 05 March 2009 03:20PM

- Posting now enabled, 05 March 2009 03:56PM

- Slow down a little... maybe?, 07 March 2009 01:34AM

- Moore's Paradox, 08 March 2009 02:27AM

- Don't Believe You'll Self-Deceive, 09 March 2009 08:03AM

- Striving to Accept, 09 March 2009 11:29PM

- Wanted: Python open source volunteers, 11 March 2009 04:59AM

- Raising the Sanity Waterline, 12 March 2009 04:28AM

- A Sense That More Is Possible, 13 March 2009 01:15AM

- Epistemic Viciousness, 13 March 2009 11:33PM

- Schools Proliferating Without Evidence, 15 March 2009 06:43AM

- 3 Levels of Rationality Verification, 15 March 2009 05:19PM

- What Do We Mean By "Rationality"?, 16 March 2009 10:33PM

- Comments for "Rationality", 16 March 2009 10:34PM

- Rationalist Fiction, 19 March 2009 08:22AM

- Why Our Kind Can't Cooperate, 20 March 2009 08:37AM

- Tolerate Tolerance, 21 March 2009 07:34AM

- You're Calling *Who* A Cult Leader?, 22 March 2009 06:57AM

- BHTV: Yudkowsky & Adam Frank on "religious experience", 23 March 2009 01:33AM

- On Things that are Awesome, 24 March 2009 03:24AM

- Hyakujo's Fox, 24 March 2009 10:14AM

- The Sacred Mundane, 25 March 2009 09:53AM

- Your Price for Joining, 26 March 2009 07:16AM

- Can Humanism Match Religion's Output?, 27 March 2009 11:32AM

- Church vs. Taskforce, 28 March 2009 09:23AM

- Rationality: Common Interest of Many Causes, 29 March 2009 10:49AM

- Helpless Individuals, 30 March 2009 11:10AM

- Money: The Unit of Caring, 31 March 2009 12:35PM

- Purchase Fuzzies and Utilons Separately, 01 April 2009 09:51AM

- Wrong Tomorrow, 02 April 2009 08:18AM

- Selecting Rationalist Groups, 02 April 2009 04:21PM

- Rationality is Systematized Winning, 03 April 2009 02:41PM

- Incremental Progress and the Valley, 04 April 2009 04:42PM

- Real-Life Anthropic Weirdness, 05 April 2009 10:26PM

- Newcomb's Problem standard positions, 06 April 2009 05:05PM

- Extenuating Circumstances, 06 April 2009 10:57PM

- Of Lies and Black Swan Blowups, 07 April 2009 06:26PM

- Whining-Based Communities, 07 April 2009 08:31PM

- Zero-based karma coming through, 08 April 2009 01:19AM

- Mandatory Secret Identities, 08 April 2009 06:10PM

- Beware of Other-Optimizing, 10 April 2009 01:58AM

- That Crisis thing seems pretty useful, 10 April 2009 05:10PM

- The Unfinished Mystery of the Shangri-La Diet, 10 April 2009 08:30PM

- Akrasia and Shangri-La, 10 April 2009 08:53PM

- Twelve Virtues booklet printing?, 11 April 2009 02:40PM

- Bystander Apathy, 13 April 2009 01:26AM

- Collective Apathy and the Internet, 14 April 2009 12:02AM

- Bayesians vs. Barbarians, 14 April 2009 11:45PM

- Of Gender and Rationality, 16 April 2009 12:56AM

- My Way, 17 April 2009 01:25AM

- Rationality Quotes - April 2009, 18 April 2009 01:26AM

- Great Books of Failure, 19 April 2009 12:59AM

- The Sin of Underconfidence, 20 April 2009 06:30AM

- Well-Kept Gardens Die By Pacifism, 21 April 2009 02:44AM

- Proposal: Use the Wiki for Concepts, 22 April 2009 05:21AM

- Go Forth and Create the Art!, 23 April 2009 01:37AM

- This Didn't Have To Happen, 23 April 2009 07:07PM

- Practical Advice Backed By Deep Theories, 25 April 2009 06:52PM

- Less Meta, 26 April 2009 05:38AM

- The Craft and the Community, 26 April 2009 05:52PM

- The End (of Sequences), 27 April 2009 09:07PM

- Final Words, 27 April 2009 09:12PM

- Special Status Needs Special Support, 04 May 2009 10:59PM

- Wiki.lesswrong.com Is Live, 06 May 2009 05:17AM

- Willpower Hax #487: Execute by Default, 12 May 2009 06:46AM

- "Open-Mindedness" - the video, 14 May 2009 06:17AM

- "What Is Wrong With Our Thoughts", 17 May 2009 07:24AM

- This Failing Earth, 24 May 2009 04:09PM

- Anime Explains the Epimenides Paradox, 27 May 2009 09:12PM

- Do Fandoms Need Awfulness?, 28 May 2009 06:03AM

- With whom shall I diavlog?, 03 June 2009 03:20AM

- Honesty: Beyond Internal Truth, 06 June 2009 02:59AM

- ESR's comments on some EY:OB/LW posts, 20 June 2009 12:16AM

- Nonparametric Ethics, 20 June 2009 11:31AM

- Atheism = Untheism + Antitheism, 01 July 2009 02:19AM

- Jul 12 Bay Area meetup - Hanson, Vassar, Yudkowsky, 11 July 2009 09:13PM

- The Strangest Thing An AI Could Tell You, 15 July 2009 02:27AM

- Are You Anosognosic?, 19 July 2009 04:35AM

- Article upvoting, 19 July 2009 05:03PM

- Timeless Decision Theory: Problems I Can't Solve, 20 July 2009 12:02AM

- Of Exclusionary Speech and Gender Politics, 21 July 2009 07:22AM

- The Hero With A Thousand Chances, 31 July 2009 04:25AM

- Unspeakable Morality, 04 August 2009 05:57AM

- Dreams with Damaged Priors, 08 August 2009 10:31PM

- Singularity Summit 2009 (quick post), 16 August 2009 11:29PM

- Ingredients of Timeless Decision Theory, 19 August 2009 01:10AM

- Timeless Decision Theory and Meta-Circular Decision Theory, 20 August 2009 10:07PM

- Working Mantras, 24 August 2009 10:08PM

- The Sword of Good, 03 September 2009 12:53AM

- Bay Area OBLW Meetup Sep 12, 06 September 2009 12:00AM

- Forcing Anthropics: Boltzmann Brains, 07 September 2009 07:02PM

- Why I'm Staying On Bloggingheads.tv, 07 September 2009 08:15PM

- FHI postdoc at Oxford, 08 September 2009 06:18PM

- Outlawing Anthropics: An Updateless Dilemma, 08 September 2009 06:31PM

- Let Them Debate College Students, 09 September 2009 06:15PM

- The Lifespan Dilemma, 10 September 2009 06:45PM

- The Finale of the Ultimate Meta Mega Crossover, 20 September 2009 05:21PM

- The Anthropic Trilemma, 27 September 2009 01:47AM

- Privileging the Hypothesis, 29 September 2009 12:40AM

- I'm Not Saying People Are Stupid, 09 October 2009 04:23PM

- PredictionBook.com - Track your calibration, 14 October 2009 12:08AM

- Waterloo, ON, Canada Meetup: 6pm Sun Oct 18 '09!, 15 October 2009 04:30AM

- Rationality Quotes: October 2009, 22 October 2009 04:06PM

- Rolf Nelson's "The Rational Entrepreneur", 04 November 2009 09:31PM

- Hamster in Tutu Shuts Down Large Hadron Collider, 06 November 2009 04:29PM

- BHTV: Yudkowsky / Robert Greene, 16 November 2009 08:26PM

- Rationality Quotes November 2009, 29 November 2009 11:36PM

- Frequentist Statistics are Frequently Subjective, 04 December 2009 08:22PM

- The Contrarian Status Catch-22, 19 December 2009 10:40PM

- Sufficiently Advanced Sanity, 20 December 2009 06:11PM

- The Correct Contrarian Cluster, 21 December 2009 10:01PM

- Karma Changes, 22 December 2009 12:17AM

- The 9/11 Meta-Truther Conspiracy Theory, 22 December 2009 06:59PM

- Singularity Institute $100K Challenge Grant / 2009 Donations Reminder, 30 December 2009 12:36AM

- Drawing Two Aces, 03 January 2010 10:33AM

- TakeOnIt: Database of Expert Opinions, 05 January 2010 08:54PM

- High Status and Stupidity: Why?, 12 January 2010 04:36PM

- Meetup: Bay Area: Jan 15th, 7pm, 13 January 2010 10:02PM

- Normal Cryonics, 19 January 2010 07:08PM

- That Magical Click, 20 January 2010 04:35PM

- Logical Rudeness, 29 January 2010 06:48AM

- You're Entitled to Arguments, But Not (That Particular) Proof, 15 February 2010 07:58AM

- Demands for Particular Proof: Appendices, 15 February 2010 07:58AM

- Conversation Halters, 20 February 2010 03:00PM

- "Outside View!" as Conversation-Halter, 24 February 2010 05:53AM

- The Last Days of the Singularity Challenge, 26 February 2010 03:46AM

- Undiscriminating Skepticism, 14 March 2010 11:23PM

- Floating Festival Jul 22-25, 21 July 2010 06:36AM

- Bay Area Events Roundup, 24 July 2010 06:04AM

- LW's first job ad, 16 September 2010 10:04AM

Sequences

- Map and Territory

- Mysterious Answers to Mysterious Questions

- A Human's Guide to Words

- Politics is the Mind-Killer

- Death Spirals and the Cult Attractor

- Seeing with Fresh Eyes

- Noticing Confusion

- Against Rationalization

- Against Doublethink

- Overly Convenient Excuses

- Letting Go

- Reductionism

- Quantum Physics

- Metaethics

- Fun Theory

- The Craft and the Community

- Joy in the Merely Real

- Zombies

- Evolution

- Challenging the Difficult

- Yudkowsky's Coming of Age

The Martial Art of Rationality

Eliezer Yudkowsky, 22 November 2006 08:00PM

I often use the metaphor that rationality is the martial art of mind. You don't need huge, bulging muscles to learn martial arts - there's a tendency toward more athletic people being more likely to learn martial arts, but that may be a matter of enjoyment as much as anything else. Some human beings are faster or stronger than others; but a martial art does not train the variance between humans. A martial art just trains your muscles - if you have the human-universal complex machinery of a hand, with tendons and muscles in the appropriate places, then you can learn to make a fist.

How can you train a variance? What does it mean to train +2 standard deviations of muscle? It's equally unclear what it would mean to train an IQ of 132.

But if you have a brain, with cortical and subcortical areas in the appropriate places, you might be able to learn to use it properly. If you're a fast learner, you might learn faster - but the art of rationality isn't about that; it's about training brain machinery we all have in common.

Alas, our minds respond less readily to our will than our hands. Muscles are evolutionarily ancient subjects of neural control, while cognitive reflectivity is a comparatively more recent innovation. We shouldn't be surprised that muscles are easier to use than brains. But it is not wise to neglect the latter training because it is more difficult. It is not by bigger muscles that the human species rose to prominence upon Earth.

If you live in an urban area, you probably don't need to walk very far to find a martial arts dojo. Why aren't there dojos that teach rationality? One reason, perhaps, is that it's harder to verify skill. To rise a level in Tae Kwon Do, you might need to break a board of a certain width. If you succeed, all the onlookers can see and applaud. If you fail, your teacher can watch how you shape a fist, and check if you shape it correctly. If not, the teacher holds out a hand and makes a fist correctly, so that you can observe how to do so. Within martial arts schools, techniques of muscle have been refined and elaborated over generations. Techniques of rationality are harder to pass on, even to the most willing student. It is also harder to give impressive public exhibitions of rationality. This may partially explain why there are no rationality dojos as yet.

Very recently - in just the last few decades - the human species has acquired a great deal of new knowledge about human rationality. The most salient example would be the heuristics and biases program in experimental psychology. There is also the Bayesian systematization of probability theory and statistics; evolutionary psychology; social psychology. Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions. These fields give us new focusing lenses through which to view the landscape of our own minds. With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move. We have a shared vocabulary in which to describe problems and solutions. Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality.

Such understanding as I have of rationality, I acquired in the course of wrestling with the challenge of Artificial General Intelligence (an endeavor which, to actually succeed, would require sufficient mastery of rationality to build a complete working rationalist out of toothpicks and rubber bands). In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier. In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable. Some of the machinery is optimized for evolutionary selection pressures that run directly counter to our declared goals in using it. Deliberately we decide that we want to seek only the truth; but our brains have hardwired support for rationalizing falsehoods. We can try to compensate for what we choose to regard as flaws of the machinery; but we can't actually rewire the neural circuitry. Nor may a martial artist plate titanium over his bones - not today, at any rate.

Trying to synthesize a personal art of rationality, using the science of rationality, may prove awkward: One imagines trying to invent a martial art using an abstract theory of physics, game theory, and human anatomy. But humans are not reflectively blind; we do have a native instinct for introspection. The inner eye is not sightless; but it sees blurrily, with systematic distortions. We need, then, to apply the science to our intuitions, to use the abstract knowledge to correct our mental movements and augment our metacognitive skills. We are not writing a computer program to make a string puppet execute martial arts forms; it is our own mental limbs that we must move. Therefore we must connect theory to practice. We must come to see what the science means, for ourselves, for our daily inner life.

And we must, above all, figure out how to communicate the skill; which may not be a matter for declarative statements alone. Martial artists spar with each other, execute standard forms, and are watched throughout by their teachers. Calculus students do homework, and check their answers. Olympic runners continually try to beat their best previous time, as measured by a stopwatch.

How to communicate procedural skills of rationality, or measure them, is probably the single largest open issue that stands between humanity and rationality dojos - at least it's the part of the problem that most baffles me. Meanwhile I lecture. So does anyone out there have ideas?

Referenced by: Debiasing as Non-Self-Destruction • Changing the Definition of Science • A Sense That More Is Possible

Original with comments: The Martial Art of Rationality

Why truth? And...

Eliezer Yudkowsky, 27 November 2006 01:49AM

Some of the comments in this blog have touched on the question of why we ought to seek truth. (Thankfully not many have questioned what truth is.) Our shaping motivation for configuring our thoughts to rationality, which determines whether a given configuration is "good" or "bad", comes from whyever we wanted to find truth in the first place.

It is written: "The first virtue is curiosity." Curiosity is one reason to seek truth, and it may not be the only one, but it has a special and admirable purity. If your motive is curiosity, you will assign priority to questions according to how the questions, themselves, tickle your personal aesthetic sense. A trickier challenge, with a greater probability of failure, may be worth more effort than a simpler one, just because it is more fun.

Some people, I suspect, may object that curiosity is an emotion and is therefore "not rational". I label an emotion as "not rational" if it rests on mistaken beliefs, or rather, on irrational epistemic conduct: "If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm." Conversely, then, an emotion which is evoked by correct beliefs or epistemically rational thinking is a "rational emotion"; and this has the advantage of letting us regard calm as an emotional state, rather than a privileged default. When people think of "emotion" and "rationality" as opposed, I suspect that they are really thinking of System 1 and System 2 - fast perceptual judgments versus slow deliberative judgments. Deliberative judgments aren't always true, and perceptual judgments aren't always false; so it is very important to distinguish that dichotomy from "rationality". Both systems can serve the goal of truth, or defeat it, according to how they are used.

Besides sheer emotional curiosity, what other motives are there for desiring truth? Well, you might want to accomplish some specific real-world goal, like building an airplane, and therefore you need to know some specific truth about aerodynamics. Or more mundanely, you want chocolate milk, and therefore you want to know whether the local grocery has chocolate milk, so you can choose whether to walk there or somewhere else. If this is the reason you want truth, then the priority you assign to your questions will reflect the expected utility of their information - how much the possible answers influence your choices, how much your choices matter, and how much you expect to find an answer that changes your choice from its default.

To seek truth merely for its instrumental value may seem impure - should we not desire the truth for its own sake? - but such investigations are extremely important because they create an outside criterion of verification: if your airplane drops out of the sky, or if you get to the store and find no chocolate milk, it's a hint that you did something wrong. You get back feedback on which modes of thinking work, and which don't. Pure curiosity is a wonderful thing, but it may not linger too long on verifying its answers, once the attractive mystery is gone. Curiosity, as a human emotion, has been around since long before the ancient Greeks. But what set humanity firmly on the path of Science was noticing that certain modes of thinking uncovered beliefs that let us manipulate the world. As far as sheer curiosity goes, spinning campfire tales of gods and heroes satisfied that desire just as well, and no one realized that anything was wrong with that.

Are there motives for seeking truth besides curiosity and pragmatism? The third reason that I can think of is morality: You believe that to seek the truth is noble and important and worthwhile. Though such an ideal also attaches an intrinsic value to truth, it's a very different state of mind from curiosity. Being curious about what's behind the curtain doesn't feel the same as believing that you have a moral duty to look there. In the latter state of mind, you are a lot more likely to believe that someone else should look behind the curtain, too, or castigate them if they deliberately close their eyes. For this reason, I would also label as "morality" the belief that truthseeking is pragmatically important to society, and therefore is incumbent as a duty upon all. Your priorities, under this motivation, will be determined by your ideals about which truths are most important (not most useful or most intriguing); or your moral ideals about when, under what circumstances, the duty to seek truth is at its strongest.

I tend to be suspicious of morality as a motivation for rationality, not because I reject the moral ideal, but because it invites certain kinds of trouble. It is too easy to acquire, as learned moral duties, modes of thinking that are dreadful missteps in the dance. Consider Mr. Spock of Star Trek, a naive archetype of rationality. Spock's emotional state is always set to "calm", even when wildly inappropriate. He often gives many significant digits for probabilities that are grossly uncalibrated. (E.g: "Captain, if you steer the Enterprise directly into that black hole, our probability of surviving is only 2.234%" Yet nine times out of ten the Enterprise is not destroyed. What kind of tragic fool gives four significant digits for a figure that is off by two orders of magnitude?) Yet this popular image is how many people conceive of the duty to be "rational" - small wonder that they do not embrace it wholeheartedly. To make rationality into a moral duty is to give it all the dreadful degrees of freedom of an arbitrary tribal custom. People arrive at the wrong answer, and then indignantly protest that they acted with propriety, rather than learning from their mistake.

And yet if we're going to improve our skills of rationality, go beyond the standards of performance set by hunter-gatherers, we'll need deliberate beliefs about how to think with propriety. When we write new mental programs for ourselves, they start out in System 2, the deliberate system, and are only slowly - if ever - trained into the neural circuitry that underlies System 1. So if there are certain kinds of thinking that we find we want to avoid - like, say, biases - it will end up represented, within System 2, as an injunction not to think that way; a professed duty of avoidance.

If we want the truth, we can most effectively obtain it by thinking in certain ways, rather than others; and these are the techniques of rationality. Some of the techniques of rationality involve overcoming a certain class of obstacles, the biases...

(Continued in next post: "What's a bias, again?")

Sequence: Map and Territory

What Do We Mean By "Rationality"?

What is Evidence?

Referenced by: Outside the Laboratory • Cultish Countercultishness • Something to Protect • Newcomb's Problem and Regret of Rationality • Zombies! Zombies? • A Sense That More Is Possible

Original with comments: Why truth? And...

...What's a bias, again?

Eliezer Yudkowsky, 27 November 2006 01:50AM

(Continued from previous post: "Why truth? And...")

A bias is a certain kind of obstacle to our goal of obtaining truth - its character as an "obstacle" stems from this goal of truth - but there are many obstacles that are not "biases".

If we start right out by asking "What is bias?", it comes at the question in the wrong order. As the proverb goes, "There are forty kinds of lunacy but only one kind of common sense." The truth is a narrow target, a small region of configuration space to hit. "She loves me, she loves me not" may be a binary question, but E=MC^2 is a tiny dot in the space of all equations, like a winning lottery ticket in the space of all lottery tickets. Error is not an exceptional condition; it is success which is a priori so improbable that it requires an explanation.

We don't start out with a moral duty to "reduce bias", because biases are bad and evil and Just Not Done. This is the sort of thinking someone might end up with if they acquired a deontological duty of "rationality" by social osmosis, which leads to people trying to execute techniques without appreciating the reason for them. (Which is bad and evil and Just Not Done, according to Surely You're Joking, Mr. Feynman, which I read as a kid.)

Rather, we want to get to the truth, for whatever reason, and we find various obstacles getting in the way of our goal. These obstacles are not wholly dissimilar to each other - for example, there are obstacles that have to do with not having enough computing power available, or information being expensive. It so happens that a large group of obstacles seem to have a certain character in common - to cluster in a region of obstacle-to-truth space - and this cluster has been labeled "biases".

What is a bias? Can we look at the empirical cluster and find a compact test for membership? Perhaps we will find that we can't really give any explanation better than pointing to a few extensional examples, and hoping the listener understands. If you are a scientist just beginning to investigate fire, it might be a lot wiser to point to a campfire and say "Fire is that orangey-bright hot stuff over there," rather than saying "I define fire as an alchemical transmutation of substances which releases phlogiston." As I said in The Simple Truth, you should not ignore something just because you can't define it. I can't quote the equations of General Relativity from memory, but nonetheless if I walk off a cliff, I'll fall. And we can say the same of biases - they won't hit any less hard if it turns out we can't define compactly what a "bias" is. So we might point to conjunction fallacies, to overconfidence, to the availability and representativeness heuristics, to base rate neglect, and say: "Stuff like that."

With all that said, we seem to label as "biases" those obstacles to truth which are produced, not by the cost of information, nor by limited computing power, but by the shape of our own mental machinery. For example, the machinery is evolutionarily optimized to purposes that actively oppose epistemic accuracy; for example, the machinery to win arguments in adaptive political contexts. Or the selection pressure ran skew to epistemic accuracy; for example, believing what others believe, to get along socially. Or, in the classic heuristic-and-bias, the machinery operates by an identifiable algorithm that does some useful work but also produces systematic errors: the availability heuristic is not itself a bias, but it gives rise to identifiable, compactly describable biases. Our brains are doing something wrong, and after a lot of experimentation and/or heavy thinking, someone identifies the problem in a fashion that System 2 can comprehend; then we call it a "bias". Even if we can do no better for knowing, it is still a failure that arises, in an identifiable fashion, from a particular kind of cognitive machinery - not from having too little machinery, but from the shape of the machinery itself.

"Biases" are distinguished from errors that arise from cognitive content, such as adopted beliefs, or adopted moral duties. These we call "mistakes", rather than "biases", and they are much easier to correct, once we've noticed them for ourselves. (Though the source of the mistake, or the source of the source of the mistake, may ultimately be some bias.)

"Biases" are distinguished from errors that arise from damage to an individual human brain, or from absorbed cultural mores; biases arise from machinery that is humanly universal.

Plato wasn't "biased" because he was ignorant of General Relativity - he had no way to gather that information, his ignorance did not arise from the shape of his mental machinery. But if Plato believed that philosophers would make better kings because he himself was a philosopher - and this belief, in turn, arose because of a universal adaptive political instinct for self-promotion, and not because Plato's daddy told him that everyone has a moral duty to promote their own profession to governorship, or because Plato sniffed too much glue as a kid - then that was a bias, whether Plato was ever warned of it or not.

Biases may not be cheap to correct. They may not even be correctable. But where we look upon our own mental machinery and see a causal account of an identifiable class of errors; and when the problem seems to come from the evolved shape of the machinery, rather from there being too little machinery, or bad specific content; then we call that a bias.

Personally, I see our quest in terms of acquiring personal skills of rationality, in improving truthfinding technique. The challenge is to attain the positive goal of truth, not to avoid the negative goal of failure. Failurespace is wide, infinite errors in infinite variety. It is difficult to describe so huge a space: "What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world." Success-space is narrower, and therefore more can be said about it.

While I am not averse (as you can see) to discussing definitions, we should remember that is not our primary goal. We are here to pursue the great human quest for truth: for we have desperate need of the knowledge, and besides, we're curious. To this end let us strive to overcome whatever obstacles lie in our way, whether we call them "biases" or not.

Original with comments: ...What's a bias, again?

The Proper Use of Humility

Eliezer Yudkowsky, 01 December 2006 07:55PM

It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial.

Consider the creationist who says: "But who can really know whether evolution is correct? It is just a theory. You should be more humble and open-minded." Is this humility? The creationist practices a very selective underconfidence, refusing to integrate massive weights of evidence in favor of a conclusion he finds uncomfortable. I would say that whether you call this "humility" or not, it is the wrong step in the dance.

What about the engineer who humbly designs fail-safe mechanisms into machinery, even though he's damn sure the machinery won't fail? This seems like a good kind of humility to me. Historically, it's not unheard-of for an engineer to be damn sure a new machine won't fail, and then it fails anyway.

What about the student who humbly double-checks the answers on his math test? Again I'd categorize that as good humility.

What about a student who says, "Well, no matter how many times I check, I can't ever be certain my test answers are correct," and therefore doesn't check even once? Even if this choice stems from an emotion similar to the emotion felt by the previous student, it is less wise.

You suggest studying harder, and the student replies: "No, it wouldn't work for me; I'm not one of the smart kids like you; nay, one so lowly as myself can hope for no better lot." This is social modesty, not humility. It has to do with regulating status in the tribe, rather than scientific process. If you ask someone to "be more humble", by default they'll associate the words to social modesty - which is an intuitive, everyday, ancestrally relevant concept. Scientific humility is a more recent and rarefied invention, and it is not inherently social. Scientific humility is something you would practice even if you were alone in a spacesuit, light years from Earth with no one watching. Or even if you received an absolute guarantee that no one would ever criticize you again, no matter what you said or thought of yourself. You'd still double-check your calculations if you were wise.

The student says: "But I've seen other students double-check their answers and then they still turned out to be wrong. Or what if, by the problem of induction, 2 + 2 = 5 this time around? No matter what I do, I won't be sure of myself." It sounds very profound, and very modest. But it is not coincidence that the student wants to hand in the test quickly, and go home and play video games.

The end of an era in physics does not always announce itself with thunder and trumpets; more often it begins with what seems like a small, small flaw... But because physicists have this arrogant idea that their models should work all the time, not just most of the time, they follow up on small flaws. Usually, the small flaw goes away under closer inspection. Rarely, the flaw widens to the point where it blows up the whole theory. Therefore it is written: "If you do not seek perfection you will halt before taking your first steps."

But think of the social audacity of trying to be right all the time! I seriously suspect that if Science claimed that evolutionary theory is true most of the time but not all of the time - or if Science conceded that maybe on some days the Earth is flat, but who really knows - then scientists would have better social reputations. Science would be viewed as less confrontational, because we wouldn't have to argue with people who say the Earth is flat - there would be room for compromise. When you argue a lot, people look upon you as confrontational. If you repeatedly refuse to compromise, it's even worse. Consider it as a question of tribal status: scientists have certainly earned some extra status in exchange for such socially useful tools as medicine and cellphones. But this social status does not justify their insistence that only scientific ideas on evolution be taught in public schools. Priests also have high social status, after all. Scientists are getting above themselves - they won a little status, and now they think they're chiefs of the whole tribe! They ought to be more humble, and compromise a little.

Many people seem to possess rather hazy views of "rationalist humility". It is dangerous to have a prescriptive principle which you only vaguely comprehend; your mental picture may have so many degrees of freedom that it can adapt to justify almost any deed. Where people have vague mental models that can be used to argue anything, they usually end up believing whatever they started out wanting to believe. This is so convenient that people are often reluctant to give up vagueness. But the purpose of our ethics is to move us, not be moved by us.

"Humility" is a virtue that is often misunderstood. This doesn't mean we should discard the concept of humility, but we should be careful using it. It may help to look at the actions recommended by a "humble" line of thinking, and ask: "Does acting this way make you stronger, or weaker?" If you think about the problem of induction as applied to a bridge that needs to stay up, it may sound reasonable to conclude that nothing is certain no matter what precautions are employed; but if you consider the real-world difference between adding a few extra cables, and shrugging, it seems clear enough what makes the stronger bridge.

The vast majority of appeals that I witness to "rationalist's humility" are excuses to shrug. The one who buys a lottery ticket, saying, "But you can't know that I'll lose." The one who disbelieves in evolution, saying, "But you can't prove to me that it's true." The one who refuses to confront a difficult-looking problem, saying, "It's probably too hard to solve." The problem is motivated skepticism aka disconfirmation bias - more heavily scrutinizing assertions that we don't want to believe. Humility, in its most commonly misunderstood form, is a fully general excuse not to believe something; since, after all, you can't be sure. Beware of fully general excuses!

A further problem is that humility is all too easy to profess. Dennett, in "Breaking the Spell", points out that while many religious assertions are very hard to believe, it is easy for people to believe that they ought to believe them. Dennett terms this "belief in belief". What would it mean to really assume, to really believe, that three is equal to one? It's a lot easier to believe that you should, somehow, believe that three equals one, and to make this response at the appropriate points in church. Dennett suggests that much "religious belief" should be studied as "religious profession" - what people think they should believe and what they know they ought to say.

It is all too easy to meet every counterargument by saying, "Well, of course I could be wrong." Then, having dutifully genuflected in the direction of Modesty, having made the required obeisance, you can go on about your way without changing a thing.

The temptation is always to claim the most points with the least effort. The temptation is to carefully integrate all incoming news in a way that lets us change our beliefs, and above all our actions, as little as possible. John Kenneth Galbraith said: "Faced with the choice of changing one's mind and proving that there is no need to do so, almost everyone gets busy on the proof." And the greater the inconvenience of changing one's mind, the more effort people will expend on the proof.

But y'know, if you're gonna do the same thing anyway, there's no point in going to such incredible lengths to rationalize it. Often I have witnessed people encountering new information, apparently accepting it, and then carefully explaining why they are going to do exactly the same thing they planned to do previously, but with a different justification. The point of thinking is to shape our plans; if you're going to keep the same plans anyway, why bother going to all that work to justify it? When you encounter new information, the hard part is to update, to react, rather than just letting the information disappear down a black hole. And humility, properly misunderstood, makes a wonderful black hole - all you have to do is admit you could be wrong. Therefore it is written: "To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty."

Sequence: Overly Convenient Excuses

The Third Alternative

Sequence: Challenging the Difficult

Tsuyoku Naritai! (I Want To Become Stronger)

Referenced by: The Modesty Argument • Outside the Laboratory • Tsuyoku Naritai! (I Want To Become Stronger) • The Error of Crowds • Debiasing as Non-Self-Destruction • The Third Alternative • Risk-Free Bonds Aren't • The Proper Use of Doubt • Could Anything Be Right? • The Level Above Mine • Crisis of Faith • Nonperson Predicates • Extenuating Circumstances • That Crisis thing seems pretty useful • The Sin of Underconfidence • The Contrarian Status Catch-22

Original with comments: The Proper Use of Humility

The Modesty Argument

Eliezer Yudkowsky, 10 December 2006 09:42PM

The Modesty Argument states that when two or more human beings have common knowledge that they disagree about a question of simple fact, they should each adjust their probability estimates in the direction of the others'. (For example, they might adopt the common mean of their probability distributions. If we use the logarithmic scoring rule, then the score of the average of a set of probability distributions is better than the average of the scores of the individual distributions, by Jensen's inequality.)

Put more simply: When you disagree with someone, even after talking over your reasons, the Modesty Argument claims that you should each adjust your probability estimates toward the other's, and keep doing this until you agree. The Modesty Argument is inspired by Aumann's Agreement Theorem, a very famous and oft-generalized result which shows that genuine Bayesians literally cannot agree to disagree; if genuine Bayesians have common knowledge of their individual probability estimates, they must all have the same probability estimate. ("Common knowledge" means that I know you disagree, you know I know you disagree, etc.)

I've always been suspicious of the Modesty Argument. It's been a long-running debate between myself and Robin Hanson.

Robin seems to endorse the Modesty Argument in papers such as Are Disagreements Honest? I, on the other hand, have held that it can be rational for an individual to not adjust their own probability estimate in the direction of someone else who disagrees with them.

How can I maintain this position in the face of Aumann's Agreement Theorem, which proves that genuine Bayesians cannot have common knowledge of a dispute about probability estimates? If genunie Bayesians will always agree with each other once they've exchanged probability estimates, shouldn't we Bayesian wannabes do the same?

To explain my reply, I begin with a metaphor: If I have five different accurate maps of a city, they will all be consistent with each other. Some philosophers, inspired by this, have held that "rationality" consists of having beliefs that are consistent among themselves. But, although accuracy necessarily implies consistency, consistency does not necessarily imply accuracy. If I sit in my living room with the curtains drawn, and make up five maps that are consistent with each other, but I don't actually walk around the city and make lines on paper that correspond to what I see, then my maps will be consistent but not accurate. When genuine Bayesians agree in their probability estimates, it's not because they're trying to be consistent - Aumann's Agreement Theorem doesn't invoke any explicit drive on the Bayesians' part to be consistent. That's what makes AAT surprising! Bayesians only try to be accurate; in the course of seeking to be accurate, they end up consistent. The Modesty Argument, that we can end up accurate in the course of seeking to be consistent, does not necessarily follow.

How can I maintain my position in the face of my admission that disputants will always improve their average score if they average together their individual probability distributions?

Suppose a creationist comes to me and offers: "You believe that natural selection is true, and I believe that it is false. Let us both agree to assign 50% probability to the proposition." And suppose that by drugs or hypnosis it was actually possible for both of us to contract to adjust our probability estimates in this way. This unquestionably improves our combined log-score, and our combined squared error. If as a matter of altruism, I value the creationist's accuracy as much as my own - if my loss function is symmetrical around the two of us - then I should agree. But what if I'm trying to maximize only my own individual accuracy? In the former case, the question is absolutely clear, and in the latter case it is not absolutely clear, to me at least, which opens up the possibility that they are different questions.

If I agree to a contract with the creationist in which we both use drugs or hypnosis to adjust our probability estimates, because I know that the group estimate must be improved thereby, I regard that as pursuing the goal of social altruism. It doesn't make creationism actually true, and it doesn't mean that I think creationism is true when I agree to the contract. If I thought creationism was 50% probable, I wouldn't need to sign a contract - I would have already updated my beliefs! It is tempting but false to regard adopting someone else's beliefs as a favor to them, and rationality as a matter of fairness, of equal compromise. Therefore it is written: "Do not believe you do others a favor if you accept their arguments; the favor is to you." Am I really doing myself a favor by agreeing with the creationist to take the average of our probability distributions?

I regard rationality in its purest form as an individual thing - not because rationalists have only selfish interests, but because of the form of the only admissible question: "Is is actually true?" Other considerations, such as the collective accuracy of a group that includes yourself, may be legitimate goals, and an important part of human existence - but they differ from that single pure question.

In Aumann's Agreement Theorem, all the individual Bayesians are trying to be accurate as individuals. If their explicit goal was to maximize group accuracy, AAT would not be surprising. So the improvement of group score is not a knockdown argument as to what an individual should do if they are trying purely to maximize their own accuracy, and it is that last quest which I identify as rationality. It is written: "Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory. If you fail to achieve a correct answer, it is futile to protest that you acted with propriety." From the standpoint of social altruism, someone may wish to be Modest, and enter a drug-or-hypnosis-enforced contract of Modesty, even if they fail to achieve a correct answer thereby.

The central argument for Modesty proposes something like a Rawlsian veil of ignorance - how can you know which of you is the honest truthseeker, and which the stubborn self-deceiver? The creationist believes that he is the sane one and you are the fool. Doesn't this make the situation symmetric around the two of you? If you average your estimates together, one of you must gain, and one of you must lose, since the shifts are in opposite directions; but by Jensen's inequality it is a positive-sum game. And since, by something like a Rawlsian veil of ignorance, you don't know which of you is really the fool, you ought to take the gamble. This argues that the socially altruistic move is also always the individually rational move.

And there's also the obvious reply: "But I know perfectly well who the fool is. It's the other guy. It doesn't matter that he says the same thing - he's still the fool."

This reply sounds bald and unconvincing when you consider it abstractly. But if you actually face a creationist, then it certainly feels like the correct answer - you're right, he's wrong, and you have valid evidence to know that, even if the creationist can recite exactly the same claim in front of a TV audience.

Robin Hanson sides with symmetry - this is clearest in his paper Uncommon Priors Require Origin Disputes - and therefore endorses the Modesty Argument. (Though I haven't seen him analyze the particular case of the creationist.)

I respond: Those who dream do not know they dream; but when you wake you know you are awake. Dreaming, you may think you are awake. You may even be convinced of it. But right now, when you really are awake, there isn't any doubt in your mind - nor should there be. If you, persuaded by the clever argument, decided to start doubting right now that you're really awake, then your Bayesian score would go down and you'd become that much less accurate. If you seriously tried to make yourself doubt that you were awake - in the sense of wondering if you might be in the midst of an ordinary human REM cycle - then you would probably do so because you wished to appear to yourself as rational, or because it was how you conceived of "rationality" as a matter of moral duty. Because you wanted to act with propriety. Not because you felt genuinely curious as to whether you were awake or asleep. Not because you felt you might really and truly be asleep. But because you didn't have an answer to the clever argument, just an (ahem) incommunicable insight that you were awake.

Russell Wallace put it thusly: "That we can postulate a mind of sufficiently low (dreaming) or distorted (insane) consciousness as to genuinely not know whether it's Russell or Napoleon doesn't mean I (the entity currently thinking these thoughts) could have been Napoleon, any more than the number 3 could have been the number 7. If you doubt this, consider the extreme case: a rock doesn't know whether it's me or a rock. That doesn't mean I could have been a rock."

There are other problems I see with the Modesty Argument, pragmatic matters of human rationality - if a fallible human tries to follow the Modesty Argument in practice, does this improve or disimprove personal rationality? To me it seems that the adherents of the Modesty Argument tend to profess Modesty but not actually practice it.

For example, let's say you're a scientist with a controversial belief - like the Modesty Argument itself, which is hardly a matter of common accord - and you spend some substantial amount of time and effort trying to prove, argue, examine, and generally forward this belief. Then one day you encounter the Modesty Argument, and it occurs to you that you should adjust your belief toward the modal belief of the scientific field. But then you'd have to give up your cherished hypothesis. So you do the obvious thing - I've seen at least two people do this on two different occasions - and say: "Pursuing my personal hypothesis has a net expected utility to Science. Even if I don't really believe that my theory is correct, I can still pursue it because of the categorical imperative: Science as a whole will be better off if scientists go on pursuing their own hypotheses." And then they continue exactly as before.

I am skeptical to say the least. Integrating the Modesty Argument as new evidence ought to produce a large effect on someone's life and plans. If it's being really integrated, that is, rather than flushed down a black hole. Your personal anticipation of success, the bright emotion with which you anticipate the confirmation of your theory, should diminish by literally orders of magnitude after accepting the Modesty Argument. The reason people buy lottery tickets is that the bright anticipation of winning ten million dollars, the dancing visions of speedboats and mansions, is not sufficiently diminished - as a strength of emotion - by the probability factor, the odds of a hundred million to one. The ticket buyer may even profess that the odds are a hundred million to one, but they don't anticipate it properly - they haven't integrated the mere verbal phrase "hundred million to one" on an emotional level.

So, when a scientist integrates the Modesty Argument as new evidence, should the resulting nearly total loss of hope have no effect on real-world plans originally formed in blessed ignorance and joyous anticipation of triumph? Especially when you consider that the scientist knew about the social utility to start with, while making the original plans? I think that's around as plausible as maintaining your exact original investment profile after the expected returns on some stocks change by a factor of a hundred. What's actually happening, one naturally suspects, is that the scientist finds that the Modesty Argument has uncomfortable implications; so they reach for an excuse, and invent on-the-fly the argument from social utility as a way of exactly cancelling out the Modesty Argument and preserving all their original plans.

But of course if I say that this is an argument against the Modesty Argument, that is pure ad hominem tu quoque. If its adherents fail to use the Modesty Argument properly, that does not imply it has any less force as logic.

Rather than go into more detail on the manifold ramifications of the Modesty Argument, I'm going to close with the thought experiment that initially convinced me of the falsity of the Modesty Argument. In the beginning it seemed to me reasonable that if feelings of 99% certainty were associated with a 70% frequency of true statements, on average across the global population, then the state of 99% certainty was like a "pointer" to 70% probability. But at one point I thought: "What should an (AI) superintelligence say in the same situation? Should it treat its 99% probability estimates as 70% probability estimates because so many human beings make the same mistake?" In particular, it occurred to me that, on the day the first true superintelligence was born, it would be undeniably true that - across the whole of Earth's history - the enormously vast majority of entities who had believed themselves superintelligent would be wrong. The majority of the referents of the pointer "I am a superintelligence" would be schizophrenics who believed they were God.

A superintelligence doesn't just believe the bald statement that it is a superintelligence - it presumably possesses a very detailed, very accurate self-model of its own cognitive systems, tracks in detail its own calibration, and so on. But if you tell this to a mental patient, the mental patient can immediately respond: "Ah, but I too possess a very detailed, very accurate self-model!" The mental patient may even come to sincerely believe this, in the moment of the reply. Does that mean the superintelligence should wonder if it is a mental patient? This is the opposite extreme of Russell Wallace asking if a rock could have been you, since it doesn't know if it's you or the rock.

One obvious reply is that human beings and superintelligences occupy different classes - we do not have the same ur-priors, or we are not part of the same anthropic reference class; some sharp distinction renders it impossible to group together superintelligences and schizophrenics in probability arguments. But one would then like to know exactly what this "sharp distinction" is, and how it is justified relative to the Modesty Argument. Can an evolutionist and a creationist also occupy different reference classes? It sounds astoundingly arrogant; but when I consider the actual, pragmatic situation, it seems to me that this is genuinely the case.

Or here's a more recent example - one that inspired me to write today's blog post, in fact. It's the true story of a customer struggling through five levels of Verizon customer support, all the way up to floor manager, in an ultimately futile quest to find someone who could understand the difference between .002 dollars per kilobyte and .002 cents per kilobyte. Audio [27 minutes], Transcript. It has to be heard to be believed. Sample of conversation: "Do you recognize that there's a difference between point zero zero two dollars and point zero zero two cents?" "No."

The key phrase that caught my attention and inspired me to write today's blog post is from the floor manager: "You already talked to a few different people here, and they've all explained to you that you're being billed .002 cents, and if you take it and put it on your calculator... we take the .002 as everybody has told you that you've called in and spoken to, and as our system bills accordingly, is correct."

Should George - the customer - have started doubting his arithmetic, because five levels of Verizon customer support, some of whom cited multiple years of experience, told him he was wrong? Should he have adjusted his probability estimate in their direction? A straightforward extension of Aumann's Agreement Theorem to impossible possible worlds, that is, uncertainty about the results of computations, proves that, had all parties been genuine Bayesians with common knowledge of each other's estimates, they would have had the same estimate. Jensen's inequality proves even more straightforwardly that, if George and the five levels of tech support had averaged together their probability estimates, they would have improved their average log score. If such arguments fail in this case, why do they succeed in other cases? And if you claim the Modesty Argument carries in this case, are you really telling me that if George had wanted only to find the truth for himself, he would have been wise to adjust his estimate in Verizon's direction? I know this is an argument from personal incredulity, but I think it's a good one.

On the whole, and in practice, it seems to me like Modesty is sometimes a good idea, and sometimes not. I exercise my individual discretion and judgment to decide, even knowing that I might be biased or self-favoring in doing so, because the alternative of being Modest in every case seems to me much worse.

But the question also seems to have a definite anthropic flavor. Anthropic probabilities still confuse me; I've read arguments but I have been unable to resolve them to my own satisfaction. Therefore, I confess, I am not able to give a full account of how the Modesty Argument is resolved.

Modest, aren't I?

Referenced by: The Error of Crowds • On Expressing Your Concerns • Lonely Dissent • My Strange Beliefs • Dissolving the Question • The Rhythm of Disagreement • Are You Anosognosic?

Original with comments: The Modesty Argument

"I don't know."

Eliezer Yudkowsky, 21 December 2006 06:27PM

An edited transcript of a long instant-messenger conversation that took place regarding the phrase, "I don't know", sparked by Robin Hanson's previous post, "You Are Never Entitled to Your Opinion."

[08:50] Eliezer: http://www.overcomingbias.com/2006/12/you_are_never_e.html

[09:01] X: it still seems that saying "i don't know" in some situations is better than giving your best guess

[09:01] X: especially if you are dealing with people who will take you at your word who are not rationalists

[09:02] Eliezer: in real life, you have to choose, and bet, at some betting odds

[09:02] Eliezer: i.e., people who want to say "I don't know" for cryonics still have to sign up or not sign up, and they'll probably do the latter

[09:03] Eliezer: "I don't know" is usually just a screen that people think is defensible and unarguable before they go on to do whatever they feel like, and it's usually the wrong thing because they refused to admit to themselves what their guess was, or examine their justifications, or even realize that they're guessing

[09:02] X: how many apples are in a tree outside?

[09:02] X: i've never seen it and neither have you

[09:02] Eliezer: 10 to 1000

[09:04] Eliezer: if you offer to bet me a million dollars against one dollar that the tree outside has fewer than 20 apples, when neither of us have seen it, I will take your bet

[09:04] X: is it better to say "maybe 10 to 1000" to make it clear that you are guessing when talking to people

[09:04] Eliezer: therefore I have assigned a nonzero and significant probability to apples < 20 whether I admit it or not

[09:05] Eliezer: what you *say* is another issue, especially when speaking to nonrationalists, and then it is well to bear in mind that words don't have fixed meanings; the meaning of the sounds that issue from your lips is whatever occurs in the mind of the listener. If they're going to misinterpret something then you shouldn't say it to them no matter what the words mean inside your own head

[09:06] Eliezer: often you are just screwed unless you want to go back and teach them rationality from scratch, and in a case like that, all you can do is say whatever creates the least inaccurate image

[09:06] X: 10 to 1000 is misleading when you say it to a nonrationalist?

[09:06] Eliezer: "I don't know" is a good way to duck when you say it to someone who doesn't know about probability distributions

[09:07] Eliezer: if they thought I was certain, or that my statement implied actual knowledge of the tree

[09:07] Eliezer: then the statement would mislead them

[09:07] Eliezer: and if I knew this, and did it anyway for my own purposes, it would be a lie

[09:08] Eliezer: if I just couldn't think of anything better to say, then it would be honest but not true, if you can see the distinction

[09:08] Eliezer: honest for me, but the statement that formed in their minds would still not be true

[09:09] X: most people will say to you.... but you said....10-1000 apples

[09:09] Eliezer: then you're just screwed

[09:10] Eliezer: nothing you can do will create in their minds a true understanding, not even "I don't know"

[09:10] X: why bother, why not say i don't know?

[09:10] Eliezer: honesty therefore consists of misleading them the least and telling them the most

[09:10] X: it's better than misleading them with 10-1000

[09:10] Eliezer: as for "why bother", well, if you're going to ask that question, just don't reply to their email or whatever

[09:11] Eliezer: what if you're dealing with someone who thinks my saying "I don't know" is a license for them to make up their own ideas, which will be a lot worse?

[09:11] X: they may act on your guess, and then say "but you said...." and lose money or get in trouble or have less respect for you

[09:11] Eliezer: then you choose to wave them off

[09:11] Eliezer: with "I don't know"

[09:11] Eliezer: but it's for your own sake, not for their sake

[09:12] X: [09:11] Eliezer: what if you're dealing with someone who thinks my saying "I don't know" is a license for them to make up their own ideas, which will be a lot worse?

[09:12] X: here i could see why

[09:12] X: but it's difficult working with typical people in the real world

[09:13] Eliezer: the first thing to decide is, are you trying to accomplish something for yourself (like not getting in trouble) or are you trying to improve someone else's picture of reality

[09:13] Eliezer: "I don't know" is often a good way of not getting in trouble

[09:13] Eliezer: as for it being difficult to work with people in the real world, well, yeah

[09:13] X: if you say...10-1000, and you are wrong, and they are mad, then you say, i don't know, they will be even madder

[09:13] Eliezer: are you trying not to get in trouble?

[09:14] Eliezer: or are you trying to improve their picture of reality?

[09:14] Eliezer: these are two different tasks

[09:14] X: especially if they have lost money or have been proven wrong by someone else

[09:14] Eliezer: if they intersect you have to decide what your tradeoff is

[09:14] Eliezer: which is more important to you

[09:14] Eliezer: then decide whether to explain for their benefit or say "I don't know" for yours

[09:15] X: well, if it was my job, i would say i don't know rather than be wrong, because who knows what your boss will do after he loses money listening to you

[09:15] Eliezer: okay

[09:16] Eliezer: just be clear that this is not because "I don't know" is the rational judgment, but because "I don't know" is the political utterance

[09:16] X: he may take your guess, and try to turn it into an actual anwser because no one around you has a better plan

[09:17] Eliezer: you can easily see this by looking at your stated reason: you didn't talk about evidence and reality and truth, but, how you might get in trouble based on someone's reaction

[09:17] X: yes

[09:17] X: that's what you have to put up with in the real world

[09:17] Eliezer: if you're really worried about your boss's welfare then you should consider that if you say "I don't know" he must do something anyway - refusing to choose is also a choice, and refusing to act is like refusing to let time pass - and he will construct that plan based on some information, which doesn't include your information

[09:18] Eliezer: if your life isn't worth more than someone else's, neither it is worth any less, and it is often proper to let fools make their own mistakes

[09:18] Eliezer: you can only throw yourself in front of so many bullets before you run out of flesh to stop them with

[09:19] X: ?

[09:19] Eliezer: in other words, you cannot always save people from themselves

[09:23] Eliezer: but all of this is wandering away from the original point, which is true and correct, that no one is ever entitled to their own opinion

[09:26] X: what is his name?

[09:26] Eliezer: ?

[09:26] X: a man outside

[09:26] X: random guy

[09:26] Eliezer: It's probably not "Xpchtl Vaaaaaarax"

[09:26] X: probably not

[09:27] Eliezer: I suppose I could construct a second-order Markov transition diagram for the letters in names expressed in English, weighted by their frequency

[09:27] Eliezer: but that would be a lot of work

[09:28] Eliezer: so I could say "I don't know" as shorthand for the fact that, although I possess a lot of knowledge about possible and probable names, I don't know anything *more* than you do

[09:28] X: ok, so you say ruling out what you see as likely not correct is ok?

[09:28] Eliezer: what I'm saying is that I possess a large amount of knowledge about possible names

[09:28] Eliezer: all of which influences what I would bet on

[09:28] Eliezer: if I had to take a real-world action, like, guessing someone's name with a gun to my head

[09:29] Eliezer: if I had to choose it would suddenly become very relevant that I knew Michael was one of the most statistically common names, but couldn't remember for which years it was the most common, and that I knew Michael was more likely to be a male name than a female name

[09:29] Eliezer: if an alien had a gun to its head, telling it "I don't know" at this point would not be helpful

[09:29] Eliezer: because there's a whole lot I know that it doesn't

[09:30] X: ok

[09:33] X: what about a question for which you really don't have any information?

[09:33] X: like something only an alien would know

[09:34] Eliezer: if I have no evidence I use an appropriate Ignorance Prior, which distributes probability evenly across all possibilities, and assigns only a very small amount to any individual possibility because there are so many

[09:35] Eliezer: if the person I'm talking to already knows to use an ignorance prior, I say "I don't know" because we already have the same probability distribution and I have nothing to add to that

[09:35] Eliezer: the ignorance prior tells me my betting odds

[09:35] Eliezer: it governs my choices

[09:35] X: and what if you don't know how to use an ignorance prior

[09:36] X: have never heard of it etc

[09:36] Eliezer: if I'm dealing with someone who doesn't know about ignorance priors, and who is dealing with the problem by making up this huge elaborate hypothesis with lots of moving parts and many places to go wrong, then the truth is that I automatically know s/he's wrong

[09:36] Eliezer: it may not be possible to explain this to them, short of training them from scratch in rationality

[09:36] Eliezer: but it is true

[09:36] Eliezer: and if the person trusts me for a rationalist, it may be both honest and helpful to tell them, "No, that's wrong"

[09:36] X: what if that person says, "i don't know what their name is", that ok?

[09:37] Eliezer: in real life you cannot choose "I don't know", it's not an option on your list of available actions

[09:37] Eliezer: in real life it's always, "I don't know, so I'm going to say Vkktor Blackdawn because I think it sounds cool"

[09:39] Eliezer: Vkktor Blackdawn is as (im)probable as anything else, but if you start assigning more probability to it than the ignorance prior calls for - because it sounds cool, because you don't have room in your mind for more than one possibility, or because you've started to construct an elaborate mental explanation of how the alien might end up named Vkktor Blackdawn

[09:39] Eliezer: then I know better

[09:40] Eliezer: and if you trust me, I may be able to honestly and usefully tell you so

[09:40] Eliezer: rather than saying "I don't know", which is always something to say, not to think

[09:40] Eliezer: this is important if someone asks you, "At what odds would you bet that the alien is named Vkktor Blackdawn?"

[09:41] Eliezer: or if you have to do anything else, based on your guesses and the weight you assign to them

[09:41] Eliezer: which is what probability is all about

[09:41] X: and if they say "I don't know, I don't know anything about probability"?

[09:41] Eliezer: then either they trust me blindly or I can't help them

[09:41] Eliezer: that's how it goes

[09:41] Eliezer: you can't always save people from themselves

[09:42] X: trust you blindly about what you are saying or about your guess as to what the alien's name is?

[09:42] Eliezer: trust me blindly when I tell them, "Don't bet at those odds."

Original with comments: "I don't know."

A Fable of Science and Politics

Eliezer Yudkowsky, 23 December 2006 04:50AM

In the time of the Roman Empire, civic life was divided between the Blue and Green factions. The Blues and the Greens murdered each other in single combats, in ambushes, in group battles, in riots. Procopius said of the warring factions: "So there grows up in them against their fellow men a hostility which has no cause, and at no time does it cease or disappear, for it gives place neither to the ties of marriage nor of relationship nor of friendship, and the case is the same even though those who differ with respect to these colors be brothers or any other kin." Edward Gibbon wrote: "The support of a faction became necessary to every candidate for civil or ecclesiastical honors."

Who were the Blues and the Greens? They were sports fans - the partisans of the blue and green chariot-racing teams.

Imagine a future society that flees into a vast underground network of caverns and seals the entrances. We shall not specify whether they flee disease, war, or radiation; we shall suppose the first Undergrounders manage to grow food, find water, recycle air, make light, and survive, and that their descendants thrive and eventually form cities. Of the world above, there are only legends written on scraps of paper; and one of these scraps of paper describes the sky, a vast open space of air above a great unbounded floor. The sky is cerulean in color, and contains strange floating objects like enormous tufts of white cotton. But the meaning of the word "cerulean" is controversial; some say that it refers to the color known as "blue", and others that it refers to the color known as "green".

In the early days of the underground society, the Blues and Greens contested with open violence; but today, truce prevails - a peace born of a growing sense of pointlessness. Cultural mores have changed; there is a large and prosperous middle class that has grown up with effective law enforcement and become unaccustomed to violence. The schools provide some sense of historical perspective; how long the battle between Blues and Greens continued, how many died, how little changed as a result. Minds have been laid open to the strange new philosophy that people are people, whether they be Blue or Green.

The conflict has not vanished. Society is still divided along Blue and Green lines, and there is a "Blue" and a "Green" position on almost every contemporary issue of political or cultural importance. The Blues advocate taxes on individual incomes, the Greens advocate taxes on merchant sales; the Blues advocate stricter marriage laws, while the Greens wish to make it easier to obtain divorces; the Blues take their support from the heart of city areas, while the more distant farmers and watersellers tend to be Green; the Blues believe that the Earth is a huge spherical rock at the center of the universe, the Greens that it is a huge flat rock circling some other object called a Sun. Not every Blue or every Green citizen takes the "Blue" or "Green" position on every issue, but it would be rare to find a city merchant who believed the sky was blue, and yet advocated an individual tax and freer marriage laws.

The Underground is still polarized; an uneasy peace. A few folk genuinely think that Blues and Greens should be friends, and it is now common for a Green to patronize a Blue shop, or for a Blue to visit a Green tavern. Yet from a truce originally born of exhaustion, there is a quietly growing spirit of tolerance, even friendship.

One day, the Underground is shaken by a minor earthquake. A sightseeing party of six is caught in the tremblor while looking at the ruins of ancient dwellings in the upper caverns. They feel the brief movement of the rock under their feet, and one of the tourists trips and scrapes her knee. The party decides to turn back, fearing further earthquakes. On their way back, one person catches a whiff of something strange in the air, a scent coming from a long-unused passageway. Ignoring the well-meant cautions of fellow travellers, the person borrows a powered lantern and walks into the passageway. The stone corridor wends upward... and upward... and finally terminates in a hole carved out of the world, a place where all stone ends. Distance, endless distance, stretches away into forever; a gathering space to hold a thousand cities. Unimaginably far above, too bright to look at directly, a searing spark casts light over all visible space, the naked filament of some huge light bulb. In the air, hanging unsupported, are great incomprehensible tufts of white cotton. And the vast glowing ceiling above... the color... is...

Now history branches, depending on which member of the sightseeing party decided to follow the corridor to the surface.

Aditya the Blue stood under the blue forever, and slowly smiled. It was not a pleasant smile. There was hatred, and wounded pride; it recalled every argument she'd ever had with a Green, every rivalry, every contested promotion. "You were right all along," the sky whispered down at her, "and now you can prove it." For a moment Aditya stood there, absorbing the message, glorying in it, and then she turned back to the stone corridor to tell the world. As Aditya walked, she curled her hand into a clenched fist. "The truce," she said, "is over."

Barron the Green stared incomprehendingly at the chaos of colors for long seconds. Understanding, when it came, drove a pile-driver punch into the pit of his stomach. Tears started from his eyes. Barron thought of the Massacre of Cathay, where a Blue army had massacred every citizen of a Green town, including children; he thought of the ancient Blue general, Annas Rell, who had declared Greens "a pit of disease; a pestilence to be cleansed"; he thought of the glints of hatred he'd seen in Blue eyes and something inside him cracked. "How can you be on their side?" Barron screamed at the sky, and then he began to weep; because he knew, standing under the malevolent blue glare, that the universe had always been a place of evil.

Charles the Blue considered the blue ceiling, taken aback. As a professor in a mixed college, Charles had carefully emphasized that Blue and Green viewpoints were equally valid and deserving of tolerance: The sky was a metaphysical construct, and cerulean a color that could be seen in more than one way. Briefly, Charles wondered whether a Green, standing in this place, might not see a green ceiling above; or if perhaps the ceiling would be green at this time tomorrow; but he couldn't stake the continued survival of civilization on that. This was merely a natural phenomenon of some kind, having nothing to do with moral philosophy or society... but one that might be readily misinterpreted, Charles feared. Charles sighed, and turned to go back into the corridor. Tomorrow he would come back alone and block off the passageway.

Daria, once Green, tried to breathe amid the ashes of her world. I will not flinch, Daria told herself, I will not look away. She had been Green all her life, and now she must be Blue. Her friends, her family, would turn from her. Speak the truth, even if your voice trembles, her father had told her; but her father was dead now, and her mother would never understand. Daria stared down the calm blue gaze of the sky, trying to accept it, and finally her breathing quietened. I was wrong, she said to herself mournfully; it's not so complicated, after all. She would find new friends, and perhaps her family would forgive her... or, she wondered with a tinge of hope, rise to this same test, standing underneath this same sky? "The sky is blue," Daria said experimentally, and nothing dire happened to her; but she couldn't bring herself to smile. Daria the Blue exhaled sadly, and went back into the world, wondering what she would say.

Eddin, a Green, looked up at the blue sky and began to laugh cynically. The course of his world's history came clear at last; even he couldn't believe they'd been such fools. "Stupid," Eddin said, "stupid, stupid, and all the time it was right here." Hatred, murders, wars, and all along it was just a thing somewhere, that someone had written about like they'd write about any other thing. No poetry, no beauty, nothing that any sane person would ever care about, just one pointless thing that had been blown out of all proportion. Eddin leaned against the cave mouth wearily, trying to think of a way to prevent this information from blowing up the world, and wondering if they didn't all deserve it.

Ferris gasped involuntarily, frozen by sheer wonder and delight. Ferris's eyes darted hungrily about, fastening on each sight in turn before moving reluctantly to the next; the blue sky, the white clouds, the vast unknown outside, full of places and things (and people?) that no Undergrounder had ever seen. "Oh, so that's what color it is," Ferris said, and went exploring.

Sequence: Politics is the Mind-Killer

Politics is the Mind-Killer

Referenced by: Politics is the Mind-Killer • The Scales of Justice, the Notebook of Rationality • Blue or Green on Regulation? • Belief in Belief • Professing and Cheering • Belief as Attire • What Evidence Filtered Evidence? • The Meditation on Curiosity • Fake Optimization Criteria • When None Dare Urge Restraint

Original with comments: A Fable of Science and Politics

Some Claims Are Just Too Extraordinary

Eliezer Yudkowsky, 20 January 2007 07:14AM

"I would sooner believe that two Yankee professors would lie, than that stones would fall from heaven."

-- Thomas Jefferson, on meteors

"How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn't. It isn't going to happen."

-- Eliezer Yudkowsky, "A Technical Explanation of Technical Explanation"

"If a ship landed in my yard and LGMs stepped out, I’d push past their literature and try to find the cable that dropped the saucer on my roses. Lack of a cable or any significant burning to the flowers, I’d then grab a hammer and start knocking about in the ship till I was convinced that nothing said “Intel Inside.” Then when I discovered a “Flux Capacitor” type thing I would finally stop and say, “Hey, cool gadget!” Assuming the universal benevolence of the LGMs, I’d yank it out and demand from the nearest "Grey” (they are the tall nice ones), “where the hell did this come from?” Greys don’t talk, they communicate via telepathy, so I’d ignore the voice inside my head. Then stepping outside the saucer and sitting in a lawn chair, I’d throw pebbles at the aliens till I was sure they were solid. Then I’d look down at the “Flux Capacitor” and make sure it hadn’t morphed into my bird feeder. Finally, with proof in my hand and aliens sitting on my deck (they’d be offered beers, though I’ve heard that they absorb energy like a plant) I’d grab my cell phone and tell my doctor that I’m having a serious manic episode with full-blown visual hallucinations."

-- Peter K. Bertine, on the Extropian mailing list

We underestimate the power of science, and overestimate the power of personal observation. A peer-reviewed, journal-published, replicated report is worth far more than what you see with your own eyes. Our own eyes can deceive us. People can fool themselves, hallucinate, and even go insane. The controls on publication in major journals are more trustworthy than the very fabric of your brain. If you see with your own eyes that the sky is blue, and Science says it is green, then sir, I advise that you trust in Science.

This is not what most scientists will tell you, of course; but I think it is pragmatically true. Because in real life, what happens is that your eyes have a little malfunction and decide that the sky is green, and science will tell you that the sky is blue.

A replicated scientific report is a special kind of extraordinary claim, designed by the surrounding process to be more extraordinary evidence than simple verbal claims. It is more extraordinary evidence because the surrounding process - and I would place a far greater premium on the replication than on the peer review, by the way - is constructed to deny entrance to claims that are in fact false. In this way, the replicated scientific report becomes capable of overcoming greater burdens of prior improbability.

There are some burdens of prior improbability so great that simple verbal claims cannot overcome them. I would not believe someone who claimed that their coffee was disobeying conservation of angular momentum - but I might believe the same report published in Physics Today, with at least three replications. Who would believe in quantum mechanics if a stranger walked up to us on the street and whispered it to us?

Are there some burdens of prior improbability so great that science itself cannot overcome them?

What about the claim that 2 + 2 = 5?

What about journals that claim to publish replicated reports of ESP?

Sometimes, even claims deliberately constructed to be extraordinary evidence end up just not being extraordinary enough.

Original with comments: Some Claims Are Just Too Extraordinary

Outside the Laboratory

Eliezer Yudkowsky, 21 January 2007 03:46AM

"Outside the laboratory, scientists are no wiser than anyone else." Sometimes this proverb is spoken by scientists, humbly, sadly, to remind themselves of their own fallibility. Sometimes this proverb is said for rather less praiseworthy reasons, to devalue unwanted expert advice. Is the proverb true? Probably not in an absolute sense. It seems much too pessimistic to say that scientists are literally no wiser than average, that there is literally zero correlation.

But the proverb does appear true to some degree, and I propose that we should be very disturbed by this fact. We should not sigh, and shake our heads sadly. Rather we should sit bolt upright in alarm. Why? Well, suppose that an apprentice shepherd is laboriously trained to count sheep, as they pass in and out of a fold. Thus the shepherd knows when all the sheep have left, and when all the sheep have returned. Then you give the shepherd a few apples, and say: "How many apples?" But the shepherd stares at you blankly, because they weren't trained to count apples - just sheep. You would probably suspect that the shepherd didn't understand counting very well.

Now suppose we discover that a Ph.D. economist buys a lottery ticket every week. We have to ask ourselves: Does this person really understand expected utility, on a gut level? Or have they just been trained to perform certain algebra tricks?

One thinks of Richard Feynman's account of a failing physics education program:

"The students had memorized everything, but they didn't know what anything meant. When they heard 'light that is reflected from a medium with an index', they didn't know that it meant a material such as water. They didn't know that the 'direction of the light' is the direction in which you see something when you're looking at it, and so on. Everything was entirely memorized, yet nothing had been translated into meaningful words. So if I asked, 'What is Brewster's Angle?' I'm going into the computer with the right keywords. But if I say, 'Look at the water,' nothing happens - they don't have anything under 'Look at the water'!"

Suppose we have an apparently competent scientist, who knows how to design an experiment on N subjects; the N subjects will receive a randomized treatment; blinded judges will classify the subject outcomes; and then we'll run the results through a computer and see if the results are significant at the 0.05 confidence level. Now this is not just a ritualized tradition. This is not a point of arbitrary etiquette like using the correct fork for salad. It is a ritualized tradition for testing hypotheses experimentally. Why should you test your hypothesis experimentally? Because you know the journal will demand so before it publishes your paper? Because you were trained to do it in college? Because everyone else says in unison that it's important to do the experiment, and they'll look at you funny if you say otherwise?

No: because, in order to map a territory, you have to go out and look at the territory. It isn't possible to produce an accurate map of a city while sitting in your living room with your eyes closed, thinking pleasant thoughts about what you wish the city was like. You have to go out, walk through the city, and write lines on paper that correspond to what you see. It happens, in miniature, every time you look down at your shoes to see if your shoelaces are untied. Photons arrive from the Sun, bounce off your shoelaces, strike your retina, are transduced into neural firing frequences, and are reconstructed by your visual cortex into an activation pattern that is strongly correlated with the current shape of your shoelaces. To gain new information about the territory, you have to interact with the territory. There has to be some real, physical process whereby your brain state ends up correlated to the state of the environment. Reasoning processes aren't magic; you can give causal descriptions of how they work. Which all goes to say that, to find things out, you've got to go look.

Now what are we to think of a scientist who seems competent inside the laboratory, but who, outside the laboratory, believes in a spirit world? We ask why, and the scientist says something along the lines of: "Well, no one really knows, and I admit that I don't have any evidence - it's a religious belief, it can't be disproven one way or another by observation." I cannot but conclude that this person literally doesn't know why you have to look at things. They may have been taught a certain ritual of experimentation, but they don't understand the reason for it - that to map a territory, you have to look at it - that to gain information about the environment, you have to undergo a causal process whereby you interact with the environment and end up correlated to it. This applies just as much to a double-blind experimental design that gathers information about the efficacy of a new medical device, as it does to your eyes gathering information about your shoelaces.

Maybe our spiritual scientist says: "But it's not a matter for experiment. The spirits spoke to me in my heart." Well, if we really suppose that spirits are speaking in any fashion whatsoever, that is a causal interaction and it counts as an observation. Probability theory still applies. If you propose that some personal experience of "spirit voices" is evidence for actual spirits, you must propose that there is a favorable likelihood ratio for spirits causing "spirit voices", as compared to other explanations for "spirit voices", which is sufficient to overcome the prior improbability of a complex belief with many parts. Failing to realize that "the spirits spoke to me in my heart" is an instance of "causal interaction", is analogous to a physics student not realizing that a "medium with an index" means a material such as water.

It is easy to be fooled, perhaps, by the fact that people wearing lab coats use the phrase "causal interaction" and that people wearing gaudy jewelry use the phrase "spirits speaking". Discussants wearing different clothing, as we all know, demarcate independent spheres of existence - "separate magisteria", in Stephen J. Gould's immortal blunder of a phrase. Actually, "causal interaction" is just a fancy way of saying, "Something that makes something else happen", and probability theory doesn't care what clothes you wear.

In modern society there is a prevalent notion that spiritual matters can't be settled by logic or observation, and therefore you can have whatever religious beliefs you like. If a scientist falls for this, and decides to live their extralaboratorial life accordingly, then this, to me, says that they only understand the experimental principle as a social convention. They know when they are expected to do experiments and test the results for statistical significance. But put them in a context where it is socially conventional to make up wacky beliefs without looking, and they just as happily do that instead.

The apprentice shepherd is told that if "seven" sheep go out, and "eight" sheep go out, then "fifteen" sheep had better come back in. Why "fifteen" instead of "fourteen" or "three"? Because otherwise you'll get no dinner tonight, that's why! So that's professional training of a kind, and it works after a fashion - but if social convention is the only reason why seven sheep plus eight sheep equals fifteen sheep, then maybe seven apples plus eight apples equals three apples. Who's to say that the rules shouldn't be different for apples?

But if you know why the rules work, you can see that addition is the same for sheep and for apples. Isaac Newton is justly revered, not for his outdated theory of gravity, but for discovering that - amazingly, surprisingly - the celestial planets, in the glorious heavens, obeyed just the same rules as falling apples. In the macroscopic world - the everyday ancestral environment - different trees bear different fruits, different customs hold for different people at different times. A genuinely unified universe, with stationary universal laws, is a highly counterintuitive notion to humans! It is only scientists who really believe it, though some religions may talk a good game about the "unity of all things".

As Richard Feynman put it:

"If we look at a glass closely enough we see the entire universe. There are the things of physics: the twisting liquid which evaporates depending on the wind and weather, the reflections in the glass, and our imaginations adds the atoms. The glass is a distillation of the Earth's rocks, and in its composition we see the secret of the universe's age, and the evolution of the stars. What strange array of chemicals are there in the wine? How did they come to be? There are the ferments, the enzymes, the substrates, and the products. There in wine is found the great generalization: all life is fermentation. Nobody can discover the chemistry of wine without discovering, as did Louis Pasteur, the cause of much disease. How vivid is the claret, pressing its existence into the consciousness that watches it! If our small minds, for some convenience, divide this glass of wine, this universe, into parts — physics, biology, geology, astronomy, psychology, and so on — remember that Nature does not know it! So let us put it all back together, not forgetting ultimately what it is for. Let it give us one more final pleasure: drink it and forget it all!"

A few religions, especially the ones invented or refurbished after Isaac Newton, may profess that "everything is connected to everything else". (Since there is a trivial isomorphism between graphs and their complements, this profound wisdom conveys exactly the same useful information as a graph with no edges.) But when it comes to the actual meat of the religion, prophets and priests follow the ancient human practice of making everything up as they go along. And they make up one rule for women under twelve, another rule for men over thirteen; one rule for the Sabbath and another rule for weekdays; one rule for science and another rule for sorcery...

Reality, we have learned to our shock, is not a collection of separate magisteria, but a single unified process governed by mathematically simple low-level rules. Different buildings on a university campus do not belong to different universes, though it may sometimes seem that way. The universe is not divided into mind and matter, or life and nonlife; the atoms in our heads interact seamlessly with the atoms of the surrounding air. Nor is Bayes's Theorem different from one place to another.

If, outside of their specialist field, some particular scientist is just as susceptible as anyone else to wacky ideas, then they probably never did understand why the scientific rules work. Maybe they can parrot back a bit of Popperian falsificationism; but they don't understand on a deep level, the algebraic level of probability theory, the causal level of cognition-as-machinery. They've been trained to behave a certain way in the laboratory, but they don't like to be constrained by evidence; when they go home, they take off the lab coat and relax with some comfortable nonsense. And yes, that does make me wonder if I can trust that scientist's opinions even in their own field - especially when it comes to any controversial issue, any open question, anything that isn't already nailed down by massive evidence and social convention.

Maybe we can beat the proverb - be rational in our personal lives, not just our professional lives. We shouldn't let a mere proverb stop us: "A witty saying proves nothing," as Voltaire said. Maybe we can do better, if we study enough probability theory to know why the rules work, and enough experimental psychology to see how they apply in real-world cases - if we can learn to look at the water. An ambition like that lacks the comfortable modesty of being able to confess that, outside your specialty, you're no better than anyone else. But if our theories of rationality don't generalize to everyday life, we're doing something wrong. It's not a different universe inside and outside the laboratory.

Addendum: If you think that (a) science is purely logical and therefore opposed to emotion, or (b) that we shouldn't bother to seek truth in everyday life, see "Why Truth?" For new readers, I also recommend "Twelve Virtues of Rationality."

Referenced by: Think Like Reality • The Lens That Sees Its Flaws • Guardians of the Truth • Do Scientists Already Know This Stuff? • My Childhood Role Model • Against Devil's Advocacy • Excluding the Supernatural • Above-Average AI Scientists • Raising the Sanity Waterline • That Magical Click

Original with comments: Outside the Laboratory

Politics is the Mind-Killer

Eliezer Yudkowsky, 18 February 2007 09:23PM

People go funny in the head when talking about politics. The evolutionary reasons for this are so obvious as to be worth belaboring: In the ancestral environment, politics was a matter of life and death. And sex, and wealth, and allies, and reputation... When, today, you get into an argument about whether "we" ought to raise the minimum wage, you're executing adaptations for an ancestral environment where being on the wrong side of the argument could get you killed. Being on the right side of the argument could let you kill your hated rival!

If you want to make a point about science, or rationality, then my advice is to not choose a domain from contemporary politics if you can possibly avoid it. If your point is inherently about politics, then talk about Louis XVI during the French Revolution. Politics is an important domain to which we should individually apply our rationality - but it's a terrible domain in which to learn rationality, or discuss rationality, unless all the discussants are already rational.

Politics is an extension of war by other means. Arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back - providing aid and comfort to the enemy. People who would be level-headed about evenhandedly weighing all sides of an issue in their professional life as scientists, can suddenly turn into slogan-chanting zombies when there's a Blue or Green position on an issue.

In Artificial Intelligence, and particularly in the domain of nonmonotonic reasoning, there's a standard problem: "All Quakers are pacifists. All Republicans are not pacifists. Nixon is a Quaker and a Republican. Is Nixon a pacifist?"

What on Earth was the point of choosing this as an example? To rouse the political emotions of the readers and distract them from the main question? To make Republicans feel unwelcome in courses on Artificial Intelligence and discourage them from entering the field? (And no, before anyone asks, I am not a Republican. Or a Democrat.)

Why would anyone pick such a distracting example to illustrate nonmonotonic reasoning? Probably because the author just couldn't resist getting in a good, solid dig at those hated Greens. It feels so good to get in a hearty punch, y'know, it's like trying to resist a chocolate cookie.

As with chocolate cookies, not everything that feels pleasurable is good for you. And it certainly isn't good for our hapless readers who have to read through all the angry comments your blog post inspired.

I'm not saying that I think Overcoming Bias should be apolitical, or even that we should adopt Wikipedia's ideal of the Neutral Point of View. But try to resist getting in those good, solid digs if you can possibly avoid it. If your topic legitimately relates to attempts to ban evolution in school curricula, then go ahead and talk about it - but don't blame it explicitly on the whole Republican Party; some of your readers may be Republicans, and they may feel that the problem is a few rogues, not the entire party. As with Wikipedia's NPOV, it doesn't matter whether (you think) the Republican Party really is at fault. It's just better for the spiritual growth of the community to discuss the issue without invoking color politics.

(Now that I've been named as a co-moderator, I guess I'd better include a disclaimer: This article is my personal opinion, not a statement of official Overcoming Bias policy. This will always be the case unless explicitly specified otherwise.)

Sequence: Politics is the Mind-Killer

A Fable of Science and Politics

Policy Debates Should Not Appear One-Sided

Referenced by: Policy Debates Should Not Appear One-Sided • Blue or Green on Regulation? • The Third Alternative • Are Your Enemies Innately Evil? • Update Yourself Incrementally • What Evidence Filtered Evidence? • The Wonder of Evolution • Fake Optimization Criteria • Resist the Happy Death Spiral • Uncritical Supercriticality • Misc Meta • Dark Side Epistemology • Dunbar's Function • Of Exclusionary Speech and Gender Politics

Original with comments: Politics is the Mind-Killer

Just Lose Hope Already

Eliezer Yudkowsky, 25 February 2007 12:39AM

Casey Serin, a 24-year-old web programmer with no prior experience in real estate, owes banks 2.2 million dollars after lying on mortgage applications in order to simultaneously buy 8 different houses in different states. He took cash out of the mortgage (applied for larger amounts than the price of the house) and spent the money on living expenses and real-estate seminars. He was expecting the market to go up, it seems.

That's not even the sad part. The sad part is that he still hasn't given up. Casey Serin does not accept defeat. He refuses to declare bankruptcy, or get a job; he still thinks he can make it big in real estate. He went on spending money on seminars. He tried to take out a mortgage on a 9th house. He hasn't failed, you see, he's just had a learning experience.

That's what happens when you refuse to lose hope.

While this behavior may seem to be merely stupid, it also puts me in mind of two Nobel-Prize-winning economists...

...namely Merton and Scholes of Long-Term Capital Management.

While LTCM raked in giant profits over its first three years, in 1998 the inefficiences that LTCM were exploiting had started to vanish - other people knew about the trick, so it stopped working.

LTCM refused to lose hope. Addicted to 40% annual returns, they borrowed more and more leverage to exploit tinier and tinier margins. When everything started to go wrong for LTCM, they had equity of $4.72 billion, leverage of $124.5 billion, and derivative positions of $1.25 trillion.

Every profession has a different way to be smart - different skills to learn and rules to follow. You might therefore think that the study of "rationality", as a general discipline, wouldn't have much to contribute to real-life success. And yet it seems to me that how to not be stupid has a great deal in common across professions. If you set out to teach someone how to not turn little mistakes into big mistakes, it's nearly the same art whether in hedge funds or romance, and one of the keys is this: Be ready to admit you lost.

Sequence: Letting Go

The Crackpot Offer

The Proper Use of Doubt

Referenced by: Debiasing as Non-Self-Destruction • On Doing the Impossible • ...And Say No More Of It • Unteachable Excellence • Unspeakable Morality

Original with comments: Just Lose Hope Already

You Are Not Hiring the Top 1%

Eliezer Yudkowsky, 02 March 2007 05:34AM

Today's statistical fallacy (slightly redacted by editor) comes from Joel on Software:

Everyone thinks they're hiring the top 1%. Martin Fowler said, "We are still working hard to hire only the very top fraction of software developers (the target is around the top 0.5 to 1%)." I hear this from almost every software company. "We hire the top 1% or less," they all say. Could they all be hiring the top 1%? Where are all the other 99%? General Motors?

When you get 200 resumes, and hire the best person, does that mean you're hiring the top 0.5%? Think about what happens to the other 199 that you didn't hire. They go look for another job.

The entire world could consist of 1,000,000 programmers, of whom the worst 199 keep applying for every job and never getting them, but the best 999,801 always get jobs as soon as they apply for one. So every time a job is listed the 199 losers apply, as usual, and one guy from the pool of 999,801 applies, and he gets the job, of course, because he's the best, and now, in this contrived example, every employer thinks they're getting the top 0.5% when they're actually getting the top 99.9801%.

I'm exaggerating a lot, but the point is, when you select 1 out of 200 applicants, the other 199 don't give up and go into plumbing (although I wish they would... plumbers are impossible to find). They apply again somewhere else, and contribute to some other employer's self-delusions about how selective they are.

This might explain some other phenomena I've heard of, such as the "slush heap" of inconceivably awful stories received in the mail by every fiction publisher that accepts unsolicited manuscripts. When a story is good enough to be published, it's accepted and removed from the system. Otherwise the hapless author sends it to another publisher!

Perhaps most novice writers aren't quite as dreadful (on average) as editors seem to believe? Editors will disproportionately encounter the work of novice writers who are not only awkward, but so incompetent as to be unaware of their own incompetence, and so proud that they can't take a hint.

PS: Two other areas where this might apply: Students who apply to your program/department for admission. And, grant proposals.

PPS: Robert Scarth comments, "This might also explain why some women often think that 'all men are bastards' - there are a few cads out there in continual circulation and with no intention to settle down, whereas men with honourable intentions have much fewer relationships, and settle down more quickly."

Original with comments: You Are Not Hiring the Top 1%

Policy Debates Should Not Appear One-Sided

Eliezer Yudkowsky, 03 March 2007 06:53PM

Robin Hanson recently proposed stores where banned products could be sold. There are a number of excellent arguments for such a policy - an inherent right of individual liberty, the career incentive of bureaucrats to prohibit everything, legislators being just as biased as individuals. But even so (I replied), some poor, honest, not overwhelmingly educated mother of 5 children is going to go into these stores and buy a "Dr. Snakeoil's Sulfuric Acid Drink" for her arthritis and die, leaving her orphans to weep on national television.

I was just making a simple factual observation. Why did some people think it was an argument in favor of regulation?

On questions of simple fact (for example, whether Earthly life arose by natural selection) there's a legitimate expectation that the argument should be a one-sided battle; the facts themselves are either one way or another, and the so-called "balance of evidence" should reflect this. Indeed, under the Bayesian definition of evidence, "strong evidence" is just that sort of evidence which we only expect to find on one side of an argument.

But there is no reason for complex actions with many consequences to exhibit this onesidedness property. Why do people seem to want their policy debates to be one-sided?

Politics is the mind-killer. Arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back. If you abide within that pattern, policy debates will also appear one-sided to you - the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary.

One should also be aware of a related failure pattern, thinking that the course of Deep Wisdom is to compromise with perfect evenness between whichever two policy positions receive the most airtime. A policy may legitimately have lopsided costs or benefits. If policy questions were not tilted one way or the other, we would be unable to make decisions about them. But there is also a human tendency to deny all costs of a favored policy, or deny all benefits of a disfavored policy; and people will therefore tend to think policy tradeoffs are tilted much further than they actually are.

If you allow shops that sell otherwise banned products, some poor, honest, poorly educated mother of 5 kids is going to buy something that kills her. This is a prediction about a factual consequence, and as a factual question it appears rather straightforward - a sane person should readily confess this to be true regardless of which stance they take on the policy issue. You may also think that making things illegal just makes them more expensive, that regulators will abuse their power, or that her individual freedom trumps your desire to meddle with her life. But, as a matter of simple fact, she's still going to die.

We live in an unfair universe. Like all primates, humans have strong negative reactions to perceived unfairness; thus we find this fact stressful. There are two popular methods of dealing with the resulting cognitive dissonance. First, one may change one's view of the facts - deny that the unfair events took place, or edit the history to make it appear fair. Second, one may change one's morality - deny that the events are unfair.

Some libertarians might say that if you go into a "banned products shop", passing clear warning labels that say "THINGS IN THIS STORE MAY KILL YOU", and buy something that kills you, then it's your own fault and you deserve it. If that were a moral truth, there would be no downside to having shops that sell banned products. It wouldn't just be a net benefit, it would be a one-sided tradeoff with no drawbacks.

Others argue that regulators can be trained to choose rationally and in harmony with consumer interests; if those were the facts of the matter then (in their moral view) there would be no downside to regulation.

Like it or not, there's a birth lottery for intelligence - though this is one of the cases where the universe's unfairness is so extreme that many people choose to deny the facts. The experimental evidence for a purely genetic component of 0.6-0.8 is overwhelming, but even if this were to be denied, you don't choose your parental upbringing or your early schools either.

I was raised to believe that denying reality is a moral wrong. If I were to engage in wishful optimism about how Sulfuric Acid Drink was likely to benefit me, I would be doing something that I was warned against and raised to regard as unacceptable. Some people are born into environments - we won't discuss their genes, because that part is too unfair - where the local witch doctor tells them that it is right to have faith and wrong to be skeptical. In all goodwill, they follow this advice and die. Unlike you, they weren't raised to believe that people are responsible for their individual choices to follow society's lead. Do you really think you're so smart that you would have been a proper scientific skeptic even if you'd been born in 500 C.E.? Yes, there is a birth lottery, no matter what you believe about genes.

Saying "People who buy dangerous products deserve to get hurt!" is not tough-minded. It is a way of refusing to live in an unfair universe. Real tough-mindedness is saying, "Yes, sulfuric acid is a horrible painful death, and no, that mother of 5 children didn't deserve it, but we're going to keep the shops open anyway because we did this cost-benefit calculation." Can you imagine a politician saying that? Neither can I. But insofar as economists have the power to influence policy, it might help if they could think it privately - maybe even say it in journal articles, suitably dressed up in polysyllabismic obfuscationalization so the media can't quote it.

I don't think that when someone makes a stupid choice and dies, this is a cause for celebration. I count it as a tragedy. It is not always helping people, to save them from the consequences of their own actions; but I draw a moral line at capital punishment. If you're dead, you can't learn from your mistakes.

Unfortunately the universe doesn't agree with me. We'll see which one of us is still standing when this is over.

Sequence: Politics is the Mind-Killer

Politics is the Mind-Killer

The Scales of Justice, the Notebook of Rationality

Referenced by: Burch's Law • Superstimuli and the Collapse of Western Civilization • The Majority Is Always Wrong • Debiasing as Non-Self-Destruction • Are Your Enemies Innately Evil? • Update Yourself Incrementally • Leaky Generalizations • When None Dare Urge Restraint • Absolute Authority • Against Devil's Advocacy • Is Morality Given? • Fighting a Rearguard Action Against the Truth • Crisis of Faith • Traditional Capitalist Values • Living By Your Own Strength • Harmful Options • That Crisis thing seems pretty useful

Original with comments: Policy Debates Should Not Appear One-Sided

Burch's Law

Eliezer Yudkowsky, 08 March 2007 02:39AM

Greg Burch said:

"I think people should have a right to be stupid and, if they have that right, the market's going to respond by supplying as much stupidity as can be sold."

Greg Burch was speaking about sport-utility vehicles, which he feels are very poorly designed. Note that Burch was not advocating banning SUVs. Burch did not even advocate regulating SUVs. Burch thinks people should have a right to be stupid. But Burch also openly acknowledges the real-world consequence of that right, which is that the market will respond by supplying as much stupidity as can be sold. Perhaps Burch is strongly libertarian, and sees the case against regulation as a slam-dunk regardless of the consequences, and therefore has an easier time acknowledging the downside of his policy. Or perhaps Burch is just a skillful rationalist. Either way, I hereby canonize his observation as Burch's Law.

Burch's Law is a special case of a more general rule: Just because your ethics require an action doesn't mean the universe will exempt you from the consequences. If the universe were fair, like a sympathetic human, the universe would understand that you had overriding ethical reasons for your action, and would exempt you from the usual penalties. The judge would rule "justifiable homicide" instead of "murder" and exempt you from the usual prison term. Well, the universe isn't fair and it won't exempt you from the consequences. We know the equations of physics in enough detail to know that the equations don't contain any quantities reflective of ethical considerations.

We don't send automobile manufacturers to jail, even though manufactured cars kill an estimated 1.2 million people per year worldwide. (Roughly 2% of the annual planetary death rate.) Not everyone who dies in an automobile accident is someone who decided to drive a car. The tally of casualties includes pedestrians. It includes minor children who had to be pushed screaming into the car on the way to school. And yet we still manufacture automobiles, because, well, we're in a hurry. I don't even disagree with this decision. I drive a car myself. The point is that the consequences don't change no matter how good the ethical justification sounds. The people who die in automobile accidents are still dead. We can suspend the jail penalty, but we can't suspend the laws of physics.

Humanity hasn't had much luck suspending the laws of economics, either. If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold.

Referenced by: Blue or Green on Regulation? • Superstimuli and the Collapse of Western Civilization

Original with comments: Burch's Law

The Scales of Justice, the Notebook of Rationality

Eliezer Yudkowsky, 13 March 2007 04:00PM

Lady Justice is widely depicted as carrying a scales. A scales has the property that whatever pulls one side down, pushes the other side up. This makes things very convenient and easy to track. It's also usually a gross distortion.

In human discourse there is a natural tendency to treat discussion as a form of combat, an extension of war, a sport; and in sports you only need to keep track of how many points have been scored by each team. There are only two sides, and every point scored against one side, is a point in favor of the other. Everyone in the audience keeps a mental running count of how many points each speaker scores against the other. At the end of the debate, the speaker who has scored more points is, obviously, the winner; so everything he says must be true, and everything the loser says must be wrong.

"The Affect Heuristic in Judgments of Risks and Benefits" studied whether subjects mixed up their judgments of the possible benefits of a technology (e.g. nuclear power), and the possible risks of that technology, into a single overall good or bad feeling about the technology. Suppose that I first tell you that a particular kind of nuclear reactor generates less nuclear waste than competing reactor designs. But then I tell you that the reactor is more unstable than competing designs, with a greater danger of undergoing meltdown if a sufficiently large number of things go wrong simultaneously.

If the reactor is more likely to melt down, this seems like a 'point against' the reactor, or a 'point against' someone who argues for building the reactor. And if the reactor produces less waste, this is a 'point for' the reactor, or a 'point for' building it. So are these two facts opposed to each other? No. In the real world, no. These two facts may be cited by different sides of the same debate, but they are logically distinct; the facts don't know whose side they're on. The amount of waste produced by the reactor arises from physical properties of that reactor design. Other physical properties of the reactor make the nuclear reaction more unstable. Even if some of the same design properties are involved, you have to separately consider the probability of meltdown, and the expected annual waste generated. These are two different physical questions with two different factual answers.

But studies such as the above show that people tend to judge technologies - and many other problems - by an overall good or bad feeling. If you tell people a reactor design produces less waste, they rate its probability of meltdown as lower. This means getting the wrong answer to physical questions with definite factual answers, because you have mixed up logically distinct questions - treated facts like human soldiers on different sides of a war, thinking that any soldier on one side can be used to fight any soldier on the other side.

A scales is not wholly inappropriate for Lady Justice if she is investigating a strictly factual question of guilt or innocence. Either John Smith killed John Doe, or not. We are taught (by E. T. Jaynes) that all Bayesian evidence consists of probability flows between hypotheses; there is no such thing as evidence that "supports" or "contradicts" a single hypothesis, except insofar as other hypotheses do worse or better. So long as Lady Justice is investigating a single, strictly factual question with a binary answer space, a scales would be an appropriate tool. If Justitia must consider any more complex issue, she should relinquish her scales or relinquish her sword.

Not all arguments reduce to mere up or down. Lady Rationality carries a notebook, wherein she writes down all the facts that aren't on anyone's side.

Sequence: Politics is the Mind-Killer

Policy Debates Should Not Appear One-Sided

Correspondence Bias

Referenced by: Blue or Green on Regulation? • What Evidence Filtered Evidence?

Original with comments: The Scales of Justice, the Notebook of Rationality

Blue or Green on Regulation?

Eliezer Yudkowsky, 15 March 2007 06:04PM

In recent posts, I have predicted that, if not otherwise prevented from doing so, some people will behave stupidly and suffer the consequences: "If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold." People misinterpret this as indicating that I take a policy stance in favor of regulation. It indicates no such thing. It is meant purely as guess about empirical consequences - a testable prediction on a question of simple fact.

Perhaps I would be less misinterpreted if I also told "the other side of the story" - inveighed at length about the reasons why bureaucrats are not perfect rationalists guarding our net best interests. But ideally, I shouldn't have to go to such lengths. Ideally, I could make a prediction about a strictly factual question without this being interpreted as a policy stance, or as a stance on logically distinct factual questions.

Yet it would appear that there are two and only two sides to the issue - pro-regulation and anti-regulation. All arguments are either allied soldiers or enemy soldiers; they fight on one side or the other. Any allied soldier can be deployed to fight any enemy soldier and vice versa. Whatever argument pushes one side up, pushes the other side down.

I understand that there are continuing fights about regulation, that this battle is viewed as important, and that people caught up in such battle may not want to let a pro-Green point go past without parrying with a Blue counterpoint. But these battle reflexes have developed too far. If I remark that victims of car accidents include minor children who had to be pushed screaming into the car on the way to school, anyone who is anti-regulation instantly suspects me of trying to pull out an emotional trump card. But I was not trying to get cars banned. I was trying to make a point about how emotional trump cards fail to trump the universe.

I have previously predicted on the strictly factual matter of whether, in the absence of regulation, people will get hurt. (Yes.) I have also indicated as a matter of moral judgment that I do not think they deserve to get hurt, because being stupid is not the same as being malicious. Furthermore there are such things as minor children and pedestrians.

I shouldn't have to say this, but apparently I do, so, for the record, here is "the other side of the story":

The FDA prevents 5,000 casualties per year but causes at least 20,000-120,000 casualties by delaying approval of beneficial medications. The second number is calculated only by looking at delays in the introduction of medications eventually approved - not medications never approved, or medications for which approval was never sought. FDA fatalities are comparable to the annual number of fatal car accidents, but the noneffects of medications not approved don't make the evening news. A bureaucrat's chief incentive is not to approve anything that will ever harm anyone in a way that makes it into the newspaper; no other cost-benefit calculus is involved as an actual career incentive. The bureaucracy as a whole may have an incentive to approve at least some new products - if the FDA never approved a new medication, Congress would become suspicious - but any individual bureaucrat has an unlimited incentive to say no. Regulators have no career motive to do any sort of cost-benefit calculation - except of course for the easy career-benefit calculation. A product with a failure mode spectacular enough to make the newspapers will be banned regardless of what other good it might do; one-reason decisionmaking. As with the FAA banning toenail clippers on planes, "safety precautions" are primarily an ostentatious display of costly efforts so that, when a catastrophe does occur, the agency will be seen to have tried its hardest.

Government = ordinary human fallibility + poor incentives + organizational overhead + guns.

But this does not change the consequences of nonregulation. Children will still die horrible deaths in car accidents and they still will not deserve it.

I understand that debates are not conducted in front of perfectly rational audiences. We all know what happens when you try to trade off a sacred value against a nonsacred value. It's why, when someone says, "But if you don't ban cars, people will die in car crashes!" you don't say "Yes, people will die horrible flaming deaths and they don't deserve it. But it's worth it so I don't have to walk to work in the morning." Instead you say, "How dare you take away our freedom to drive? We'll decide for ourselves; we're just as good at making decisions as you are." So go ahead and say that, then. But think to yourself, in the silent privacy of your thoughts if you must: And yet they will still die, and they will not deserve it.

That way, when Sebastian Thrun comes up with a scheme to automate the highways, and claims it will eliminate nearly all traffic accidents, you can pay appropriate attention.

So too with those other horrible consequences of stupidity that I may dwell upon in later posts. Just because (you believe) regulation may not be able to solve these problems, doesn't mean we wouldn't be very interested in a proposal to solve them by other means.

People are hurt by free markets, just as they're hurt by automobiles - torn up by huge powerful mindless machines with imperfect human operators. It may not be the course of wisdom to fix these problems by resorting to the blunt sledgehammer of ban-the-bad-thing, by wishing to the fairy godmother of government and her magic wand of law. But then people will still get hurt. They will lose their jobs, lose their pensions, lose their health insurance, be ground down to bloody stumps by poverty, perhaps die, and they won't deserve it either.

So am I Blue or Green on regulation, then? I consider myself neither. Imagine, for a moment, that much of what the Greens said about the downside of the Blue policy was true - that, left to the mercy of the free market, many people would be crushed by powers far beyond their understanding, nor would they deserve it. And imagine that most of what the Blues said about the downside of the Green policy was also true - that regulators were fallible humans with poor incentives, whacking on delicately balanced forces with a sledgehammer.

Close your eyes and imagine it. Extrapolate the result. If that were true, then... then you'd have a big problem and no easy way to fix it, that's what you'd have. Does this universe look familiar?

Referenced by: Superstimuli and the Collapse of Western Civilization • Useless Medical Disclaimers

Original with comments: Blue or Green on Regulation?

Superstimuli and the Collapse of Western Civilization

Eliezer Yudkowsky, 16 March 2007 06:10PM

At least three people have died playing online games for days without rest. People have lost their spouses, jobs, and children to World of Warcraft. If people have the right to play video games - and it's hard to imagine a more fundamental right - then the market is going to respond by supplying the most engaging video games that can be sold, to the point that exceptionally engaged consumers are removed from the gene pool.

How does a consumer product become so involving that, after 57 hours of using the product, the consumer would rather use the product for one more hour than eat or sleep? (I suppose one could argue that the consumer makes a rational decision that they'd rather play Starcraft for the next hour than live out the rest of their lives, but let's just not go there. Please.)

A candy bar is a superstimulus: it contains more concentrated sugar, salt, and fat than anything that exists in the ancestral environment. A candy bar matches taste buds that evolved in a hunter-gatherer environment, but it matches those taste buds much more strongly than anything that actually existed in the hunter-gatherer environment. The signal that once reliably correlated to healthy food has been hijacked, blotted out with a point in tastespace that wasn't in the training dataset - an impossibly distant outlier on the old ancestral graphs. Tastiness, formerly representing the evolutionarily identified correlates of healthiness, has been reverse-engineered and perfectly matched with an artificial substance. Unfortunately there's no equally powerful market incentive to make the resulting food item as healthy as it is tasty. We can't taste healthfulness, after all.

The now-famous Dove Evolution video shows the painstaking construction of another superstimulus: an ordinary woman transformed by makeup, careful photography, and finally extensive Photoshopping, into a billboard model - a beauty impossible, unmatchable by human women in the unretouched real world. Actual women are killing themselves (e.g. supermodels using cocaine to keep their weight down) to keep up with competitors that literally don't exist.

And likewise, a video game can be so much more engaging than mere reality, even through a simple computer monitor, that someone will play it without food or sleep until they literally die. I don't know all the tricks used in video games, but I can guess some of them - challenges poised at the critical point between ease and impossibility, intermittent reinforcement, feedback showing an ever-increasing score, social involvement in massively multiplayer games.

Is there a limit to the market incentive to make video games more engaging? You might hope there'd be no incentive past the point where the players lose their jobs; after all, they must be able to pay their subscription fee. This would imply a "sweet spot" for the addictiveness of games, where the mode of the bell curve is having fun, and only a few unfortunate souls on the tail become addicted to the point of losing their jobs. As of 2007, playing World of Warcraft for 58 hours straight until you literally die is still the exception rather than the rule. But video game manufacturers compete against each other, and if you can make your game 5% more addictive, you may be able to steal 50% of your competitor's customers. You can see how this problem could get a lot worse.

If people have the right to be tempted - and that's what free will is all about - the market is going to respond by supplying as much temptation as can be sold. The incentive is to make your stimuli 5% more tempting than those of your current leading competitors. This continues well beyond the point where the stimuli become ancestrally anomalous superstimuli. Consider how our standards of product-selling feminine beauty have changed since the advertisements of the 1950s. And as candy bars demonstrate, the market incentive also continues well beyond the point where the superstimulus begins wreaking collateral damage on the consumer.

So why don't we just say no? A key assumption of free-market economics is that, in the absence of force and fraud, people can always refuse to engage in a harmful transaction. (To the extent this is true, a free market would be, not merely the best policy on the whole, but a policy with few or no downsides.)

An organism that regularly passes up food will die, as some video game players found out the hard way. But, on some occasions in the ancestral environment, a typically beneficial (and therefore tempting) act may in fact be harmful. Humans, as organisms, have an unusually strong ability to perceive these special cases using abstract thought. On the other hand we also tend to imagine lots of special-case consequences that don't exist, like ancestor spirits commanding us not to eat perfectly good rabbits.

Evolution seems to have struck a compromise, or perhaps just aggregated new systems on top of old. Homo sapiens are still tempted by food, but our oversized prefrontal cortices give us a limited ability to resist temptation. Not unlimited ability - our ancestors with too much willpower probably starved themselves to sacrifice to the gods, or failed to commit adultery one too many times. The video game players who died must have exercised willpower (in some sense) to keep playing for so long without food or sleep; the evolutionary hazard of self-control.

Resisting any temptation takes conscious expenditure of an exhaustible supply of mental energy. It is not in fact true that we can "just say no" - not just say no, without cost to ourselves. Even humans who won the birth lottery for willpower or foresightfulness still pay a price to resist temptation. The price is just more easily paid.

Our limited willpower evolved to deal with ancestral temptations; it may not operate well against enticements beyond anything known to hunter-gatherers. Even where we successfully resist a superstimulus, it seems plausible that the effort required would deplete willpower much faster than resisting ancestral temptations.

Is public display of superstimuli a negative externality, even to the people who say no? Should we ban chocolate cookie ads, or storefronts that openly say "Ice Cream"?

Just because a problem exists doesn't show (without further justification and a substantial burden of proof) that the government can fix it. The regulator's career incentive does not focus on products that combine low-grade consumer harm with addictive superstimuli; it focuses on products with failure modes spectacular enough to get into the newspaper. Conversely, just because the government may not be able to fix something, doesn't mean it isn't going wrong.

I leave you with a final argument from fictional evidence: Simon Funk's online novel After Life depicts (among other plot points) the planned extermination of biological Homo sapiens - not by marching robot armies, but by artificial children that are much cuter and sweeter and more fun to raise than real children. Perhaps the demographic collapse of advanced societies happens because the market supplies ever-more-tempting alternatives to having children, while the attractiveness of changing diapers remains constant over time. Where are the advertising billboards that say "BREED"? Who will pay professional image consultants to make arguing with sullen teenagers seem more alluring than a vacation in Tahiti?

"In the end," Simon Funk wrote, "the human species was simply marketed out of existence."

Referenced by: Lotteries: A Waste of Hope • New Improved Lottery • Adaptation-Executers, not Fitness-Maximizers • Protein Reinforcement and DNA Consequentialism • Not for the Sake of Happiness (Alone) • The Two-Party Swindle • Three Fallacies of Teleology • Sensual Experience • Devil's Offers

Original with comments: Superstimuli and the Collapse of Western Civilization

Useless Medical Disclaimers

Eliezer Yudkowsky, 19 March 2007 04:48PM

I recently underwent a minor bit of toe surgery and had to sign a scary-looking disclaimer form in which I acknowledged that there was a risk of infection, repeat surgery, chronic pain, amputation, spontaneous combustion, meteor strikes, and a plague of locusts o'er the land.

It was the most pointless damned form I've ever seen in a doctor's office. What are the statistical incidences of any of these risks? Should I be more or less worried about dying in a car crash on the way home? Taken literally, that kind of "information" is absolutely useless for making decisions. You can't translate something into an expected utility, even a qualitative and approximate one, if it doesn't come with a probability attached.

Taken literally, saying that there is a "possibility" of infection tells me nothing. The probability could be 1/1,000,000,000,000 and it would still be technically correct to describe the outcome as "possible". I'm not the litigious type, but I seriously wonder if it would be possible to sue based on the theory that "possibilities" with no probabilities attached to them are not useful information and therefore should not constitute a "disclaimer" under the law.

Staring at this pointless list of disasters, I also wondered why the form contained no useful information.

The thought that occurred to me was that, innumeracy being so widespread, no one would dare put numbers on that sheet of paper. If "amputation" is listed as a consequence with a probability of 0.0001%, patients will run screaming out of the office, crying, "Not my toe! I don't want to lose my toe!" No amount of patient explanation will suffice to convince them that they ought to diminish the emotional force of their fear by a factor of one million. Each extra zero after the decimal point would only be one more symbol for their eyes to glaze over; it would not diminish the emotional force of the anticipation by an additional factor of ten.

And so I don't get any useful statistical information! Hmph.

Clearly, innumeracy produces negative externalities and it ought to be regulated. In particular, we should impose a tax on people who can't properly diminish the emotional impact of their anticipations by tiny probability factors.

Two classic objections to regulation are that (a) it infringes on personal freedom and (b) the individual always knows more about their own situation than the regulator. However, my proposed policy addresses both of these issues: rather than administering a math test, we can ask each individual whether or not they're innumerate. If they do declare themselves to be innumerate, they can decide for themselves the amount of the tax to pay.

What do you think? Would this tax give people an incentive to become less innumerate, as standard economics would predict?

Original with comments: Useless Medical Disclaimers

Archimedes's Chronophone

Eliezer Yudkowsky, 23 March 2007 05:43PM

Think of how many generations of humanity would have benefited if certain ideas had been invented sooner, rather than later - if the Greeks had invented science - if the Romans had possessed printing presses - if Western civilization had turned against slavery in the thirteenth century.

Archimedes of Syracuse was the greatest mathematician and engineer of the ancient world. Imagine that Archimedes invented a temporal telephone ("chronophone" for short) which lets him talk to you, here in the 21st century. You can make suggestions! For purposes of the thought experiment, ignore the morality of altering history - just assume that it is proper to optimize post-Archimedean history as though it were simply the ordinary future. If so, it would seem that you are in a position to accomplish a great deal of good.

Unfortunately, Archimedes's chronophone comes with certain restrictions upon its use: It cannot transmit information that is, in a certain sense, "too anachronistic".

You cannot suggest, for example, that women should have the vote. Maybe you could persuade Archimedes of Syracuse of the issue, and maybe not; but it is a moot point, the chronophone will not transmit the advice. Or rather, it will transmit the advice, but it will come out as: "Install a tyrant of great personal virtue, such as Hiero II, under whose rule Syracuse experienced fifty years of peace and prosperity." That's how the chronophone avoids transmitting overly anachronistic information - it transmits cognitive strategies rather than words. If you follow the policy of "Check my brain's memory to see what my contemporary culture recommends as a wise form of political organization", what comes out of the chronophone is the result of Archimedes following the same policy of looking up in his brain what his era lauds as a wise form of political organization.

You might think the next step would be to prepare a careful series of Plato-style philosophical arguments, starting from known territory, and intended to convince an impartial audience, with which to persuade Archimedes that all sentient beings should be equal before the law. Unfortunately, if you try this, what comes out on Archimedes's end is a careful series of Plato-style philosophical analogies which argue that wealthy male landowners should have special privileges. You followed the policy of "Come up with a line of philosophical argument intended to persuade a neutral observer to my own era's point of view on political privilege," so what comes out of the chronophone is what Archimedes would think up if he followed the same cognitive strategy.

In Archimedes's time, slavery was thought right and proper; in our time, it is held an abomination. If, today, you need to argue that slavery is bad, you can invent all sorts of moral arguments which lead to that conclusion - all sorts of justifications leap readily to mind. If you could talk to Archimedes of Syracuse directly, you might even be able to persuade him to your viewpoint (or not). But the really odd thing is that, at some point in time, someone must have turned against slavery - gone from pro-slavery to anti-slavery - even though they didn't start out wanting to persuade themselves against slavery. By the time someone gets to the point of wanting to construct persuasive anti-slavery arguments, they must have already turned against slavery. If you know your desired moral destination, you are already there. Thus, that particular cognitive strategy - searching for ways to persuade people against slavery - can't explain how we got here from there, how Western culture went from pro-slavery to anti-slavery.

The chronophone, to prevent paradox, will not transmit arguments that you constructed already knowing the desired destination. And because this is a law of physics governing time travel, the chronophone cannot be fooled. No matter how cleverly you construct your neutral-sounding philosophical argument, the chronophone "knows" you started with the desired conclusion already in mind.

The same dilemma applies to scientific issues. if you say "The Earth circles the Sun" it comes out of the chronophone as "The Sun circles the Earth". It doesn't matter that our civilization is right and their civilization is wrong - the chronophone takes no notice of facts, only beliefs and cognitive strategies. You tried to transmit your own belief about heavenly mechanics, so it comes out as Archimedes's belief about heavenly mechanics.

Obviously, what you need to transmit is the scientific method - that's how our own civilization went from geocentrism to heliocentrism without having the destination already in mind. Unfortunately, you also can't say to Archimedes, "Use mathematical laws instead of heroic mythology to explain empirical phenomena." It will come out as "If anyone should throw back his head and learn something by staring at the varied patterns on a ceiling, apparently you would think that he was contemplating with his reason, when he was only staring with his eyes... I cannot but believe that no study makes the soul look on high except that which is concerned with real being and the unseen." (Plato, The Republic, Book VII.) That is Archimedes's culture's stance on epistemology, just as science is your own culture's stance.

Can you suggest that Archimedes pay attention to facts, and authorities, and think about which one should ought to take precedence - by way of leading him down a garden path to the scientific method? But humanity did not invent the scientific method by setting out to invent the scientific method - by looking for a garden path that would lead to the scientific method. If you know your desired destination, you are already there. And no matter how you try to prevent your garden path from looking like a garden path, the laws of time travel know the difference.

So what can you say into the chronophone?

Suppose that, at some point in your life, you've genuinely thought that the scientific method might not be correct - that our culture's preferred method of factual investigation might be flawed. Then, perhaps, you could talk into the chronophone about how you've doubted that the scientific method as commonly practiced is correct, and it would come out of the chronophone as doubts about whether deference to authority is correct. After all, something like that must be how humanity got to science from nonscience - individuals who genuinely questioned whether their own culture's preferred method of epistemological investigation was correct.

If you try to follow this strategy, your own doubts had better be genuine. Otherwise what will come out of the chronophone is a line of Socratic questioning that argues for deference to authority. If your doubts are genuine, surface doubts will come out as surface doubts, deep doubts as deep doubts. The chronophone always knows how much you really doubted, and how much you merely tried to convince yourself you doubted so that you could say it into the chronophone. Such is the unavoidable physics of time travel.

Now... what advice do you give to Archimedes, and how do you say it into the chronophone?

Addendum: A basic principle of the chronophone is that to get nonobvious output, you need nonobvious input. If you say something that is considered obvious in your home culture, it comes out of the chronophone as something that is considered obvious in Archimedes's culture.

Referenced by: Chronophone Motivations • Distinct Configurations

Original with comments: Archimedes's Chronophone

Chronophone Motivations

Eliezer Yudkowsky, 24 March 2007 05:23PM

Followup to: Archimedes's Chronophone.

Suppose you could send messages back in time to Archimedes of Syracuse, using a chronophone which - to avoid transmitting anachronistic information - transmits the results of executing cognitive strategies, rather than words. If you say "Women should have the vote", it comes out as "Install a tyrant of great personal virtue", because you repeated what your culture considers a wise form of political arrangement, and what comes out of the chronophone is the result of executing the same cognitive policy in Archimedes's era.

The chronophone won't transmit arguments you rationalize using your home culture's foreknowledge of the desired conclusion - it will substitute the result of executing that cognitive policy using Archimedes's culture's belief as the intended conclusion. A basic principle of the chronophone is that if you say something considered obvious in your home culture, it comes out as something considered obvious in Archimedes's culture.

The challenge was to say something useful under this restriction. This challenge is supposed to be difficult. It's really hard to get somewhere when you don't already know your destination. If there were some simple cognitive policy you could follow to spark moral and technological revolutions, without your home culture having advance knowledge of the destination, you could execute that cognitive policy today - which is what the whole parable is about!

A surprising number of respondents seemed to completely miss the point of the chronophone, just thinking up things they would like to say directly to Archimedes. The classic question of "If you went back in time, how would you start up an industrial civilization?" has been done many times in science fiction (Lord Kalvan of Otherwhen, The Cross-Time Engineer). There are thousands of things we'd like to say to the Past. The difficult part of the question is: How do you get it to come out of the chronophone?

Ger suggested teaching Archimedes decimal notation. Well, if you speak decimal notation - our home culture's standard representation of numbers - into the chronophone, then the chronophone outputs the standard representation of numbers used in Syracuse. To get a culturally nonobvious output, you need a culturally nonobvious input. Place notation is revolutionary because it makes it easier for ordinary people, not just trained accountants, to manipulate large numbers. Maybe an equivalent new idea in our own era would be Python, which makes it easier for novices to program computers - or a mathematician trying to standardize on category theory instead of set theory as a foundation for mathematics. Coming up with that chronophone input suggests that maybe we should pay more attention, in this era, to Python or category theory! A new representation that makes math easier can add up to a lot of benefit over time.

Hertzlinger remarked: "Some of Archimedes's most potentially-important research involved things he regarded as trivial toys. So if we advise him to get interested in Rubik's cube..." Of course you cannot directly describe a Rubik's Cube into the chronophone. So I asked what corresponding input Hertzlinger would say into the chronophone - has Hertzlinger followed the cognitive policy of playing with toy ideas? Maybe if this would have been such a good policy for Archimedes to follow, we should follow it ourselves.

Robin Hanson proposed an (admittedly clever) meta-trick for fine-tuning the chronophone's output. If that worked, Robin wanted to suggest trying to make useful devices that make money, and creating a tradition of this activity. I asked Robin if he'd ever tried to make such useful devices himself - if this is so important to human progress, why isn't Robin doing it? Perhaps Robin could reply that we've already gotten a huge amount of progress out of inventing gadgets, so now this no longer offers the greatest marginal returns. But that, in turn, points up one of the essential difficulties of the challenge. In this era it is culturally obvious - a non-surprising idea - that money-making new technologies benefit humanity. What could you say into the chronophone that would correspond to the nonobviousness of that idea in Archimedes's era? I don't know if it's important enough to qualify, but, for example, Robin's thoughts about prediction markets are not considered obvious in modern culture. That makes them a better bet for chronophone input than if Robin were to describe his efforts to invent a fancy new gadget. Everyone's doing that these days; it would probably come out of the chronophone as a suggestion to become a great warrior.

Richard Hamming used to ask his fellow researchers two questions: "What are the most important problems of your field?" and "Why aren't you working on them?"

What kind of ideas have provided the greatest benefit to humanity? Why aren't you thinking them?

Most of what we desperately want to say to Archimedes is not obvious relative to Archimedes's culture. This strongly suggests that the most important things the Future would want to say to us are, amazingly enough, not things that everyone already knows. If you want to really benefit humanity, you've got to do some original thinking - come up with the sort of nonobvious idea that you would speak into a chronophone. And you have to do some hard thinking about areas of application, directions of effort. You can't just run off in the direction of what your contemporary culture has instilled as the reflex answer to the question "How can I benefit humanity?" In those orchards the low-hanging fruit is gone.

The point of the chronophone dilemma is to make us think about what kind of cognitive policies are good to follow when you don't know your destination in advance. If you can just tell Archimedes to build a capitalist society because your culture already knows this is a good idea, it defeats the purpose of the dilemma. The chronophone transmits cognitive policies, not sentences. What sort of thinking are we doing now that is analogous to the kind of thinking we wish Archimedes had done then?

Referenced by: Distinct Configurations • LA-602 vs. RHIC Review • Mirrors and Paintings • Crisis of Faith

Original with comments: Chronophone Motivations

Self-deception: Hypocrisy or Akrasia?

Eliezer Yudkowsky, 26 March 2007 05:03PM

What are we to think when someone says with their lips that they desire truth, but by their other cognitive deeds choose comfortable illusions over reality (or comfortable cynicism over reality)?

Robin Hanson has labeled such individuals hypocrites. In the traditional sense of the term, a hypocrite is a moral liar: someone who says a morality which they do not, themselves, believe. On the other hand, we don't always live up to the goals we set for ourselves. If I really believe that I ought to exercise at least 3 times per week, but I don't always do so, am I properly termed a "hypocrite"? The term akrasia, meaning "weakness of will" or "failure of self-control", seems more appropriate. Even if I tell all my friends that they ought to exercise 3 times per week, that doesn't necessarily make me a hypocrite. It's good advice. (Now, if I claimed to always exercise 3 times per week, knowing that this claim was false, that would be dishonest.)

Accusations of hypocrisy garner a lot more attention than accusations of akrasia - because hypocrisy is a deliberate transgression. It is tempting to say "hypocrisy" when you really mean "akrasia", because you'll get more attention, but that can cause damage to innocent bystanders. In akrasia, your transgression is your failure of will - it's fine that you advocate going to the gym more often, you just need to live up to the principle yourself. In hypocrisy, the transgression is claiming to care: you have no right to publicly advocate the moral principle, because (the accuser says) you don't believe in it yourself.

Will Wilkinson asked Hanson: "Would it be a kind of victory if people who now say that they care about truth, but who really don't, started admitting that they really don't?"

But much more importantly: who says that people who claim to care about truth, and then deceive themselves, "really don't care" about the truth? Why not say that they really care about the truth (as is right and proper), but they aren't living up to their own morals?

It may be standard practice in economics to deduce "preferences" from actions rather than declarations, but that's because you're trying to predict, in a scientific sense, what the subject will do next - trying to build good economic models. Moral philosophy is a different bag o' worms. At the very least, it is a controversial step in moral reasoning to decide that people's emotional impulses and subconscious pressures, rather than their declarative moral reasoning processes and the words that issue from their lips, constitute their "real selves". We should then call akrasia, not weakness of will, but strength of will.

To put the dilemma more sharply: The one comes before you and pleads, "I know that I have many times been guilty of self-deception. I have bought lottery tickets, I have overestimated my driving skills, I have planned optimistically, I have refused to confront contradictory evidence. I am weak. And yet I desire to do better. Will you help me?"

So that is words issuing from the lips, which say one thing. And it may be that the one has committed other deeds which say something else. Who is the real person? Does that question have an answer, or only a definition?

I do not frame an answer. It is only needful for me to know that something has asked for my help. There is something here that can ally to me, in our quest for truth - whether or not you call it the "real self". Whether or not, for that matter, you call me my "real self". If the word "I", when I use it, does not refer to the cognitive pattern that authors these words on your computer screen, what does it refer to? And if the words that issue from some other's lips should declare me to be a ghost, then I will seek out my fellow truthseeking ghosts, and have company in my phantom quest.

Referenced by: Fake Norms, or "Truth" vs. Truth • Inseparably Right; or, Joy in the Merely Good • Moral Error and Moral Disagreement

Original with comments: Self-deception: Hypocrisy or Akrasia?

Tsuyoku Naritai! (I Want To Become Stronger)

Eliezer Yudkowsky, 27 March 2007 05:49PM

In Orthodox Judaism there is a saying: "The previous generation is to the next one as angels are to men; the next generation is to the previous one as donkeys are to men." This follows from the Orthodox Jewish belief that all Judaic law was given to Moses by God at Mount Sinai. After all, it's not as if you could do an experiment to gain new halachic knowledge; the only way you can know is if someone tells you (who heard it from someone else, who heard it from God). Since there is no new source of information, it can only be degraded in transmission from generation to generation.

Thus, modern rabbis are not allowed to overrule ancient rabbis. Crawly things are ordinarily unkosher, but it is permissible to eat a worm found in an apple - the ancient rabbis believed the worm was spontaneously generated inside the apple, and therefore was part of the apple. A modern rabbi cannot say, "Yeah, well, the ancient rabbis knew diddly-squat about biology. Overruled!" A modern rabbi cannot possibly know a halachic principle the ancient rabbis did not, because how could the ancient rabbis have passed down the answer from Mount Sinai to him? Knowledge derives from authority, and therefore is only ever lost, not gained, as time passes.

When I was first exposed to the angels-and-donkeys proverb in (religious) elementary school, I was not old enough to be a full-blown atheist, but I still thought to myself: "Torah loses knowledge in every generation. Science gains knowledge with every generation. No matter where they started out, sooner or later science must surpass Torah."

The most important thing is that there should be progress. So long as you keep moving forward you will reach your destination; but if you stop moving you will never reach it.

Tsuyoku naritai is Japanese. Tsuyoku is "strong"; naru is "becoming" and the form naritai is "want to become". Together it means "I want to become stronger" and it expresses a sentiment embodied more intensely in Japanese works than in any Western literature I've read. You might say it when expressing your determination to become a professional Go player - or after you lose an important match, but you haven't given up - or after you win an important match, but you're not a ninth-dan player yet - or after you've become the greatest Go player of all time, but you still think you can do better. That is tsuyoku naritai, the will to transcendence.

Tsuyoku naritai is the driving force behind my essay The Proper Use of Humility, in which I contrast the student who humbly double-checks his math test, and the student who modestly says "But how can we ever really know? No matter how many times I check, I can never be absolutely certain." The student who double-checks his answers wants to become stronger; he reacts to a possible inner flaw by doing what he can to repair the flaw, not with resignation.

Each year on Yom Kippur, an Orthodox Jew recites a litany which begins Ashamnu, bagadnu, gazalnu, dibarnu dofi, and goes on through the entire Hebrew alphabet: We have acted shamefully, we have betrayed, we have stolen, we have slandered...

As you pronounce each word, you strike yourself over the heart in penitence. There's no exemption whereby, if you manage to go without stealing all year long, you can skip the word gazalnu and strike yourself one less time. That would violate the community spirit of Yom Kippur, which is about confessing sins - not avoiding sins so that you have less to confess.

By the same token, the Ashamnu does not end, "But that was this year, and next year I will do better."

The Ashamnu bears a remarkable resemblance to the notion that the way of rationality is to beat your fist against your heart and say, "We are all biased, we are all irrational, we are not fully informed, we are overconfident, we are poorly calibrated..."

Fine. Now tell me how you plan to become less biased, less irrational, more informed, less overconfident, better calibrated.

There is an old Jewish joke: During Yom Kippur, the rabbi is seized by a sudden wave of guilt, and prostrates himself and cries, "God, I am nothing before you!" The cantor is likewise seized by guilt, and cries, "God, I am nothing before you!" Seeing this, the janitor at the back of the synagogue prostrates himself and cries, "God, I am nothing before you!" And the rabbi nudges the cantor and whispers, "Look who thinks he's nothing."

Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws. This is akin to the principle of not taking pride in confessing your ignorance; for if your ignorance is a source of pride to you, you may become loathe to relinquish your ignorance when evidence comes knocking. Likewise with our flaws - we should not gloat over how self-aware we are for confessing them; the occasion for rejoicing is when we have a little less to confess.

Otherwise, when the one comes to us with a plan for correcting the bias, we will snarl, "Do you think to set yourself above us?" We will shake our heads sadly and say, "You must not be very self-aware."

Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it. Afterward you will still have plenty of flaws left, but that's not the point; the important thing is to do better, to keep moving ahead, to take one more step forward. Tsuyoku naritai!

Sequence: Challenging the Difficult

The Proper Use of Humility

Tsuyoku vs. the Egalitarian Instinct

Referenced by: Tsuyoku vs. the Egalitarian Instinct • Debiasing as Non-Self-Destruction • Lotteries: A Waste of Hope • The Third Alternative • Risk-Free Bonds Aren't • The Importance of Saying "Oops" • Update Yourself Incrementally • Mysterious Answers to Mysterious Questions • Why I'm Blooking • Avoiding Your Belief's Real Weak Points • The Meditation on Curiosity • Explainers Shoot High. Aim Low! • Why Are Individual IQ Differences OK? • Guardians of the Truth • Guardians of Ayn Rand • My Strange Beliefs • The Fallacy of Gray • Something to Protect • Savanna Poets • Where Philosophy Meets Science • No Safe Defense, Not Even Science • My Childhood Role Model • The Opposite Sex • The Meaning of Right • You Provably Can't Trust Yourself • The Level Above Mine • Make an Extraordinary Effort • Shut up and do the impossible! • Amputation of Destiny • Free to Optimize • 31 Laws of Fun • Cynical About Cynicism • A Sense That More Is Possible • Extenuating Circumstances • Of Gender and Rationality • The Sin of Underconfidence • Conversation Halters

Original with comments: Tsuyoku Naritai! (I Want To Become Stronger)

Tsuyoku vs. the Egalitarian Instinct

Eliezer Yudkowsky, 28 March 2007 05:49PM

Followup to: Tsuyoku naritai! (I Want To Become Stronger)

Hunter-gatherer tribes are usually highly egalitarian (at least if you're male) - the all-powerful tribal chieftain is found mostly in agricultural societies, rarely in the ancestral environment. Among most hunter-gatherer tribes, a hunter who brings in a spectacular kill will carefully downplay the accomplishment to avoid envy.

Maybe, if you start out below average, you can improve yourself without daring to pull ahead of the crowd. But sooner or later, if you aim to do the best you can, you will set your aim above the average.

If you can't admit to yourself that you've done better than others - or if you're ashamed of wanting to do better than others - then the median will forever be your concrete wall, the place where you stop moving forward. And what about people who are below average? Do you dare say you intend to do better than them? How prideful of you!

Maybe it's not healthy to pride yourself on doing better than someone else. Personally I've found it to be a useful motivator, despite my principles, and I'll take all the useful motivation I can get. Maybe that kind of competition is a zero-sum game, but then so is Go; it doesn't mean we should abolish that human activity, if people find it fun and it leads somewhere interesting.

But in any case, surely it isn't healthy to be ashamed of doing better.

And besides, life is not graded on a curve. The will to transcendence has no point beyond which it ceases and becomes the will to do worse; and the race that has no finish line also has no gold or silver medals. Just run as fast as you can, without worrying that you might pull ahead of other runners. (But be warned: If you refuse to worry about that possibility, someday you may pull ahead. If you ignore the consequences, they may happen to you.)

Sooner or later, if your path leads true, you will set out to mitigate a flaw that most people have not mitigated. Sooner or later, if your efforts bring forth any fruit, you will find yourself with fewer sins to confess.

Perhaps you will find it the course of wisdom to downplay the accomplishment, even if you succeed. People may forgive a touchdown, but not dancing in the end zone. You will certainly find it quicker, easier, more convenient, to publicly disclaim your worthiness, to pretend that you are just as much a sinner as everyone else. Just so long, of course, as everyone knows it isn't true. It can be fun to proudly display your modesty, so long as everyone knows how very much you have to be modest about.

But do not let that be the endpoint of your journeys. Even if you only whisper it to yourself, whisper it still: Tsuyoku, tsuyoku! Stronger, stronger!

And then set yourself a higher target. That's the true meaning of the realization that you are still flawed (though a little less so). It means always reaching higher, without shame.

Tsuyoku naritai! I'll always run as fast as I can, even if I pull ahead, I'll keep on running; and someone, someday, will surpass me; but even though I fall behind, I'll always run as fast as I can.

Sequence: Challenging the Difficult

Tsuyoku Naritai! (I Want To Become Stronger)

Guardians of the Truth

Referenced by: The Error of Crowds • No One Can Exempt You From Rationality's Laws • The Argument from Common Usage • Whining-Based Communities • Well-Kept Gardens Die By Pacifism • Conversation Halters

Original with comments: Tsuyoku vs. the Egalitarian Instinct

"Statistical Bias"

Eliezer Yudkowsky, 30 March 2007 06:55PM

(Part one in a series on "statistical bias", "inductive bias", and "cognitive bias".)

"Bias" as used in the field of statistics refers to directional error in an estimator. Statistical bias is error you cannot correct by repeating the experiment many times and averaging together the results.

The famous bias-variance decomposition states that the expected squared error is equal to the squared directional error, or bias, plus the squared random error, or variance. The law of large numbers says that you can reduce variance, not bias, by repeating the experiment many times and averaging the results.

An experiment has some randomness in it, so if you repeat the experiment many times, you may get slightly different data each time; and if you run a statistical estimator over the data, you may get a slightly different estimate each time. In classical statistics, we regard the true value of the parameter as a constant, and the experimental estimate as a probabilistic variable. The bias is the systematic, or average, difference between these two values; the variance is the leftover probabilistic component.

Let's say you have a repeatable experiment intended to estimate, for example, the height of the Emperor of China. In fact, the Emperor's height is 200 cm. Suppose that every single American believes, without variation, that the Emperor's height is 180 cm. Then if you poll a random American and ask "How tall is the Emperor of China?", the answer is always "180 cm", the error is always -20 cm, and the squared error is always 400 (I shall omit the units on squared errors). But now suppose that Americans have normally distributed beliefs about the Emperor's height, with mean belief 180 cm, and standard deviation 10 cm. You conduct two independent repetitions of the poll, and one American says "190 cm", and the other says "170 cm", with errors respectively of -10 cm and -30 cm, and squared errors of 100 and 900. The average error is -20 cm, as before, but the average squared error is 100 + 900 / 2 = 500. So even though the average (directional) error didn't change as the result of adding noise to the experiments, the average squared error went up.

Although in one case the random perturbation of the answer happened to lead the American in the correct direction - the one who answered 190 cm, which is closer to the true value of 200 cm - the other American was led further away from the answer, replying 170 cm. Since these are equal deviations, the average answer did not change. But since the square increases faster than linear, the larger error corresponded to a still larger squared error, and the average squared error went up.

Furthermore, the new average squared error of 500 equals exactly the square of the directional error (-20 cm) plus the square of the random error (standard deviation of 10cm): 400 + 100 = 500.

In the long run, the above result is universal and exact: If the true value is constant X and the estimator is Y, then E[(X - Y)^2] = (X - E[Y])^2 + E[(E[Y] - Y)^2]. Expected squared error = squared expected bias + expected variance of estimator. This is the bias-variance decomposition.

If we averaged together the two Americans above, we would get an average estimate of 180 cm, with a squared error of 400, which is less than the average error of both experiments taken individually, but still erroneous.

If the true value is constant X and the estimator is Y, then by averaging many estimates together we converge toward the expected value of Y, E[Y], by the law of large numbers, and if we subtract this from X, we are left with a squared error of (X - E[Y])^2, which is the bias term of the bias-variance decomposition. If your estimator is all over the map and highly sensitive to noise in the experiment, then by repeating the experiment many times you can get the expected value of your estimator, and so you are left with only the systematic error of that estimator, and not the random noise in the estimator that varies from experiment to experiment. That's what the law of large numbers is good for.

Referenced by: Useful Statistical Biases • The Error of Crowds • "Inductive Bias"

Original with comments: "Statistical Bias"

Useful Statistical Biases

Eliezer Yudkowsky, 01 April 2007 04:51AM

Friday's post on statistical bias and the bias-variance decomposition discussed how the squared error of an estimator equals the directional error of the estimator plus the variance of the estimator. All else being equal, bias is bad - you want to get rid of it. But all else is not always equal. Sometimes, by accepting a small amount of bias in your estimator, you can eliminate a large amount of variance. This is known as the "bias-variance tradeoff".

A linear regression tries to estimate a quantity by attaching weights to various signals associated with that quantity - for example, you could try to predict the gas mileage of a car using the car's mass and engine capacity.

A regularized linear regression tries to attach smaller variable weights, while still matching the data fairly well. A regularized regression may generalize to unseen data better than an unregularized regression - often quite a lot better. Assigning smaller variable weights is akin to finding a simpler explanation that fits the data almost as well. This drive for simplicity makes the regressor less sensitive to small random wobbles in the data, so it has lower variance: if you ran the regressor over different data samples, the estimates would look more similar to each other.

But the same regularization procedure also causes the estimator to ignore some actual data - and this is a systematic error, that would recur in the same direction if we repeated the experiment many times. The randomness goes in both directions, so by ignoring the noise in the data, you decrease your variance. But the real evidence goes in one direction, so if you ignore some real evidence in the process of ignoring noise - because you don't know which is which - then you end up with a directional error, an error that trends in the same direction when you repeat the experiment many times.

In statistics this is known as the bias-variance tradeoff. When your data is limited, it may be better to use a simplifying estimator that doesn't try to fit every tiny squiggle of the data, and this trades off a lot of variance against a little bias.

An "unbiased estimator" is one whose expected result equals the correct result, although it may have wide random swings in either direction. This is good if you are allowed to repeat the experiment as often as you like, because you can average together the estimates and get the correct answer to arbitrarily fine precision. That's the law of large numbers.

You might have the following bright idea - why not use an unbiased estimator, like an unregularized regression, to guess the bias of a regularized regression? Then you could just subtract out the systematic bias - you could have low bias and low variance. The problem with this, you see, is that while it may be easy to find an unbiased estimator of the bias, this estimate may have very large variance - so if you subtract out an estimate of the systematic bias, you may end up subtracting out way too much, or even subtracting in the wrong direction a fair fraction of the time. In statistics, "unbiased" is not the same as "good", unless the estimator also has low variance.

When you hear that a classroom gave an average estimate of 871 beans for a jar that contained 850 beans, and that only one individual student did better than the crowd, the astounding notion is not that the crowd can be more accurate than the individual. The astounding notion is that human beings are unbiased estimators of beans in a jar, having no significant directional error on the problem, yet with large variance. It implies that we tend to get the answer wrong but there's no systematic reason why. It requires that there be lots of errors that vary from individual to individual - and this is reliably true, enough so to keep most individuals from guessing the jar correctly. And yet there are no directional errors that everyone makes, or if there are, they cancel out very precisely in the average case, despite the large individual variations. Which is just plain odd. I find myself somewhat suspicious of the claim, and wonder whether other experiments that found less amazing accuracy were not as popularly reported.

Someone is bound to suggest that cognitive biases are useful, in the sense that they represent a bias-variance tradeoff. I think this is just mixing up words - just because the word "bias" is used by two different fields doesn't mean it has the same technical definition. When we accept a statistical bias in trade, we can't get strong information about the direction and magnitude of the bias - otherwise we would just subtract it out. We may be able to get an unbiased estimate of the bias, but "unbiased" is not the same as "reliable"; if the variance is huge, we really have very little information. Now with cognitive biases, we do have some idea of the direction of the systematic error, and the whole notion of "overcoming bias" is about trying to subtract it out. Once again, we see that cognitive biases are lemons, not lemonade. To the extent we can get strong information - e.g. from cognitive psychology experiments - about the direction and magnitude of a systematic cognitive error, we can do systematically better by trying to compensate.

Original with comments: Useful Statistical Biases

The Error of Crowds

Eliezer Yudkowsky, 01 April 2007 09:50PM

I've always been annoyed at the notion that the bias-variance decomposition tells us something about modesty or Philosophical Majoritarianism. For example, Scott Page rearranges the equation to get what he calls the Diversity Prediction Theorem:

Collective Error = Average Individual Error - Prediction Diversity

I think I've finally come up with a nice, mathematical way to drive a stake through the heart of that concept and bury it beneath a crossroads at midnight, though I fully expect that it shall someday rise again and shamble forth to eat the brains of the living.

Why should the bias-variance decomposition be relevant to modesty? Because, it seems to show, the error of averaging all the estimates together, is lower than the typical error of an individual estimate. Prediction Diversity (the variance) is positive when any disagreement exists at all, so Collective Error < Average Individual Error. But then how can you justify keeping your own estimate, unless you know that you did better than average? And how can you legitimately trust that belief, when studies show that everyone believes themselves to be above-average? You should be more modest, and compromise a little.

So what's wrong with this picture?

To begin with, the bias-variance decomposition is a mathematical tautology. It applies when we ask a group of experts to estimate the 2007 close of the NASDAQ index. It would also apply if you weighed the experts on a pound scale and treated the results as estimates of the dollar cost of oil in 2020.

As Einstein put it, "Insofar as the expressions of mathematics refer to reality they are not certain, and insofar as they are certain they do not refer to reality." The real modesty argument, Aumann's Agreement Theorem, has preconditions; AAT depends on agents computing their beliefs in a particular way. AAT's conclusions can be false in any particular case, if the agents don't reason as Bayesians.

The bias-variance decomposition applies to the luminosity of fireflies treated as estimates, just as much as a group of expert opinions. This tells you that you are not dealing with a causal description of how the world works - there are not necessarily any causal quantities, things-in-the-world, that correspond to "collective error" or "prediction diversity". The bias-variance decomposition is not about modesty, communication, sharing of evidence, tolerating different opinions, humbling yourself, overconfidence, or group compromise. It's an algebraic tautology that holds whenever its quantities are defined consistently, even if they refer to the silicon content of pebbles.

More importantly, the tautology depends on a particular definition of "error": error must go as the squared difference between the estimate and the true value. By picking a different error function, just as plausible as the squared difference, you can conjure a diametrically opposed recommendation:

The professor cleared his throat. "All right," he said to the gathered students, "you've each handed in your written estimates of the value of this expression here," and he gestured to a rather complex-looking string of symbols drawn on the blackboard. "Now it so happens," the professor continued, "that this question contains a hidden gotcha. All of you missed in the same direction - that is, you all underestimated or all overestimated the true value, but I won't tell you which. Now, I'm going to take the square root of the amount by which you missed the correct answer, and subtract it from your grade on today's homework. But before I do that, I'm going to give you a chance to revise your answers. You can talk with each other and share your thoughts about the problem, if you like; or alternatively, you could stick your fingers in your ears and hum. Which do you think is wiser?"

Here we are taking the square root of the difference between the true value and the estimate, and calling this the error function, or loss function. (It goes without saying that a student's utility is linear in their grade.)

And now, your expected utility is higher if you pick a random student's estimate than if you pick the average of the class! The students would do worse, on average, by averaging their estimates together! And this again is tautologously true, by Jensen's Inequality.

A brief explanation of Jensen's Inequality:

(I strongly recommend looking at this graph while reading the following.)

Jensen's Inequality says that if X is a probabilistic variable, F(X) is a function of X, and E[expr] stands for the probabilistic expectation of expr, then:

E[F(X)] <= F(E[X]) if F is concave (second derivative negative)
E[F(X)] >= F(E[X]) if F is convex (second derivative positive)

Why? Well, think of two values, x1 and x2. Suppose F is convex - the second derivative is positive, "the cup holds water". Now imagine that we draw a line between x=x1, y=F(x1) and x=x2, y=F(x2). Pick a point halfway along this line. At the halfway point, x will equal (x1 + x2)/2, and y will equal (F(x1)+F(x2))/2. Now draw a vertical line from this halfway point to the curve - the intersection will be at x=(x1 + x2)/2, y=F((x1 + x2)/2). Since the cup holds water, the chord between two points on the curve is above the curve, and we draw the vertical line downward to intersect the curve. Thus F((x1 + x2)/2) < (F(x1) + F(x2))/2. In other words, the F of the average is less than the average of the Fs.

So:

If you define the error as the squared difference, F(x) = x^2 is a convex function, with positive second derivative, and by Jensen's Inequality, the error of the average - F(E[X]) - is less than the average of the errors - E[F(X)]. So, amazingly enough, if you square the differences, the students can do better on average by averaging their estimates. What a surprise.

But in the example above, I defined the error as the square root of the difference, which is a concave function with a negative second derivative. Poof, by Jensen's Inequality, the average error became less than the error of the average. (Actually, I also needed the professor to tell the students that they all erred in the same direction - otherwise, there would be a cusp at zero, and the curve would hold water. The real-world equivalent of this condition is that you think the directional or collective bias is a larger component of the error than individual variance.)

If, in the above dilemma, you think the students would still be wise to share their thoughts with each other, and talk over the math puzzle - I certainly think so - then your belief in the usefulness of conversation has nothing to do with a tautology defined over an error function that happens, in the case of squared error, to be convex. And it follows that you must think the process of sharing thoughts, of arguing differences, is not like averaging your opinions together; or that sticking to your opinion is not like being a random member of the group. Otherwise, you would stuff your fingers in your ears and hum when the problem had a concave error function.

When a line of reasoning starts assigning negative expected utilities to knowledge - offers to pay to avoid true information - I usually consider that a reductio.

Original with comments: The Error of Crowds

The Majority Is Always Wrong

Eliezer Yudkowsky, 03 April 2007 01:12AM

Today my coworker Marcello pointed out to me an interesting anti-majoritarian effect. There are three major interpretations of probability: the "subjective" view of probabilities as measuring the uncertainty of agents, the "propensity" view of probabilities as chances inherent within objects, and the "frequentist" view of probabilities as the limiting value of long-run frequencies. I was remarking on how odd it was that frequentism, the predominant view in mainstream statistics, is the worst of the three major alternatives (in my view, you have to presume either uncertainty or propensity in order to talk about the limiting frequency of events that have not yet happened).

And Marcello said something along the lines of, "Well, of course. If anything were worse than frequentism, it wouldn't be there." I said, "What?" And Marcello said, "Like the saying that Mac users have, 'If Macs really were worse than Windows PCs, no one would use them.'"

At this point the light bulb went on over my head - a fluorescent light bulb - and I understood what Marcello was saying: an alternative to frequentism that was even worse than frequentism would have dropped off the radar screens long ago. You can survive by being popular, or by being superior, but alternatives that are neither popular nor superior quickly go extinct.

I can personally testify that Dvorak seems to be much easier on the fingers than Qwerty - but this is not surprising, since if Dvorak really were inferior to Qwerty, it would soon cease to exist. (Yes, I am familiar with the controversy in this area - bear in mind that this is a politically charged topic since it has been used to make accusations of market failure. Nonetheless, my fingers now sweat less, my hands feel less tired, my carpal tunnel syndrome went away, and none of this is surprising because I can feel my fingers traveling shorter distances.)

In any case where you've got (1) a popularity effect (it's easier to use something other people are using) and (2) a most dominant alternative, plus a few smaller niche alternatives, then the most dominant alternative will probably be the worst of the lot - or at least strictly superior to none of the others.

Can anyone else think of examples from their experience where there are several major alternatives that you've heard of, and a popularity effect (which may be as simple as journal editors preferring well-known usages), and the most popular alternative seems to be noticeably the worst?

Addendum: Metahacker said of this hypothesis, "It's wrong, but only sometimes." Sounds about right to me.

Original with comments: The Majority Is Always Wrong

Knowing About Biases Can Hurt People

Eliezer Yudkowsky, 04 April 2007 06:01PM

Once upon a time I tried to tell my mother about the problem of expert calibration, saying: "So when an expert says they're 99% confident, it only happens about 70% of the time." Then there was a pause as, suddenly, I realized I was talking to my mother, and I hastily added: "Of course, you've got to make sure to apply that skepticism evenhandedly, including to yourself, rather than just using it to argue against anything you disagree with -"

And my mother said: "Are you kidding? This is great! I'm going to use it all the time!"

Taber and Lodge's Motivated skepticism in the evaluation of political beliefs describes the confirmation of six predictions:

1. Prior attitude effect. Subjects who feel strongly about an issue - even when encouraged to be objective - will evaluate supportive arguments more favorably than contrary arguments.

2. Disconfirmation bias. Subjects will spend more time and cognitive resources denigrating contrary arguments than supportive arguments.

3. Confirmation bias. Subjects free to choose their information sources will seek out supportive rather than contrary sources.

4. Attitude polarization. Exposing subjects to an apparently balanced set of pro and con arguments will exaggerate their initial polarization.

5. Attitude strength effect. Subjects voicing stronger attitudes will be more prone to the above biases.

6. Sophistication effect. Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to the above biases.

If you're irrational to start with, having more knowledge can hurt you. For a true Bayesian, information would never have negative expected utility. But humans aren't perfect Bayes-wielders; if we're not careful, we can cut ourselves.

I've seen people severely messed up by their own knowledge of biases. They have more ammunition with which to argue against anything they don't like. And that problem - too much ready ammunition - is one of the primary ways that people with high mental agility end up stupid, in Stanovich's "dysrationalia" sense of stupidity.

You can think of people who fit this description, right? People with high g-factor who end up being less effective because they are too sophisticated as arguers? Do you think you'd be helping them - making them more effective rationalists - if you just told them about a list of classic biases?

I recall someone who learned about the calibration / overconfidence problem. Soon after he said: "Well, you can't trust experts; they're wrong so often as experiments have shown. So therefore, when I predict the future, I prefer to assume that things will continue historically as they have -" and went off into this whole complex, error-prone, highly questionable extrapolation. Somehow, when it came to trusting his own preferred conclusions, all those biases and fallacies seemed much less salient - leapt much less readily to mind - than when he needed to counter-argue someone else.

I told the one about the problem of disconfirmation bias and sophisticated argument, and lo and behold, the next time I said something he didn't like, he accused me of being a sophisticated arguer. He didn't try to point out any particular sophisticated argument, any particular flaw - just shook his head and sighed sadly over how I was apparently using my own intelligence to defeat itself. He had acquired yet another Fully General Counterargument.

Even the notion of a "sophisticated arguer" can be deadly, if it leaps all too readily to mind when you encounter a seemingly intelligent person who says something you don't like.

I endeavor to learn from my mistakes. The last time I gave a talk on heuristics and biases, I started out by introducing the general concept by way of the conjunction fallacy and representativeness heuristic. And then I moved on to confirmation bias, disconfirmation bias, sophisticated argument, motivated skepticism, and other attitude effects. I spent the next thirty minutes hammering on that theme, reintroducing it from as many different perspectives as I could.

I wanted to get my audience interested in the subject. Well, a simple description of conjunction fallacy and representativeness would suffice for that. But suppose they did get interested. Then what? The literature on bias is mostly cognitive psychology for cognitive psychology's sake. I had to give my audience their dire warnings during that one lecture, or they probably wouldn't hear them at all.

Whether I do it on paper, or in speech, I now try to never mention calibration and overconfidence unless I have first talked about disconfirmation bias, motivated skepticism, sophisticated arguers, and dysrationalia in the mentally agile. First, do no harm!

Sequence: Against Rationalization

Conservation of Expected Evidence

Referenced by: Debiasing as Non-Self-Destruction • My Wild and Reckless Youth • Conjunction Controversy (Or, How They Nail It Down) • Occam's Razor • We Change Our Minds Less Often Than We Think • The Meditation on Curiosity • Priming and Contamination • Fake Optimization Criteria • Hug the Query • My Strange Beliefs • Infinite Certainty • Trust in Math • Disputing Definitions • Categorizing Has Consequences • Principles of Disagreement • Against Devil's Advocacy • Surface Analogies and Deep Causes • Does Your Morality Care What You Think? • A Genius for Destruction • Contaminated by Optimism • A Prodigy of Refutation • Crisis of Faith • Dark Side Epistemology • Pretending to be Wise • Why Our Kind Can't Cooperate • That Crisis thing seems pretty useful • Go Forth and Create the Art!

Original with comments: Knowing About Biases Can Hurt People

Debiasing as Non-Self-Destruction

Eliezer Yudkowsky, 07 April 2007 08:20PM

Nick Bostrom asks:

One sign that science is not all bogus is that it enables us to do things, like go the moon. What practical things does debiassing enable us to do, other than refraining from buying lottery tickets?

It seems to me that how to be smart varies widely between professions. A hedge-fund trader, a research biologist, and a corporate CEO must learn different skill sets in order to be actively excellent - an apprenticeship in one would not serve for the other.

Yet such concepts as "be willing to admit you lost", or "policy debates should not appear one-sided", or "plan to overcome your flaws instead of just confessing them", seem like they could apply to many professions. And all this advice is not so much about how to be extraordinarily clever, as, rather, how to not be stupid. Each profession has its own way to be clever, but their ways of not being stupid have much more in common. And while victors may prefer to attribute victory to their own virtue, my small knowledge of history suggests that far more battles have been lost by stupidity than won by genius.

Debiasing is mostly not about how to be extraordinarily clever, but about how to not be stupid. Its great successes are disasters that do not materialize, defeats that never happen, mistakes that no one sees because they are not made. Often you can't even be sure that something would have gone wrong if you had not tried to debias yourself. You don't always see the bullet that doesn't hit you.

The great victories of debiasing are exactly the lottery tickets we didn't buy - the hopes and dreams we kept in the real world, instead of diverting them into infinitesimal probabilities. The triumphs of debiasing are cults not joined; optimistic assumptions rejected during planning; time not wasted on blind alleys. It is the art of non-self-destruction.

Admittedly, none of this is spectacular enough to make the evening news. It's not a moon landing - though the moon landing did surely require thousands of things to not go wrong.

So how can we know that our debiasing efforts are genuinely useful? Well, this is the worst sort of anecdotal evidence - but people do sometimes ignore my advice, and then, sometimes, catastrophe ensues of just the sort I told them to expect. That is a very weak kind of confirmation, and I would like to see controlled studies... but most of the studies I've read consist of taking a few undergraduates who are in it for the course credit, merely telling them about the bias, and then waiting to see if they improve. What we need is longitudinal studies of life outcomes, and I can think of few people I would name as candidates for the experimental group.

The fact is, most people who take a halfhearted potshot at debiasing themselves do not get huge amounts of mileage out of it. This is one of those things you have to work at for quite a while before you get good at it, especially since there's currently no source of systematic training, or even a decent manual. If for many years you practice the techniques and submit yourself to strict constraints, it may be that you will glimpse the center. But until then, mistakes avoided are often just replaced by other mistakes. It takes time for your mind to become significantly quieter. Indeed, a little knowledge of cognitive bias often does more harm than good.

As for public proof, I can see at least three ways that it could come about. First, there might be founded an Order of Bayescraft for people who are serious about it, and the graduates of these dojos might prove systematically more successful even after controlling for measures of fluid intelligence. Second, you could wait for some individual or group, working on an important domain-specific problem but also known for their commitment to debiasing, to produce a spectacularly huge public success. Third, there might be found techniques that can be taught easily and that have readily measureable results; and then simple controlled experiments could serve as public proof, at least for people who attend to Science.

Referenced by: Lotteries: A Waste of Hope • The Magnitude of His Own Folly

Original with comments: Debiasing as Non-Self-Destruction

"Inductive Bias"

Eliezer Yudkowsky, 08 April 2007 07:52PM

(Part two in a series on "statistical bias", "inductive bias", and "cognitive bias".)

Suppose that you see a swan for the first time, and it is white. It does not follow logically that the next swan you see must be white, but white seems like a better guess than any other color. A machine learning algorithm of the more rigid sort, if it sees a single white swan, may thereafter predict that any swan seen will be white. But this, of course, does not follow logically - though AIs of this sort are often misnamed "logical". For a purely logical reasoner to label the next swan white as a deductive conclusion, it would need an additional assumption: "All swans are the same color." This is a wonderful assumption to make if all swans are, in reality, the same color; otherwise, not so good. Tom Mitchell's Machine Learning defines the inductive bias of a machine learning algorithm as the assumptions that must be added to the observed data to transform the algorithm's outputs into logical deductions.

A more general view of inductive bias would identify it with a Bayesian's prior over sequences of observations...

Consider the case of an urn filled with red and white balls, from which we are to sample without replacement. I might have prior information that the urn contains 5 red balls and 5 white balls. Or, I might have prior information that a random number was selected from a uniform distribution between 0 and 1, and this number was then used as a fixed probability to independently generate a series of 10 balls. In either case, I will estimate a 50% probability that the first ball is red, a 50% probability that the second ball is red, etc., which you might foolishly think indicated the same prior belief. But, while the marginal probabilities on each round are equivalent, the probabilities over sequences are different. In the first case, if I see 3 red balls initially, I will estimate a probability of 2/7 that the next ball will be red. In the second case, if I see 3 red balls initially, I will estimate a 4/5 chance that the next ball will be red (by Laplace's Law of Succession, thus named because it was proved by Thomas Bayes). In both cases we refine our future guesses based on past data, but in opposite directions, which demonstrates the importance of prior information.

Suppose that your prior information about the urn is that a monkey tosses balls into the urn, selecting red balls with 1/4 probability and white balls with 3/4 probability, each ball selected independently. The urn contains 10 balls, and we sample without replacement. (E. T. Jaynes called this the "binomial monkey prior".) Now suppose that on the first three rounds, you see three red balls. What is the probability of seeing a red ball on the fourth round?

First, we calculate the prior probability that the monkey tossed 0 red balls and 10 white balls into the urn; then the prior probability that the monkey tossed 1 red ball and 9 white balls into the urn; and so on. Then we take our evidence (three red balls, sampled without replacement) and calculate the likelihood of seeing that evidence, conditioned on each of the possible urn contents. Then we update and normalize the posterior probability of the possible remaining urn contents. Then we average over the probability of drawing a red ball from each possible urn, weighted by that urn's posterior probability. And the answer is... (scribbles frantically for quite some time)... 1/4!

Of course it's 1/4. We specified that each ball was independently tossed into the urn, with a known 1/4 probability of being red. Imagine that the monkey is tossing the balls to you, one by one; if it tosses you a red ball on one round, that doesn't change the probability that it tosses you a red ball on the next round. When we withdraw one ball from the urn, it doesn't tell us anything about the other balls in the urn.

If you start out with a maximum-entropy prior, then you never learn anything, ever, no matter how much evidence you observe. You do not even learn anything wrong - you always remain as ignorant as you began.

The more inductive bias you have, the faster you learn to predict the future, but only if your inductive bias does in fact concentrate more probability into sequences of observations that actually occur. If your inductive bias concentrates probability into sequences that don't occur, this diverts probability mass from sequences that do occur, and you will learn more slowly, or not learn at all, or even - if you are unlucky enough - learn in the wrong direction.

Inductive biases can be probabilistically correct or probabilistically incorrect, and if they are correct, it is good to have as much of them as possible, and if they are incorrect, you are left worse off than if you had no inductive bias at all. Which is to say that inductive biases are like any other kind of belief; the true ones are good for you, the bad ones are worse than nothing. In contrast, statistical bias is always bad, period - you can trade it off against other ills, but it's never a good thing for itself. Statistical bias is a systematic direction in errors; inductive bias is a systematic direction in belief revisions.

As the example of maximum entropy demonstrates, without a direction to your belief revisions, you end up not revising your beliefs at all. No future prediction based on past experience follows as a matter of strict logical deduction. Which is to say: All learning is induction, and all induction takes place through inductive bias.

Why is inductive bias called "bias"? Because it has systematic qualities, like a statistical bias? Because it is a form of pre-evidential judgment, which resembles the word "prejudice", which resembles the political concept of bias? Damned if I know, really - I'm not the one who decided to call it that. Words are only words; that's why humanity invented mathematics.

Referenced by: Priors as Mathematical Objects • Superexponential Conceptspace, and Simple Words • Recognizing Intelligence • Selling Nonapples

Original with comments: "Inductive Bias"

Suggested Posts

Eliezer Yudkowsky, 09 April 2007 02:32AM

Kaj Sotala asked:

I was wondering, is there an avenue for us non-contributor readers to raise questions we think would be interesting to discuss?

If you have a suggested Overcoming Bias topic you'd like to see discussed, post it in a comment here. But please don't actually discuss the topic with further comments, just give us the suggestion. This post is for topic suggestions, not topic discussions.

Original with comments: Suggested Posts

Futuristic Predictions as Consumable Goods

Eliezer Yudkowsky, 10 April 2007 12:18AM

The Wikipedia entry on Friedman Units tracks over 30 different cases between 2003 and 2007 in which someone labeled the "next six months" as the "critical period in Iraq". Apparently one of the worst offenders is journalist Thomas Friedman after whom the unit was named (8 different predictions in 4 years). In similar news, some of my colleagues in Artificial Intelligence (you know who you are) have been predicting the spectacular success of their projects in "3-5 years" for as long as I've known them, that is, since at least 2000.

Why do futurists make the same mistaken predictions over and over? The same reason politicians abandon campaign promises and switch principles as expediency demands. Predictions, like promises, are sold today and consumed today. They produce a few chewy bites of delicious optimism or delicious horror, and then they're gone. If the tastiest prediction is allegedly about a time interval "3-5 years in the future" (for AI projects) or "6 months in the future" (for Iraq), then futurists will produce tasty predictions of that kind. They have no reason to change the formulation any more than Hershey has to change the composition of its chocolate bars. People won't remember the prediction in 6 months or 3-5 years, any more than chocolate sits around in your stomach for a year and keeps you full.

The futurists probably aren't even doing it deliberately; they themselves have long since digested their own predictions. Can you remember what you had for breakfast on April 9th, 2006? I bet you can't, and I bet you also can't remember what you predicted for "one year from now".

Referenced by: Disappointment in the Future • Wrong Tomorrow

Original with comments: Futuristic Predictions as Consumable Goods

Marginally Zero-Sum Efforts

Eliezer Yudkowsky, 11 April 2007 05:22AM

Bostrom recently noted the problem of the commons in labeling efforts "important"; each managerial player has an incentive to label their project world-shakingly important, even though this devalues the priority label as used at other times or other projects, creating positive feedback in inflated labels.

This reminds me of how my grandfather, a pioneer in quantitative genetics, regularly bemoans the need to write more and more grant proposals to maintain a constant level of funding. It's not that the funding is drying up in his field. But suppose there's money for 20 grants, and 21 scientists in need of grants - or one scientist who'd like to run two projects, or receive more funding for one project... One scientist doesn't get his first grant proposal funded, so he writes another one. His second grant proposal does get funded, which uses up a grant that could have gone to another scientist, who now also has his first grant proposal denied, and has to write and send off a second grant proposal too...

The problem here is that, while some initial level of effort is beneficial, all effort beyond that is marginally zero-sum; there's a marginal return to the individual on additional efforts, but no marginal return to the group. If there are 20 grants, then ultimately only 20 grant proposals are going to be funded. No matter how many grant proposals anyone writes, the total funding available remains the same. Everyone would be better off if everyone agreed to write only one grant proposal. But in this case, there wouldn't be much competition for any given grant, and the rewards for writing another two or three grant proposals would be huge... until everyone else started doing the same thing.

There's no obvious limit to this process; the 21 scientists could write 1,000 grant proposals apiece, and still get only 20 grants between them. They'd all be better off if they only wrote one grant proposal apiece; but anyone who cuts back unilaterally will be snowed under.

In a way, this is even worse than the classic problem of the commons. A common grazing field eventually gets eaten down to bedrock and the farmers find something else to do with their herds. When professional efforts are marginally zero-sum, but yield positive returns to the individual, the resulting cycle of busy-work can expand to the limits of individual endurance.

I've often suspected that a similar effect governs bureaucracies (both government and corporate); the longer you stay at your desk each day, the more you are perceived as a hard worker and get promoted. But there's only a limited number of promotions to go around... and only a limited amount of genuinely important work to do.

Social approbation is the usual method for dealing with non-positive-sum actions. Theft has positive returns to the individual, but not positive returns to society, so we put thieves in jail. But in this case, the social dilemma is that neither writing grant proposals, nor showing up at your office desk, is inherently an evil deed. Some grant proposals do need to get written. It's not inherently a zero-sum activity. It's just marginally zero-sum beyond a certain point.

Original with comments: Marginally Zero-Sum Efforts

Priors as Mathematical Objects

Eliezer Yudkowsky, 12 April 2007 03:24AM

Followup to: "Inductive Bias"

What exactly is a "prior", as a mathematical object? Suppose you're looking at an urn filled with red and white balls. When you draw the very first ball, you haven't yet had a chance to gather much evidence, so you start out with a rather vague and fuzzy expectation of what might happen - you might say "fifty/fifty, even odds" for the chance of getting a red or white ball. But you're ready to revise that estimate for future balls as soon as you've drawn a few samples. So then this initial probability estimate, 0.5, is not repeat not a "prior".

An introduction to Bayes's Rule for confused students might refer to the population frequency of breast cancer as the "prior probability of breast cancer", and the revised probability after a mammography as the "posterior probability". But in the scriptures of Deep Bayesianism, such as Probability Theory: The Logic of Science, one finds a quite different concept - that of prior information, which includes e.g. our beliefs about the sensitivity and specificity of mammography exams. Our belief about the population frequency of breast cancer is only one small element of our prior information.

In my earlier post on inductive bias, I discussed three possible beliefs we might have about an urn of red and white balls, which will be sampled without replacement:

- Case 1: The urn contains 5 red balls and 5 white balls;

- Case 2: A random number was generated between 0 and 1, and each ball was selected to be red (or white) at this probability;

- Case 3: A monkey threw balls into the urn, each with a 50% chance of being red or white.

In each case, if you ask me - before I draw any balls - to estimate my marginal probability that the fourth ball drawn will be red, I will respond "50%". And yet, once I begin observing balls drawn from the urn, I reason from the evidence in three different ways:

- Case 1: Each red ball drawn makes it less likely that future balls will be red, because I believe there are fewer red balls left in the urn.

- Case 2: Each red ball drawn makes it more plausible that future balls will be red, because I will reason that the random number was probably higher, and that the urn is hence more likely to contain mostly red balls.

- Case 3: Observing a red or white ball has no effect on my future estimates, because each ball was independently selected to be red or white at a fixed, known probability.

Suppose I write a Python program to reproduce my reasoning in each of these scenarios. The program will take in a record of balls observed so far, and output an estimate of the probability that the next ball drawn will be red. It turns out that the only necessary information is the count of red balls seen and white balls seen, which we will respectively call R and W. So each program accepts inputs R and W, and outputs the probability that the next ball drawn is red:

- Case 1: return (5 - R)/(10 - R - W) # Number of red balls remaining / total balls remaining

- Case 2: return (R + 1)/(R + W + 2) # Laplace's Law of Succession

- Case 3: return 0.5

These programs are correct so far as they go. But unfortunately, probability theory does not operate on Python programs. Probability theory is an algebra of uncertainty, a calculus of credibility, and Python programs are not allowed in the formulas. It is like trying to add 3 to a toaster oven.

To use these programs in the probability calculus, we must figure out how to convert a Python program into a more convenient mathematical object - say, a probability distribution.

Suppose I want to know the combined probability that the sequence observed will be RWWRR, according to program 2 above. Program 2 does not have a direct faculty for returning the joint or combined probability of a sequence, but it is easy to extract anyway. First, I ask what probability program 2 assigns to observing R, given that no balls have been observed. Program 2 replies "1/2". Then I ask the probability that the next ball is R, given that one red ball has been observed; program 2 replies "2/3". The second ball is actually white, so the joint probability so far is 1/2 * 1/3 = 1/6. Next I ask for the probability that the third ball is red, given that the previous observation is RW; this is summarized as "one red and one white ball", and the answer is 1/2. The third ball is white, so the joint probability for RWW is 1/12. For the fourth ball, given the previous observation RWW, the probability of redness is 2/5, and the joint probability goes to 1/30. We can write this as p(RWWR|RWW) = 2/5, which means that if the sequence so far is RWW, the probability assigned by program 2 to the sequence continuing with R and forming RWWR equals 2/5. And then p(RWWRR|RWWR) = 1/2, and the combined probability is 1/60.

We can do this with every possible sequence of ten balls, and end up with a table of 1024 entries. This table of 1024 entries constitutes a probability distribution over sequences of observations of length 10, and it says everything the Python program had to say (about 10 or fewer observations, anyway). Suppose I have only this probability table, and I want to know the probability that the third ball is red, given that the first two balls drawn were white. I need only sum over the probability of all entries beginning with WWR, and divide by the probability of all entries beginning with WW.

We have thus transformed a program that computes the probability of future events given past experiences, into a probability distribution over sequences of observations.

You wouldn't want to do this in real life, because the Python program is ever so much more compact than a table with 1024 entries. The point is not that we can turn an efficient and compact computer program into a bigger and less efficient giant lookup table; the point is that we can view an inductive learner as a mathematical object, a distribution over sequences, which readily fits into standard probability calculus. We can take a computer program that reasons from experience and think about it using probability theory.

Why might this be convenient? Say that I'm not sure which of these three scenarios best describes the urn - I think it's about equally likely that each of the three cases holds true. How should I reason from my actual observations of the urn? If you think about the problem from the perspective of constructing a computer program that imitates my inferences, it looks complicated - we have to juggle the relative probabilities of each hypothesis, and also the probabilities within each hypothesis. If you think about it from the perspective of probability theory, the obvious thing to do is to add up all three distributions with weightings of 1/3 apiece, yielding a new distribution (which is in fact correct). Then the task is just to turn this new distribution into a computer program, which turns out not to be difficult.

So that is what a prior really is - a mathematical object that represents all of your starting information plus the way you learn from experience.

Referenced by: No Universally Compelling Arguments • Where Recursive Justification Hits Bottom • Probability is Subjectively Objective • Efficient Cross-Domain Optimization • Complexity and Intelligence • Recognizing Intelligence

Original with comments: Priors as Mathematical Objects

Lotteries: A Waste of Hope

Eliezer Yudkowsky, 13 April 2007 05:36AM

The classic criticism of the lottery is that the people who play are the ones who can least afford to lose; that the lottery is a sink of money, draining wealth from those who most need it. Some lottery advocates, and even some commentors on this blog, have tried to defend lottery-ticket buying as a rational purchase of fantasy - paying a dollar for a day's worth of pleasant anticipation, imagining yourself as a millionaire.

But consider exactly what this implies. It would mean that you're occupying your valuable brain with a fantasy whose real probability is nearly zero - a tiny line of likelihood which you, yourself, can do nothing to realize. The lottery balls will decide your future. The fantasy is of wealth that arrives without effort - without conscientiousness, learning, charisma, or even patience.

Which makes the lottery another kind of sink: a sink of emotional energy. It encourages people to invest their dreams, their hopes for a better future, into an infinitesimal probability. If not for the lottery, maybe they would fantasize about going to technical school, or opening their own business, or getting a promotion at work - things they might be able to actually do, hopes that would make them want to become stronger. Their dreaming brains might, in the 20th visualization of the pleasant fantasy, notice a way to really do it. Isn't that what dreams and brains are for? But how can such reality-limited fare compete with the artificially sweetened prospect of instant wealth - not after herding a dot-com startup through to IPO, but on Tuesday?

Seriously, why can't we just say that buying lottery tickets is stupid? Human beings are stupid, from time to time - it shouldn't be so surprising a hypothesis.

Unsurprisingly, the human brain doesn't do 64-bit floating-point arithmetic, and it can't devalue the emotional force of a pleasant anticipation by a factor of 0.00000001 without dropping the line of reasoning entirely. Unsurprisingly, many people don't realize that a numerical calculation of expected utility ought to override or replace their imprecise financial instincts, and instead treat the calculation as merely one argument to be balanced against their pleasant anticipations - an emotionally weak argument, since it's made up of mere squiggles on paper, instead of visions of fabulous wealth.

This seems sufficient to explain the popularity of lotteries. Why do so many arguers feel impelled to defend this classic form of self-destruction?

The process of overcoming bias requires (1) first noticing the bias, (2) analyzing the bias in detail, (3) deciding that the bias is bad, (4) figuring out a workaround, and then (5) implementing it. It's unfortunate how many people get through steps 1 and 2 and then bog down in step 3, which by rights should be the easiest of the five. Biases are lemons, not lemonade, and we shouldn't try to make lemonade out of them - just burn those lemons down.

Sequence: Challenging the Difficult

Guardians of Ayn Rand

New Improved Lottery

Referenced by: New Improved Lottery • Your Rationality is My Business • How Much Evidence Does It Take? • Perpetual Motion Beliefs • Bind Yourself to Reality • Timeless Identity • Living in Many Worlds • Trying to Try • Justified Expectation of Pleasant Surprises • Unteachable Excellence • Incremental Progress and the Valley

Original with comments: Lotteries: A Waste of Hope

New Improved Lottery

Eliezer Yudkowsky, 13 April 2007 11:42PM

People are still suggesting that the lottery is not a waste of hope, but a service which enables purchase of fantasy - "daydreaming about becoming a millionaire for much less money than daydreaming about hollywood stars in movies". One commenter wrote: "There is a big difference between zero chance of becoming wealthy, and epsilon. Buying a ticket allows your dream of riches to bridge that gap."

Actually, one of the points I was trying to make is that between zero chance of becoming wealthy, and epsilon chance, there is an order-of-epsilon difference. If you doubt this, let epsilon equal one over googolplex.

Anyway: If we pretend that the lottery sells epsilon hope, this suggests a design for a New Improved Lottery. The New Improved Lottery pays out every five years on average, at a random time - determined, say, by the decay of a not-very-radioactive element. You buy in once, for a single dollar, and get not just a few days of epsilon chance of becoming rich, but a few years of epsilon. Not only that, your wealth could strike at any time! At any minute, the phone could ring to inform you that you, yes, you are a millionaire!

Think of how much better this would be than an ordinary lottery drawing, which only takes place at defined times, a few times per week. Let's say the boss comes in and demands you rework a proposal, or restock inventory, or something similarly annoying. Instead of getting to work, you could turn to the phone and stare, hoping for that call - because there would be epsilon chance that, at that exact moment, you yes you would be awarded the Grand Prize! And even if it doesn't happen this minute, why, there's no need to be disappointed - it might happen the next minute!

Think of how many more fantasies this New Improved Lottery would enable. You could shop at the store, adding expensive items to your shopping cart - if your cellphone doesn't ring with news of a lottery win, you could always put the items back, right?

Maybe the New Improved Lottery could even show a constantly fluctuating probability distribution over the likelihood of a win occurring, and the likelihood of particular numbers being selected, with the overall expectation working out to the aforesaid Poisson distribution. Think of how much fun that would be! Oh, goodness, right this minute the chance of a win occurring is nearly ten times higher than usual! And look, the number 42 that I selected for the Mega Ball has nearly twice the usual chance of winning! You could feed it to a display on people's cellphones, so they could just flip open the cellphone and see their chances of winning. Think of how exciting that would be! Much more exciting than trying to balance your checkbook! Much more exciting than doing your homework! This new dream would be so much tastier that it would compete with, not only hopes of going to technical school, but even hopes of getting home from work early. People could just stay glued to the screen all day long, why, they wouldn't need to dream about anything else!

Yep, offering people tempting daydreams that will not actually happen sure is a valuable service, all right. People are willing to pay, it must be valuable. The alternative is that consumers are making mistakes, and we all know that can't happen.

And yet current governments, with their vile monopoly on lotteries, don't offer this simple and obvious service. Why? Because they want to overcharge people. They want them to spend money every week. They want them to spend a hundred dollars for the thrill of believing their chance of winning is a hundred times as large, instead of being able to stare at a cellphone screen waiting for the likelihood to spike. So if you believe that the lottery is a service, it is clearly an enormously overpriced service - charged to the poorest members of society - and it is your solemn duty as a citizen to demand the New Improved Lottery instead.

Sequence: Challenging the Difficult

Lotteries: A Waste of Hope

The Failures of Eld Science

Referenced by: But There's Still A Chance, Right? • Absolute Authority • The Allais Paradox • Living in Many Worlds • Dunbar's Function

Original with comments: New Improved Lottery

Your Rationality is My Business

Eliezer Yudkowsky, 15 April 2007 07:31AM

Some responses to Lotteries: A Waste of Hope chided me for daring to criticize others' decisions; if someone else chooses to buy lottery tickets, who am I to disagree? This is a special case of a more general question: What business is it of mine, if someone else chooses to believe what is pleasant rather than what is true? Can't we each choose for ourselves whether to care about the truth?

An obvious snappy comeback is: "Why do you care whether I care whether someone else cares about the truth?" It is somewhat inconsistent for your utility function to contain a negative term for anyone else's utility function having a term for someone else's utility function. But that is only a snappy comeback, not an answer.

So here then is my answer: I believe that it is right and proper for me, as a human being, to have an interest in the future, and what human civilization becomes in the future. One of those interests is the human pursuit of truth, which has strengthened slowly over the generations (for there was not always Science). I wish to strengthen that pursuit further, in this generation. That is a wish of mine, for the Future. For we are all of us players upon that vast gameboard, whether we accept the responsibility or not.

And that makes your rationality my business.

Is this a dangerous idea? Yes, and not just pleasantly edgy "dangerous". People have been burned to death because some priest decided that they didn't think the way they should. Deciding to burn people to death because they "don't think properly" - that's a revolting kind of reasoning, isn't it? You wouldn't want people to think that way, why, it's disgusting. People who think like that, well, we'll have to do something about them...

I agree! Here's my proposal: Let's argue against bad ideas but not set their bearers on fire.

The syllogism we desire to avoid runs: "I think Susie said a bad thing, therefore, Susie should be set on fire." Some try to avoid the syllogism by labeling it improper to think that Susie said a bad thing. No one should judge anyone, ever; anyone who judges is committing a terrible sin, and should be publicly pilloried for it.

As for myself, I deny the therefore. My syllogism runs, "I think Susie said something wrong, therefore, I will argue against what she said, but I will not set her on fire, or try to stop her from talking by violence or regulation..."

We are all of us players upon that vast gameboard; and one of my interests for the Future is to make the game fair. The counterintuitive idea underlying science is that factual disagreements should be fought out with experiments and mathematics, not violence and edicts. This incredible notion can be extended beyond science, to a fair fight for the whole Future. You should have to win by convincing people, and should not be allowed to burn them. This is one of the principles of Rationality, to which I have pledged my allegiance.

People who advocate relativism or selfishness do not appear to me to be truly relativistic or selfish. If they were really relativistic, they would not judge. If they were really selfish, they would get on with making money instead of arguing passionately with others. Rather, they have chosen the side of Relativism, whose goal upon that vast gameboard is to prevent the players - all the players - from making certain kinds of judgments. Or they have chosen the side of Selfishness, whose goal is to make all players selfish. And then they play the game, fairly or unfairly according to their wisdom.

If there are any true Relativists or Selfishes, we do not hear them - they remain silent, non-players.

I cannot help but care how you think, because - as I cannot help but see the universe - each time a human being turns away from the truth, the unfolding story of humankind becomes a little darker. In many cases, it is a small darkness only. (Someone doesn't always end up getting hurt.) Lying to yourself, in the privacy of your own thoughts, does not shadow humanity's history so much as telling public lies or setting people on fire. Yet there is a part of me which cannot help but mourn. And so long as I don't try to set you on fire - only argue with your ideas - I believe that it is right and proper to me, as a human, that I care about my fellow humans. That, also, is a position I defend into the Future.

Referenced by: Consolidated Nature of Morality Thread

Original with comments: Your Rationality is My Business

Consolidated Nature of Morality Thread

Eliezer Yudkowsky, 15 April 2007 11:00PM

My intended next OB post will, in passing, distinguish between moral judgments and factual beliefs. Several times before, this has sparked a debate about the nature of morality. (E.g., Believing in Todd.) Such debates often repeat themselves, reinvent the wheel each time, start all over from previous arguments. To avoid this, I suggest consolidating the debate. Whenever someone feels tempted to start a debate about the nature of morality in the comments thread of another post, the comment should be made to this post, instead, with an appropriate link to the article commented upon. Otherwise it does tend to take over discussions like kudzu. (This isn't the first blog/list where I've seen it happen.)

I'll start the ball rolling with ten points to ponder about the nature of morality...

1. It certainly looks like there is an important distinction between a statement like "The total loss of human life caused by World War II was roughly 72 million people" and "We ought to avoid a repeat of World War II." Anyone who argues that these statements are of the same fundamental kind must explain away the apparent structural differences between them. What are the exact structural differences?

2. We experience some of our morals and preferences as being voluntary choices, others as involuntary perceptions. I choose to play on the side of Rationality, but I don't think I could choose to believe that death is good any more than I could choose to believe the sky is green. What psychological factors account for these differences in my perceptions of my own preferences?

3. At a relatively young age, children begin to believe that while the teacher can make it all right to stand on your chair by giving permission, the teacher cannot make it all right to steal from someone else's backpack. (I can't recall the exact citation on this.) Do young children in a religious environment believe that God can make it all right to steal from someone's backpack?

4. Both individual human beings and civilizations appear to change at least some of their moral beliefs over the course of time. Some of these changes are experienced as "decisions", others are experienced as "discoveries". Is there a systematic direction to at least some of these changes? How does this systematic direction arise causally?

5. To paraphrase Alfred Tarski, the statement "My car is painted green" is true if and only if my car is painted green. Similarly, someone might try to get away with asserting that the statement "Human deaths are bad" is true if and only if human deaths are bad. Is this valid?

6. Suppose I involuntarily administered to you a potion which would cause you to believe that human deaths were good. Afterward, would you believe truly that human deaths were good, or would you believe falsely that human deaths were good?

7. Although the statement "My car is painted green" is presently false, I can make it true at a future time by painting my car green. However, I can think of no analogous action I could take which would make it right to kill people. Does this make the moral statement stronger, weaker, or is there no sense in making the comparison?

8. There does not appear to be any "place" in the environment where the referents of moral statements are stored, analogous to the place where my car is stored. Does this necessarily indicate that moral statements are empty of content, or could they correspond to something else? Is the statement 2 + 2 = 4 true? Could it be made untrue? Is it falsifiable? Where is its content?

9. The phrase "is/ought" gap refers to the notion that no ought statement can be logically derived from any number of is statements, without at least one ought statement in the mix. For example, suppose I have a remote control with two buttons, and the red button kills an innocent prisoner, and the green button sets them free. I cannot derive the ought-statement, "I ought not to press the red button", without both the is-statement "If I press the red button, an innocent will die" and the ought-statement "I ought not to kill innocents." Should we distinguish mixed ought-statements like "I ought not to press the red button" from pure ought-statements like "I ought not to kill innocents"? If so, is there really any such thing as a "pure" ought-statement, or do they all have is-statements mixed into them somewhere?

10. The statement "This painting is beautiful" could be rendered untrue by flinging a bucket of mud on the painting. Similarly, in the remote-control example above, the statement "It is wrong to press the red button" can be rendered untrue by rewiring the remote. Are there pure aesthetic judgments? Are there pure preferences?

Original with comments: Consolidated Nature of Morality Thread

Feeling Rational

Eliezer Yudkowsky, 26 April 2007 04:48AM

A popular belief about "rationality" is that rationality opposes all emotion - that all our sadness and all our joy are automatically anti-logical by virtue of being feelings. Yet strangely enough, I can't find any theorem of probability theory which proves that I should appear ice-cold and expressionless.

So is rationality orthogonal to feeling? No; our emotions arise from our models of reality. If I believe that my dead brother has been discovered alive, I will be happy; if I wake up and realize it was a dream, I will be sad. P. C. Hodgell said: "That which can be destroyed by the truth should be." My dreaming self's happiness was opposed by truth. My sadness on waking is rational; there is no truth which destroys it.

Rationality begins by asking how-the-world-is, but spreads virally to any other thought which depends on how we think the world is. By talking about your beliefs about "how-the-world-is", I mean anything you believe is out there in reality, anything that either does or does not exist, any member of the class "things that can make other things happen". If you believe that there is a goblin in your closet that ties your shoe's laces together, then this is a belief about how-the-world-is. Your shoes are real - you can pick them up. If there's something out there which can reach out and tie your shoelaces together, it must be real too, part of the vast web of causes and effects we call the "universe".

Feeling angry at the goblin who tied your shoelaces involves a state of mind that is not just about how-the-world-is. Suppose that, as a Buddhist or a lobotomy patient or just a very phlegmatic person, finding your shoelaces tied together didn't make you angry. This wouldn't affect what you expected to see in the world - you'd still expect to open up your closet and find your shoelaces tied together. Your anger or calm shouldn't affect your best guess here, because what happens in your closet does not depend on your emotional state of mind; though it may take some effort to think that clearly.

But the angry feeling is tangled up with a state of mind that is about how-the-world-is; you become angry because you think the goblin tied your shoelaces. The criterion of rationality spreads virally, from the initial question of whether or not a goblin tied your shoelaces, to the resulting anger.

Becoming more rational - arriving at better estimates of how-the-world-is - can diminish feelings or intensify them. Sometimes we run away from strong feelings by denying the facts, by flinching away from the view of the world that gave rise to the powerful emotion. If so, then as you study the skills of rationality and train yourself not to deny facts, your feelings will become stronger.

In my early days I was never quite certain whether it was all right to feel things strongly - whether it was allowed, whether it was proper. I do not think this confusion arose only from my youthful misunderstanding of rationality. I have observed similar troubles in people who do not even aspire to be rationalists; when they are happy, they wonder if they are really allowed to be happy, and when they are sad, they are never quite sure whether to run away from the emotion or not. Since the days of Socrates at least, and probably long before, the way to appear cultured and sophisticated has been to never let anyone see you care strongly about anything. It's embarrassing to feel - it's just not done in polite society. You should see the strange looks I get when people realize how much I care about rationality. It's not the unusual subject, I think, but that they're not used to seeing sane adults who visibly care about anything.

But I know, now, that there's nothing wrong with feeling strongly. Ever since I adopted the rule of "That which can be destroyed by the truth should be," I've also come to realize "That which the truth nourishes should thrive." When something good happens, I am happy, and there is no confusion in my mind about whether it is rational for me to be happy. When something terrible happens, I do not flee my sadness by searching for fake consolations and false silver linings. I visualize the past and future of humankind, the tens of billions of deaths over our history, the misery and fear, the search for answers, the trembling hands reaching upward out of so much blood, what we could become someday when we make the stars our cities, all that darkness and all that light - I know that I can never truly understand it, and I haven't the words to say. Despite all my philosophy I am still embarrassed to confess strong emotions, and you're probably uncomfortable hearing them. But I know, now, that it is rational to feel.

Sequence: Letting Go

The Importance of Saying "Oops"

Referenced by: Professing and Cheering • Semantic Stopsigns • What is Evidence? • The Lens That Sees Its Flaws • How to Convince Me That 2 + 2 = 3 • Cached Thoughts • Guardians of Ayn Rand • Effortless Technique • Zen and the Art of Rationality • On Expressing Your Concerns • The "Intuitions" Behind "Utilitarianism" • Bind Yourself to Reality • Is Humanism A Religion-Substitute? • Interpersonal Morality • Crisis of Faith • Which Parts Are "Me"? • Lawful Creativity • Singletons Rule OK • Cynical About Cynicism • What Do We Mean By "Rationality"? • Why Our Kind Can't Cooperate • Incremental Progress and the Valley

Original with comments: Feeling Rational

Universal Fire

Eliezer Yudkowsky, 27 April 2007 09:15PM

In L. Sprague de Camp's fantasy story The Incomplete Enchanter (which set the mold for the many imitations that followed), the hero, Harold Shea, is transported from our own universe into the universe of Norse mythology. This world is based on magic rather than technology; so naturally, when Our Hero tries to light a fire with a match brought along from Earth, the match fails to strike.

I realize it was only a fantasy story, but... how do I put this...

No.

In the late eighteenth century, Antoine-Laurent de Lavoisier discovered fire. "What?" you say. "Hasn't the use of fire been dated back for hundreds of thousands of years?" Well, yes, people used fire; it was hot, bright, sort of orangey-colored, and you could use it to cook things. But nobody knew how it worked. Greek and medieval alchemists thought that Fire was a basic thing, one of the Four Elements. In Lavoisier's time the alchemical paradigm had been gradually amended and greatly complicated, but fire was still held to be basic - in the form of "phlogiston", a rather mysterious substance which was said to explain fire, and also every other phenomenon in alchemy.

Lavoisier's great innovation was to weigh all the pieces of the chemical puzzle, both before and after the chemical reaction. It had previously been thought that some chemical transmutations changed the weight of the total material: If you subjected finely ground antimony to the focused sunlight of a burning glass, the antimony would be reduced to ashes after one hour, and the ashes would weigh one-tenth more than the original antimony - even though the burning had been accompanied by the loss of a thick white smoke. Lavoisier weighed all the components of such reactions, including the air in which the reaction took place, and discovered that matter was neither created nor destroyed. If the burnt ashes increased in weight, there was a corresponding decrease in the weight of the air.

Lavoisier also knew how to separate gases, and discovered that a burning candle diminished the amount of one kind of gas, vital air, and produced another gas, fixed air. Today we would call them oxygen and carbon dioxide. When the vital air was exhausted, the fire went out. One might guess, perhaps, that combustion transformed vital air into fixed air and fuel to ash, and that the ability of this transformation to continue was limited by the amount of vital air available.

Lavoisier's proposal directly contradicted the then-current phlogiston theory. That alone would have been shocking enough, but it also turned out...

To appreciate what comes next, you must put yourself into an eighteenth-century frame of mind. Forget the discovery of DNA, which occurred only in 1953. Unlearn the cell theory of biology, which was formulated in 1839. Imagine looking at your hand, flexing your fingers... and having absolutely no idea how it worked. The anatomy of muscle and bone was known, but no one had any notion of "what makes it go" - why a muscle moves and flexes, while clay molded into a similar shape just sits there. Imagine your own body being composed of mysterious, incomprehensible gloop. And then, imagine discovering...

...that humans, in the course of breathing, consumed vital air and breathed out fixed air. People also ran on combustion! Lavoisier measured the amount of heat that animals (and Lavoisier's assistant, Seguin) produced when exercising, the amount of vital air consumed, and the fixed air breathed out. When animals produced more heat, they consumed more vital air and exhaled more fixed air. People, like fire, consumed fuel and oxygen; people, like fire, produced heat and carbon dioxide. Deprive people of oxygen, or fuel, and the light goes out.

Matches catch fire because of phosphorus - "safety matches" have phosphorus on the ignition strip; strike-anywhere matches have phosphorus in the match heads. Phosphorus is highly reactive; pure phosphorus glows in the dark and may spontaneously combust. (Henning Brand, who purified phosphorus in 1669, announced that he had discovered Elemental Fire.) Phosphorus is thus also well-suited to its role in adenosine triphosphate, ATP, your body's chief method of storing chemical energy. ATP is sometimes called the "molecular currency". It invigorates your muscles and charges up your neurons. Almost every metabolic reaction in biology relies on ATP, and therefore on the chemical properties of phosphorus.

If a match stops working, so do you. You can't change just one thing.

The surface-level rules, "Matches catch fire when struck," and "Humans need air to breathe," are not obviously connected. It took centuries to discover the connection, and even then, it still seems like some distant fact learned in school, relevant only to a few specialists. It is all too easy to imagine a world where one surface rule holds, and the other doesn't; to suppress our credence in one belief, but not the other. But that is imagination, not reality. If your map breaks into four pieces for easy storage, it doesn't mean the territory is also broken into disconnected parts. Our minds store different surface-level rules in different compartments, but this does not reflect any division in the laws that govern Nature.

We can take the lesson further. Phosphorus derives its behavior from even deeper laws, electrodynamics and chromodynamics. "Phosphorus" is merely our word for electrons and quarks arranged a certain way. You cannot change the chemical properties of phosphorus without changing the laws governing electrons and quarks.

If you stepped into a world where matches failed to strike, you would cease to exist as organized matter.

Reality is laced together a lot more tightly than humans might like to believe.

Referenced by: Universal Law • Where to Draw the Boundary? • If You Demand Magic, Magic Won't Help • Amazing Breakthrough Day: April 1st

Original with comments: Universal Fire

Universal Law

Eliezer Yudkowsky, 29 April 2007 06:41AM

Followup to: Universal Fire

Antoine-Laurent de Lavoisier discovered that breathing (respiration) and fire (combustion) operated on the same principle. It was one of the most startling unifications in the history of science, for it brought together the mundane realm of matter and the sacred realm of life, which humans had divided into separate magisteria.

The first great simplification was that of Isaac Newton, who unified the course of the planets with the trajectory of a falling apple. The shock of this discovery was greater by far than Lavoisier's. It wasn't just that Newton had dared to unify the Earthly realm of base matter with the obviously different and sacred celestial realm, once thought to be the abode of the gods. Newton's discovery gave rise to the notion of a universal law, one that is the same everywhere and everywhen, with literally zero exceptions.

Human beings live in a world of surface phenomena, and surface phenomena are divided into leaky categories with plenty of exceptions. A tiger does not behave like a buffalo. Most buffalo have four legs, but perhaps this one has three. Why would anyone think there would be laws that hold everywhere? It's just so obviously untrue.

The only time when it seems like we would want a law to hold everywhere is when we are talking about moral laws - tribal rules of behavior. Some tribe members may try to take more than their fair share of the buffalo meat - perhaps coming up with some clever excuse - so in the case of moral laws we do seem to have an instinct to universality. Yes, the rule about dividing the meat evenly applies to you, right now, whether you like it or not. But even here there are exceptions. If - for some bizarre reason - a more powerful tribe threatened to spear all of you unless Bob received twice as much meat on just this one occasion, you'd give Bob twice as much meat. The idea of a rule with literally no exceptions seems insanely rigid, the product of closed-minded thinking by fanatics so in the grip of their one big idea that they can't see the richness and complexity of the real universe.

This is the customary accusation made against scientists - the professional students of the richness and complexity of the real universe. Because when you actually look at the universe, it turns out to be, by human standards, insanely rigid in applying its rules. As far as we know, there has been not one single violation of conservation of momentum from the uttermost dawn of time up until now.

Sometimes - very rarely - we observe an apparent violation of our models of the fundamental laws. Though our scientific models may last for a generation or two, they are not stable over the course of centuries... but do not fancy that this makes the universe itself whimsical. That is mixing up the map with the territory. For when the dust subsides and the old theory is overthrown, it turns out that the universe always was acting according to the new generalization we have discovered, which once again is absolutely universal as far as humanity's knowledge extends. When it was discovered that Newtonian gravitation was a special case of General Relativity, it was seen that General Relativity had been governing the orbit of Mercury for decades before any human being knew about it; and it would later become apparent that General Relativity had been governing the collapse of stars for billions of years before humanity. It is only our model that was mistaken - the Law itself was always absolutely constant - or so our new model tells us.

I may repose only 80% confidence that the lightspeed limit will last out the next hundred thousand years, but this does not mean that I think the lightspeed limit holds only 80% of the time, with occasional exceptions. The proposition to which I assign 80% probability is that the lightspeed law is absolutely inviolable throughout the entirety of space and time.

One of the reasons the ancient Greeks didn't discover science is that they didn't realize you could generalize from experiments. The Greek philosophers were interested in "normal" phenomena. If you set up a contrived experiment, you would probably get a "monstrous" result, one that had no implications for how things really worked.

So that is how humans tend to dream, before they learn better; but what of the universe's own quiet dreams that it dreamed to itself before ever it dreamed of humans? If you would learn to think like reality, then here is the Tao:

Since the beginning
not one unusual thing
has ever happened.

Referenced by: Think Like Reality • Is Molecular Nanotechnology "Scientific"? • No One Can Exempt You From Rationality's Laws • A Priori • Leaky Generalizations • Infinite Certainty • Is Reality Ugly? • Bind Yourself to Reality • Decoherent Essences • Timeless Physics • Timeless Beauty • Where Recursive Justification Hits Bottom • Mundane Magic

Original with comments: Universal Law

Think Like Reality

Eliezer Yudkowsky, 02 May 2007 06:36AM

Whenever I hear someone describe quantum physics as "weird" - whenever I hear someone bewailing the mysterious effects of observation on the observed, or the bizarre existence of nonlocal correlations, or the incredible impossibility of knowing position and momentum at the same time - then I think to myself: This person will never understand physics no matter how many books they read.

Reality has been around since long before you showed up. Don't go calling it nasty names like "bizarre" or "incredible". The universe was propagating complex amplitudes through configuration space for ten billion years before life ever emerged on Earth. Quantum physics is not "weird". You are weird. You have the absolutely bizarre idea that reality ought to consist of little billiard balls bopping around, when in fact reality is a perfectly normal cloud of complex amplitude in configuration space. This is your problem, not reality's, and you are the one who needs to change.

Human intuitions were produced by evolution and evolution is a hack. The same optimization process that built your retina backward and then routed the optic cable through your field of vision, also designed your visual system to process persistent objects bouncing around in 3 spatial dimensions because that's what it took to chase down tigers. But "tigers" are leaky surface generalizations - tigers came into existence gradually over evolutionary time, and they are not all absolutely similar to each other. When you go down to the fundamental level, the level on which the laws are stable, global, and exception-free, there aren't any tigers. In fact there aren't any persistent objects bouncing around in 3 spatial dimensions. Deal with it.

Calling reality "weird" keeps you inside a viewpoint already proven erroneous. Probability theory tells us that surprise is the measure of a poor hypothesis; if a model is consistently stupid - consistently hits on events the model assigns tiny probabilities - then it's time to discard that model. A good model makes reality look normal, not weird; a good model assigns high probability to that which is actually the case. Intuition is only a model by another name: poor intuitions are shocked by reality, good intuitions make reality feel natural. You want to reshape your intuitions so that the universe looks normal. You want to think like reality.

This end state cannot be forced. It is pointless to pretend that quantum physics feels natural to you when in fact it feels strange. This is merely denying your confusion, not becoming less confused. But it will also hinder you to keep thinking How bizarre! Spending emotional energy on incredulity wastes time you could be using to update. It repeatedly throws you back into the frame of the old, wrong viewpoint. It feeds your sense of righteous indignation at reality daring to contradict you.

The principle extends beyond physics. Have you ever caught yourself saying something like, "I just don't understand how a PhD physicist can believe in astrology?" Well, if you literally don't understand, this indicates a problem with your model of human psychology. Perhaps you are indignant - you wish to express strong moral disapproval. But if you literally don't understand, then your indignation is stopping you from coming to terms with reality. It shouldn't be hard to imagine how a PhD physicist ends up believing in astrology. People compartmentalize, enough said.

I now try to avoid using the English idiom "I just don't understand how..." to express indignation. If I genuinely don't understand how, then my model is being surprised by the facts, and I should discard it and find a better model.

Surprise exists in the map, not in the territory. There are no surprising facts, only models that are surprised by facts. Likewise for facts called such nasty names as "bizarre", "incredible", "unbelievable", "unexpected", "strange", "anomalous", or "weird". When you find yourself tempted by such labels, it may be wise to check if the alleged fact is really factual. But if the fact checks out, then the problem isn't the fact, it's you.

Referenced by: Beware the Unsurprised • Correspondence Bias • Are Your Enemies Innately Evil? • Guessing the Teacher's Password • Guardians of Ayn Rand • Bind Yourself to Reality • Zombies! Zombies? • Spooky Action at a Distance: The No-Communication Theorem • Mach's Principle: Anti-Epiphenomenal Physics • The Gift We Give To Tomorrow • Belief in Self-Deception

Original with comments: Think Like Reality

Beware the Unsurprised

Eliezer Yudkowsky, 03 May 2007 10:45PM

In Think Like Reality, I put forth the astonishing and controversial proposition that when human intuitions disagree with a fact, we need to either disprove the "fact" in question, or try to reshape the intuition. (Well, it wouldn't have been so controversial, but like a fool I picked quantum mechanics to illustrate the point. Never use quantum mechanics as an example of anything.) Probability theory says that a model which is consistently surprised on the data is probably not a very good model.

Matt Shulman pointed out in personal conversation that, in practice, we may want to be wary of people who don't appear surprised by surprising-seeming data. Some people affect to be unsurprised because it is a fakeable signal of competence. Well, a lot of things that good rationalists will do - such as appearing skeptical and appearing to take other people's opinions into account - are also fakeable signals of competence. But, in practice, Matt's point is still well-taken.

People may also appear unsurprised (Matt points out) if their models are so vague that they don't understand the implications one way or the other. (Rob Spear: "It doesn't matter to the general public whether reality has 11, 42, or 97.5 dimensions... The primary good that most modern physics provides to the people is basically light entertainment.") Or they may appear unsurprised if they fail to emotionally connect to the implications - "Oh, sure, an asteroid is going to hit Earth... but personally I don't think humanity really deserves to survive anyway... are you taking Sally to her doctor's appointment tomorrow?"

Or Cialdini on the bystander effect:

We can learn from the way the other witnesses are reacting whether the event is or is not an emergency. What is easy to forget, though, is that everybody else observing the event is likely to be looking for social evidence, too. Because we all prefer to appear poised and unflustered among others, we are likely to search for that evidence placidly, with brief, camouflaged glances at those around us. Therefore everyone is likely to see everyone else looking unruffled and failing to act.

So appearing unsurprised, or pretending to yourself that you weren't surprised, is both personally and socially detrimental. By saying that a consistently surprised model is a poor model, I didn't intend to make it more difficult for people to admit their surprise! Even rationalists are surprised sometimes - the important thing is to throw away the model, reshape your intuitions, and otherwise update yourself so that it doesn't happen again.

Think Like Reality wasn't arguing that we should never admit surprise, but that, having been surprised, we shouldn't get all indignant at reality for surprising us - because that just keeps us in the mistaken frame of mind that was surprised in the first place; instead, we should try to adjust our intuitions so that reality doesn't seem surprising the next time. That doesn't mean rationalizing the events in hindsight using your current model - hindsight bias is detrimental to this process because it leads you to underestimate how surprised you were, and hence adjust your model less than it needs to be adjusted.

Original with comments: Beware the Unsurprised

The Third Alternative

Eliezer Yudkowsky, 06 May 2007 11:47PM

"Believing in Santa Claus gives children a sense of wonder and encourages them to behave well in hope of receiving presents. If Santa-belief is destroyed by truth, the children will lose their sense of wonder and stop behaving nicely. Therefore, even though Santa-belief is false-to-fact, it is a Noble Lie whose net benefit should be preserved for utilitarian reasons."

Classically, this is known as a false dilemma, the fallacy of the excluded middle, or the package-deal fallacy. Even if we accept the underlying factual and moral premises of the above argument, it does not carry through. Even supposing that the Santa policy (encourage children to believe in Santa Claus) is better than the null policy (do nothing), it does not follow that Santa-ism is the best of all possible alternatives. Other policies could also supply children with a sense of wonder, such as taking them to watch a Space Shuttle launch or supplying them with science fiction novels. Likewise (if I recall correctly), offering children bribes for good behavior encourages the children to behave well only when adults are watching, while praise without bribes leads to unconditional good behavior.

Noble Lies are generally package-deal fallacies; and the response to a package-deal fallacy is that if we really need the supposed gain, we can construct a Third Alternative for getting it.

How can we obtain Third Alternatives? The first step in obtaining a Third Alternative is deciding to look for one, and the last step is the decision to accept it. This sounds obvious, and yet most people fail on these two steps, rather than within the search process. Where do false dilemmas come from? Some arise honestly, because superior alternatives are cognitively hard to see. But one factory for false dilemmas is justifying a questionable policy by pointing to a supposed benefit over the null action. In this case, the justifier does not want a Third Alternative; finding a Third Alternative would destroy the justification. The last thing a Santa-ist wants to hear is that praise works better than bribes, or that spaceships can be as inspiring as flying reindeer.

The best is the enemy of the good. If the goal is really to help people, then a superior alternative is cause for celebration - once we find this better strategy, we can help people more effectively. But if the goal is to justify a particular strategy by claiming that it helps people, a Third Alternative is an enemy argument, a competitor.

Modern cognitive psychology views decision-making as a search for alternatives. In real life, it's not enough to compare options, you have to generate the options in the first place. On many problems, the number of alternatives is huge, so you need a stopping criterion for the search. When you're looking to buy a house, you can't compare every house in the city; at some point you have to stop looking and decide.

But what about when our conscious motives for the search - the criteria we can admit to ourselves - don't square with subconscious influences? When we are carrying out an allegedly altruistic search, a search for an altruistic policy, and we find a strategy that benefits others but disadvantages ourselves - well, we don't stop looking there; we go on looking. Telling ourselves that we're looking for a strategy that brings greater altruistic benefit, of course. But suppose we find a policy that has some defensible benefit, and also just happens to be personally convenient? Then we stop the search at once! In fact, we'll probably resist any suggestion that we start looking again - pleading lack of time, perhaps. (And yet somehow, we always have cognitive resources for coming up with justifications for our current policy.)

Beware when you find yourself arguing that a policy is defensible rather than optimal; or that it has some benefit compared to the null action, rather than the best benefit of any action.

False dilemmas are often presented to justify unethical policies that are, by some vast coincidence, very convenient. Lying, for example, is often much more convenient than telling the truth; and believing whatever you started out with is more convenient than updating. Hence the popularity of arguments for Noble Lies; it serves as a defense of a pre-existing belief - one does not find Noble Liars who calculate an optimal new Noble Lie; they keep whatever lie they started with. Better stop that search fast!

To do better, ask yourself straight out: If I saw that there was a superior alternative to my current policy, would I be glad in the depths of my heart, or would I feel a tiny flash of reluctance before I let go? If the answers are "no" and "yes", beware that you may not have searched for a Third Alternative.

Which leads into another good question to ask yourself straight out: Did I spend five minutes with my eyes closed, brainstorming wild and creative options, trying to think of a better alternative? It has to be five minutes by the clock, because otherwise you blink - close your eyes and open them again - and say, "Why, yes, I searched for alternatives, but there weren't any." Blinking makes a good black hole down which to dump your duties. An actual, physical clock is recommended.

And those wild and creative options - were you careful not to think of a good one? Was there a secret effort from the corner of your mind to ensure that every option considered would be obviously bad?

It's amazing how many Noble Liars and their ilk are eager to embrace ethical violations - with all due bewailing of their agonies of conscience - when they haven't spent even five minutes by the clock looking for an alternative. There are some mental searches that we secretly wish would fail; and when the prospect of success is uncomfortable, people take the earliest possible excuse to give up.

Sequence: Overly Convenient Excuses

The Proper Use of Humility

Privileging the Hypothesis

Referenced by: Third Alternatives for Afterlife-ism • Belief as Attire • The Meditation on Curiosity • Priming and Contamination • How to Seem (and Be) Deep • Motivated Stopping and Motivated Continuation • A Case Study of Motivated Continuation • Fake Justification • The Tragedy of Group Selectionism • Crisis of Faith • Ethical Injunctions • Serious Stories • That Crisis thing seems pretty useful

Original with comments: The Third Alternative

Third Alternatives for Afterlife-ism

Eliezer Yudkowsky, 08 May 2007 07:41AM

One of the most commonly proposed Noble Lies is belief in an afterlife. Surely, goes the argument, the crushing certainty of absolute annihilation in a few decades is too much for any human being to bear. People need hope - if they don't believe in an afterlife, they won't be able to live.

Surely this must be the strongest of all arguments for Noble Lies. You can find Third Alternatives to many dilemmas, but can you find one to Death?

Well, did you close your eyes and think creatively about the problem for five minutes? No excuses, please; just answer "Yes" or "No". Did you, or did you not, brainstorm the problem for five minutes by the clock before giving up?

The assumed task is to find a source of hope against looming death. So at the very least I would cite medical nanotechnology, the argument from actuarial escape velocity, cryonics, or meddling with the forbidden ultimate technology. But do you think that anyone who actually argued for afterlife as a Noble Lie would be glad to hear about these Third Alternatives? No, because the point was not really to find the best strategy for supplying hope, but rather to excuse a fixed previous belief from criticism.

You can argue against the feasibility of one of the above Third Alternatives, or even argue against the feasibility of all of them, but that's not the point. Any one of those Third Alternatives stretches credulity less than a soul - that is (a) an imperishable dualistic stuff floating alongside the brain which (b) malfunctions exactly as the brain is neurologically damaged and yet (c) survives the brain's entire death. Even if we suppose the above Third Alternatives to be false-in-fact, they are packaged with far fewer associated absurdities, and put far less of a strain on the Standard Model.

Thus on the presentation of any one of these Third Alternatives, afterlife-ism stands immediately convicted because it cannot be the best strategy even as a Noble Lie. The old Noble Lie is dominated in the payoff table. If you decided to lie (to others or yourself) to soften the horror of personal extinction, then you'd nudge the balance of evidence a little on actuarial escape velocity - not spin up a soul from whole cloth.

(A truly fanatic rationalist - like me - would refuse to judge between these two lies, regarding them both as equal transgressions of the deontological commandments Thou Shalt Not Nudge Thy Probability Assignments and Thou Shalt Not Pursue Hope As An Emotion, Only Actual Positive Outcomes. Which is still no argument in favor of afterlife-ism; when a negative utility drops off my radar screen and becomes incomparable, I generally don't choose that policy.)

Original with comments: Third Alternatives for Afterlife-ism

Scope Insensitivity

Eliezer Yudkowsky, 14 May 2007 02:53AM

Once upon a time, three groups of subjects were asked how much they would pay to save 2000 / 20000 / 200000 migrating birds from drowning in uncovered oil ponds. The groups respectively answered $80, $78, and $88 [1]. This is scope insensitivity or scope neglect: the number of birds saved - the scope of the altruistic action - had little effect on willingness to pay.

Similar experiments showed that Toronto residents would pay little more to clean up all polluted lakes in Ontario than polluted lakes in a particular region of Ontario [2], or that residents of four western US states would pay only 28% more to protect all 57 wilderness areas in those states than to protect a single area [3].

People visualize "a single exhausted bird, its feathers soaked in black oil, unable to escape" [4]. This image, or prototype, calls forth some level of emotional arousal that is primarily responsible for willingness-to-pay - and the image is the same in all cases. As for scope, it gets tossed out the window - no human can visualize 2000 birds at once, let alone 200000. The usual finding is that exponential increases in scope create linear increases in willingness-to-pay - perhaps corresponding to the linear time for our eyes to glaze over the zeroes; this small amount of affect is added, not multiplied, with the prototype affect. This hypothesis is known as "valuation by prototype".

An alternative hypothesis is "purchase of moral satisfaction". People spend enough money to create a warm glow in themselves, a sense of having done their duty. The level of spending needed to purchase a warm glow depends on personality and financial situation, but it certainly has nothing to do with the number of birds.

We are insensitive to scope even when human lives are at stake: Increasing the alleged risk of chlorinated drinking water from 0.004 to 2.43 annual deaths per 1000 - a factor of 600 - increased willingness-to-pay from $3.78 to $15.23 [5]. Baron and Greene found no effect from varying lives saved by a factor of 10 [6].

A paper entitled Insensitivity to the value of human life: A study of psychophysical numbing collected evidence that our perception of human deaths follows Weber's Law - obeys a logarithmic scale where the "just noticeable difference" is a constant fraction of the whole. A proposed health program to save the lives of Rwandan refugees garnered far higher support when it promised to save 4,500 lives in a camp of 11,000 refugees, rather than 4,500 in a camp of 250,000. A potential disease cure had to promise to save far more lives in order to be judged worthy of funding, if the disease was originally stated to have killed 290,000 rather than 160,000 or 15,000 people per year. [7]

The moral: If you want to be an effective altruist, you have to think it through with the part of your brain that processes those unexciting inky zeroes on paper, not just the part that gets real worked up about that poor struggling oil-soaked bird.

[1] Desvousges, W. Johnson, R. Dunford, R. Boyle, K. J. Hudson, S. and Wilson K. N. (1992). Measuring non-use damages using contingent valuation: experimental evaluation accuracy. Research Triangle Institute Monograph 92-1.

[2] Kahneman, D. 1986. Comments on the contingent valuation method. Pp. 185-194 in Valuing environmental goods: a state of the arts assessment of the contingent valuation method, eds. R. G. Cummings, D. S. Brookshire and W. D. Schulze. Totowa, NJ: Roweman and Allanheld.

[3] McFadden, D. and Leonard, G. 1995. Issues in the contingent valuation of environmental goods: methodologies for data collection and analysis. In Contingent valuation: a critical assessment, ed. J. A. Hausman. Amsterdam: North Holland.

[4] Kahneman, D., Ritov, I. and Schkade, D. A. 1999. Economic Preferences or Attitude Expressions?: An Analysis of Dollar Responses to Public Issues, Journal of Risk and Uncertainty, 19: 203-235.

[5] Carson, R. T. and Mitchell, R. C. 1995. Sequencing and Nesting in Contingent Valuation Surveys. Journal of Environmental Economics and Management, 28(2): 155-73.

[6] Baron, J. and Greene, J. 1996. Determinants of insensitivity to quantity in valuation of public goods: contribution, warm glow, budget constraints, availability, and prominence. Journal of Experimental Psychology: Applied, 2: 107-125.

[7] Fetherstonhaugh, D., Slovic, P., Johnson, S. and Friedrich, J. 1997. Insensitivity to the value of human life: A study of psychophysical numbing. Journal of Risk and Uncertainty, 14: 238-300.

Referenced by: One Life Against the World • Making History Available • Human Evil and Muddled Thinking • The Meditation on Curiosity • Motivated Stopping and Motivated Continuation • Fake Selfishness • Zut Allais! • The "Intuitions" Behind "Utilitarianism" • Timeless Identity • Interpersonal Entanglement • Higher Purpose • ...And Say No More Of It • Can Humanism Match Religion's Output? • Rationality: Common Interest of Many Causes • Purchase Fuzzies and Utilons Separately • Practical Advice Backed By Deep Theories • The End (of Sequences) • The Lifespan Dilemma

Original with comments: Scope Insensitivity

One Life Against the World

Eliezer Yudkowsky, 18 May 2007 10:06PM

Followup to: Scope Insensitivity

"Whoever saves a single life, it is as if he had saved the whole world."

-- The Talmud, Sanhedrin 4:5

It's a beautiful thought, isn't it? Feel that warm glow.

I can testify that helping one person feels just as good as helping the whole world. Once upon a time, when I was burned out for the day and wasting time on the Internet - it's a bit complicated, but essentially, I managed to turn someone's whole life around by leaving an anonymous blog comment. I wasn't expecting it to have an effect that large, but it did. When I discovered what I had accomplished, it gave me a tremendous high. The euphoria lasted through that day and into the night, only wearing off somewhat the next morning. It felt just as good (this is the scary part) as the euphoria of a major scientific insight, which had previously been my best referent for what it might feel like to do drugs.

Saving one life probably does feel just as good as being the first person to realize what makes the stars shine. It probably does feel just as good as saving the entire world.

But if you ever have a choice, dear reader, between saving a single life and saving the whole world - then save the world. Please. Because beyond that warm glow is one heck of a gigantic difference.

For some people, the notion that saving the world is significantly better than saving one human life will be obvious, like saying that six billion dollars is worth more than one dollar, or that six cubic kilometers of gold weighs more than one cubic meter of gold. (And never mind the expected value of posterity.) Why might it not be obvious? Well, suppose there's a qualitative duty to save what lives you can - then someone who saves the world, and someone who saves one human life, are just fulfilling the same duty. Or suppose that we follow the Greek conception of personal virtue, rather than consequentialism; someone who saves the world is virtuous, but not six billion times as virtuous as someone who saves one human life. Or perhaps the value of one human life is already too great to comprehend - so that the passing grief we experience at funerals is an infinitesimal underestimate of what is lost - and thus passing to the entire world changes little.

I agree that one human life is of unimaginably high value. I also hold that two human lives are twice as unimaginably valuable. Or to put it another way: Whoever saves one life, if it is as if they had saved the whole world; whoever saves ten lives, it is as if they had saved ten worlds. Whoever actually saves the whole world - not to be confused with pretend rhetorical saving the world - it is as if they had saved an intergalactic civilization.

Two deaf children are sleeping on the railroad tracks, the train speeding down; you see this, but you are too far away to save the child. I'm nearby, within reach, so I leap forward and drag one child off the railroad tracks - and then stop, calmly sipping a Diet Pepsi as the train bears down on the second child. "Quick!" you scream to me. "Do something!" But (I call back) I already saved one child from the train tracks, and thus I am "unimaginably" far ahead on points. Whether I save the second child, or not, I will still be credited with an "unimaginably" good deed. Thus, I have no further motive to act. Doesn't sound right, does it?

Why should it be any different if a philanthropist spends $10 million on curing a rare but spectacularly fatal disease which afflicts only a hundred people planetwide, when the same money has an equal probability of producing a cure for a less spectacular disease that kills 10% of 100,000 people? I don't think it is different. When human lives are at stake, we have a duty to maximize, not satisfice; and this duty has the same strength as the original duty to save lives. Whoever knowingly chooses to save one life, when they could have saved two - to say nothing of a thousand lives, or a world - they have damned themselves as thoroughly as any murderer.

Addendum: It's not cognitively easy to spend money to save lives, since cliche methods that instantly leap to mind don't work or are counterproductive. (I will post later on why this tends to be so.) Stuart Armstrong also points out that if we are to disdain the philanthropist who spends life-saving money inefficiently, we should be consistent and disdain more those who could spend money to save lives but don't.

Referenced by: Fake Selfishness • Superhero Bias • Mere Messiahs • Passing the Recursive Buck • Rationality: Common Interest of Many Causes

Original with comments: One Life Against the World

Risk-Free Bonds Aren't

Eliezer Yudkowsky, 22 June 2007 10:30PM

I've always been annoyed by the term "risk-free bonds rate", meaning the return on US Treasury bills. Just because US bonds have not defaulted within their trading experience, people assume this is impossible? A list of major governments in 1900 would probably put the Ottoman Empire or Austria-Hungary well ahead of the relatively young United States. Citing the good track record of the US alone, and not all governments of equal apparent stability at the start of the same time period, is purest survivorship bias.

The United States is a democracy; if enough people vote for representatives who decide not to pay off the bonds, they won't get paid. Do you want to look at recent history, let alone ancient history, and tell me this is impossible? The Internet could enable coordinated populist voting that would sweep new candidates into office, in defiance of prevous political machines. Then the US economy melts under the burden of consumer debt, which causes China to stop buying US bonds and dump its dollar reserves. Then Al Qaeda finally smuggles a nuke into Washington, D.C. Then the next global pandemic hits. And these are just "good stories" - the probability of the US defaulting on its bonds for any reason, is necessarily higher than the probability of it happening for the particular reasons I've just described. I'm not saying these are high probabilities, but they are probabilities. Treasury bills are nowhere near "risk free".

I may be prejudiced here, because I anticipate particular Black Swans (AI, nanotech, biotech) that I see as having a high chance of striking over the lifetime of a 30-year Treasury bond. But even if you don't share those particular assumptions, do you expect the United States to still be around in 300 years? If not, do you know exactly when it will go bust? Then why isn't the risk of losing your capital on a 30-year Treasury bond at least, say, 10%?

Nassim Nicholas Taleb's latest, The Black Swan, is about the impact of unknown unknowns - sudden blowups, processes that seem to behave normally for long periods and then melt down, variables in which most of the movement may occur on a tiny fraction of the moves. Taleb inveighs against the dangers of induction, the ludic fallacy, hindsight, survivorship bias. And then on page 205, Taleb suggests:

Instead of putting your money in "medium risk" investments (how do you know it is medium risk? by listening to tenure-seeking "experts"?), you need to put a portion, say 85 to 90 percent, in extremely safe instruments, like Treasury bills - as safe a class of instruments as you can manage to find on this planet. The remaining 10 to 15 percent you put in extremely speculative bets, as leveraged as possible (like options), preferably venture capital-style portfolios. That way you do not depend on errors of risk management; no Black Swan can hurt you at all, beyond your "floor", the nest egg that you have in maximally safe instruments.

Does Taleb know something I don't, or has he forgotten to apply his own principles in the heat of the moment? (That's a serious question, by the way, if Taleb happens to be reading this. I'm not an experienced trader, and Taleb undoubtedly knows more than I do about how to use Black Swan thinking in trading. But we all know how hard it is to remember to apply our finely honed skepticism in the face of handy popular phrases like "risk-free bonds rate".) Regardless, I think that if you advise your readers to invest 90% of their money in "extremely safe" instruments, you should certainly also warn that it had better not all go into the same instrument - no, not even Treasury bills or gold bullion. There is always risk management, and you are always exposed to error. The safest instruments you can find on this planet aren't very safe.

Referenced by: Against Discount Rates • Markets are Anti-Inductive

Original with comments: Risk-Free Bonds Aren't

Correspondence Bias

Eliezer Yudkowsky, 25 June 2007 12:58AM

The correspondence bias is the tendency to draw inferences about a person's unique and enduring dispositions from behaviors that can be entirely explained by the situations in which they occur.
-- Gilbert and Malone

We tend to see far too direct a correspondence between others' actions and personalities. When we see someone else kick a vending machine for no visible reason, we assume they are "an angry person". But when you yourself kick the vending machine, it's because the bus was late, the train was early, your report is overdue, and now the damned vending machine has eaten your lunch money for the second day in a row. Surely, you think to yourself, anyone would kick the vending machine, in that situation.

We attribute our own actions to our situations, seeing our behaviors as perfectly normal responses to experience. But when someone else kicks a vending machine, we don't see their past history trailing behind them in the air. We just see the kick, for no reason we know about, and we think this must be a naturally angry person - since they lashed out without any provocation.

Yet consider the prior probabilities. There are more late buses in the world, than mutants born with unnaturally high anger levels that cause them to sometimes spontaneously kick vending machines. Now the average human is, in fact, a mutant. If I recall correctly, an average individual has 2-10 somatically expressed mutations. But any given DNA location is very unlikely to be affected. Similarly, any given aspect of someone's disposition is probably not very far from average. To suggest otherwise is to shoulder a burden of improbability.

Even when people are informed explicitly of situational causes, they don't seem to properly discount the observed behavior. When subjects are told that a pro-abortion or anti-abortion speaker was randomly assigned to give a speech on that position, subjects still think the speakers harbor leanings in the direction randomly assigned. (Jones and Harris 1967, "The attribution of attitudes.)

It seems quite intuitive to explain rain by water spirits; explain fire by a fire-stuff (phlogiston) escaping from burning matter; explain the soporific effect of a medication by saying that it contains a "dormitive potency". Reality usually involves more complicated mechanisms: an evaporation and condensation cycle underlying rain, oxidizing combustion underlying fire, chemical interactions with the nervous system for soporifics. But mechanisms sound more complicated than essences; they are harder to think of, less available. So when someone kicks a vending machine, we think they have an innate vending-machine-kicking-tendency.

Unless the "someone" who kicks the machine is us - in which case we're behaving perfectly normally, given our situations; surely anyone else would do the same. Indeed, we overestimate how likely others are to respond the same way we do - the "false consensus effect". Drinking students considerably overestimate the fraction of fellow students who drink, but nondrinkers considerably underestimate the fraction. The "fundamental attribution error" refers to our tendency to overattribute others' behaviors to their dispositions, while reversing this tendency for ourselves.

To understand why people act the way they do, we must first realize that everyone sees themselves as behaving normally. Don't ask what strange, mutant disposition they were born with, which directly corresponds to their surface behavior. Rather, ask what situations people see themselves as being in. Yes, people do have dispositions - but there are not enough heritable quirks of disposition to directly account for all the surface behaviors you see.

Suppose I gave you a control with two buttons, a red button and a green button. The red button destroys the world, and the green button stops the red button from being pressed. Which button would you press? The green one. Anyone who gives a different answer is probably overcomplicating the question.

And yet people sometimes ask me why I want to save the world. Like I must have had a traumatic childhood or something. Really, it seems like a pretty obvious decision... if you see the situation in those terms.

I may have non-average views which call for explanation - why do I believe such things, when most people don't? - but given those beliefs, my reaction doesn't seem to call forth an exceptional explanation. Perhaps I am a victim of false consensus; perhaps I overestimate how many people would press the green button if they saw the situation in those terms. But y'know, I'd still bet there'd be at least a substantial minority.

Most people see themselves as perfectly normal, from the inside. Even people you hate, people who do terrible things, are not exceptional mutants. No mutations are required, alas. When you understand this, you are ready to stop being surprised by human events.

Sequence: Politics is the Mind-Killer

The Scales of Justice, the Notebook of Rationality

Are Your Enemies Innately Evil?

Referenced by: Are Your Enemies Innately Evil? • Say Not "Complexity" • My Wild and Reckless Youth • Failing to Learn from History • Why is the Future So Absurd? • Radical Honesty • No One Knows What Science Doesn't Know • Every Cause Wants To Be A Cult • Guardians of Ayn Rand • Cultish Countercultishness • Einstein's Superpowers

Original with comments: Correspondence Bias

Are Your Enemies Innately Evil?

Eliezer Yudkowsky, 26 June 2007 09:13PM

Followup to: Correspondence Bias

As previously discussed, we see far too direct a correspondence between others' actions and their inherent dispositions. We see unusual dispositions that exactly match the unusual behavior, rather than asking after real situations or imagined situations that could explain the behavior. We hypothesize mutants.

When someone actually offends us - commits an action of which we (rightly or wrongly) disapprove - then, I observe, the correspondence bias redoubles. There seems to be a very strong tendency to blame evil deeds on the Enemy's mutant, evil disposition. Not as a moral point, but as a strict question of prior probability, we should ask what the Enemy might believe about their situation which would reduce the seeming bizarrity of their behavior. This would allow us to hypothesize a less exceptional disposition, and thereby shoulder a lesser burden of improbability.

On September 11th, 2001, nineteen Muslim males hijacked four jet airliners in a deliberately suicidal effort to hurt the United States of America. Now why do you suppose they might have done that? Because they saw the USA as a beacon of freedom to the world, but were born with a mutant disposition that made them hate freedom?

Realistically, most people don't construct their life stories with themselves as the villains. Everyone is the hero of their own story. The Enemy's story, as seen by the Enemy, is not going to make the Enemy look bad. If you try to construe motivations that would make the Enemy look bad, you'll end up flat wrong about what actually goes on in the Enemy's mind.

But politics is the mind-killer. Debate is war; arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the opposing side; otherwise it's like stabbing your soldiers in the back.

If the Enemy did have an evil disposition, that would be an argument in favor of your side. And any argument that favors your side must be supported, no matter how silly - otherwise you're letting up the pressure somewhere on the battlefront. Everyone strives to outshine their neighbor in patriotic denunciation, and no one dares to contradict. Soon the Enemy has horns, bat wings, flaming breath, and fangs that drip corrosive venom. If you deny any aspect of this on merely factual grounds, you are arguing the Enemy's side; you are a traitor. Very few people will understand that you aren't defending the Enemy, just defending the truth.

If it took a mutant to do monstrous things, the history of the human species would look very different. Mutants would be rare.

Or maybe the fear is that understanding will lead to forgiveness. It's easier to shoot down evil mutants. It is a more inspiring battle cry to scream, "Die, vicious scum!" instead of "Die, people who could have been just like me but grew up in a different environment!" You might feel guilty killing people who weren't pure darkness.

This looks to me like the deep-seated yearning for a one-sided policy debate in which the best policy has no drawbacks. If an army is crossing the border or a lunatic is coming at you with a knife, the policy alternatives are (a) defend yourself (b) lie down and die. If you defend yourself, you may have to kill. If you kill someone who could, in another world, have been your friend, that is a tragedy. And it is a tragedy. The other option, lying down and dying, is also a tragedy. Why must there be a non-tragic option? Who says that the best policy available must have no downside? If someone has to die, it may as well be the initiator of force, to discourage future violence and thereby minimize the total sum of death.

If the Enemy has an average disposition, and is acting from beliefs about their situation that would make violence a typically human response, then that doesn't mean their beliefs are factually accurate. It doesn't mean they're justified. It means you'll have to shoot down someone who is the hero of their own story, and in their novel the protagonist will die on page 80. That is a tragedy, but it is better than the alternative tragedy. It is the choice that every police officer makes, every day, to keep our neat little worlds from dissolving into chaos.

When you accurately estimate the Enemy's psychology - when you know what is really in the Enemy's mind - that knowledge won't feel like landing a delicious punch on the opposing side. It won't give you a warm feeling of righteous indignation. It won't make you feel good about yourself. If your estimate makes you feel unbearably sad, you may be seeing the world as it really is. More rarely, an accurate estimate may send shivers of serious horror down your spine, as when dealing with true psychopaths, or neurologically intact people with beliefs that have utterly destroyed their sanity (Scientologists or Jesus Camp).

So let's come right out and say it - the 9/11 hijackers weren't evil mutants. They did not hate freedom. They, too, were the heroes of their own stories, and they died for what they believed was right - truth, justice, and the Islamic way. If the hijackers saw themselves that way, it doesn't mean their beliefs were true. If the hijackers saw themselves that way, it doesn't mean that we have to agree that what they did was justified. If the hijackers saw themselves that way, it doesn't mean that the passengers of United Flight 93 should have stood aside and let it happen. It does mean that in another world, if they had been raised in a different environment, those hijackers might have been police officers. And that is indeed a tragedy. Welcome to Earth.

Sequence: Politics is the Mind-Killer

Correspondence Bias

The Robbers Cave Experiment

Referenced by: Belief as Attire • Terminal Values and Instrumental Values • When None Dare Urge Restraint • Cultish Countercultishness • Raised in Technophilia • Crisis of Faith • Why Does Power Corrupt? • Traditional Capitalist Values • Dark Side Epistemology • Inner Goodness • Bayesians vs. Barbarians • The Craft and the Community

Original with comments: Are Your Enemies Innately Evil?

Open Thread

Eliezer Yudkowsky, 01 July 2007 07:38PM

By request of the community, an Open Thread for free-form comments, so long as they're still related to the basic project of this blog.

A word on post requests: You're free to ask, but the authors can't commit to posting on requested topics - it's hard enough to do the ones we have in mind already.

Original with comments: Open Thread

Two More Things to Unlearn from School

Eliezer Yudkowsky, 12 July 2007 05:45PM

In Three Things to Unlearn from School, Ben Casnocha cites Bill Bullard's list of three bad habits of thought: Attaching importance to personal opinions, solving given problems, and earning the approval of others. Bullard's proposed alternatives don't look very good to me, but Bullard has surely identified some important problems.

I can think of other school-inculcated bad habits of thought, too many to list, but I'll name two of my least favorite.

I suspect the most dangerous habit of thought taught in schools is that even if you don't really understand something, you should parrot it back anyway. One of the most fundamental life skills is realizing when you are confused, and school actively destroys this ability - teaches students that they "understand" when they can successfully answer questions on an exam, which is very very very far from absorbing the knowledge and making it a part of you. Students learn the habit that eating consists of putting food into mouth; the exams can't test for chewing or swallowing, and so they starve.

Much of this problem may come from needing to take three 4-credit courses per quarter, with a textbook chapter plus homework to be done every week - the courses are timed for frantic memorization, it's not possible to deeply chew over and leisurely digest knowledge in the same period. College students aren't allowed to be confused; if they started saying, "Wait, do I really understand this? Maybe I'd better spend a few days looking up related papers, or consult another textbook," they'd fail all the courses they took that quarter. A month later they would understand the material far better and remember it much longer - but one month after finals is too late; it counts for nothing in the lunatic university utility function.

Many students who have gone through this process no longer even realize when something confuses them, or notice gaps in their understanding. They have been trained out of pausing to think.

I recall reading, though I can't remember where, that physicists in some country were more likely to become extreme religious fanatics. This confused me, until the author suggested that physics students are presented with a received truth that is actually correct, from which they learn the habit of trusting authority.

It may be dangerous to present people with a giant mass of authoritative knowledge, especially if it is actually true. It may damage their skepticism.

So what could you do? Teach students the history of physics, how each idea was replaced in turn by a new correct one? "Here's the old idea, here's the new idea, here's the experiment - the new idea wins!" Repeat this lesson ten times and what is the habit of thought learned? "New ideas always win; every new idea in physics turns out to be correct." You still haven't taught any critical thinking, because you only showed them history as seen with perfect hindsight. You've taught them the habit that distinguishing true ideas from false ones is perfectly clear-cut and straightforward, so if a shiny new idea has anything to recommend it, it's probably true.

Maybe it would be possible to teach the history of physics from a historically realistic point of view, without benefit of hindsight: show students the different alternatives that were considered historically plausible, re-enact the historical disagreements and debates.

Maybe you could avoid handing students knowledge on a silver platter: show students different versions of physics equations that looked plausible, and ask them to figure out which was the correct one, or invent experiments that would distinguish between alternatives. This wouldn't be as challenging as needing to notice anomalies without hints and invent alternatives from scratch, but it would be a vast improvement over memorizing a received authority.

Then, perhaps, you could teach the habit of thought: "The ideas of received authority are often imperfect but it takes a great effort to find a new idea that is better. Most possible changes are for the worse, even though every improvement is necessarily a change."

Referenced by: Focus Your Uncertainty • Guessing the Teacher's Password

Original with comments: Two More Things to Unlearn from School

Making Beliefs Pay Rent (in Anticipated Experiences)

Eliezer Yudkowsky, 28 July 2007 10:59PM

Thus begins the ancient parable:

If a tree falls in a forest and no one hears it, does it make a sound? One says, "Yes it does, for it makes vibrations in the air." Another says, "No it does not, for there is no auditory processing in any brain."

Suppose that, after the tree falls, the two walk into the forest together. Will one expect to see the tree fallen to the right, and the other expect to see the tree fallen to the left? Suppose that before the tree falls, the two leave a sound recorder next to the tree. Would one, playing back the recorder, expect to hear something different from the other? Suppose they attach an electroencephalograph to any brain in the world; would one expect to see a different trace than the other? Though the two argue, one saying "No," and the other saying "Yes," they do not anticipate any different experiences. The two think they have different models of the world, but they have no difference with respect to what they expect will happen to them.

It's tempting to try to eliminate this mistake class by insisting that the only legitimate kind of belief is an anticipation of sensory experience. But the world does, in fact, contain much that is not sensed directly. We don't see the atoms underlying the brick, but the atoms are in fact there. There is a floor beneath your feet, but you don't experience the floor directly; you see the light reflected from the floor, or rather, you see what your retina and visual cortex have processed of that light. To infer the floor from seeing the floor is to step back into the unseen causes of experience. It may seem like a very short and direct step, but it is still a step.

You stand on top of a tall building, next to a grandfather clock with an hour, minute, and ticking second hand. In your hand is a bowling ball, and you drop it off the roof. On which tick of the clock will you hear the crash of the bowling ball hitting the ground?

To answer precisely, you must use beliefs like Earth's gravity is 9.8 meters per second per second, and This building is around 120 meters tall. These beliefs are not wordless anticipations of a sensory experience; they are verbal-ish, propositional. It probably does not exaggerate much to describe these two beliefs as sentences made out of words. But these two beliefs have an inferential consequence that is a direct sensory anticipation - if the clock's second hand is on the 12 numeral when you drop the ball, you anticipate seeing it on the 1 numeral when you hear the crash five seconds later. To anticipate sensory experiences as precisely as possible, we must process beliefs that are not anticipations of sensory experience.

It is a great strength of Homo sapiens that we can, better than any other species in the world, learn to model the unseen. It is also one of our great weak points. Humans often believe in things that are not only unseen but unreal.

The same brain that builds a network of inferred causes behind sensory experience, can also build a network of causes that is not connected to sensory experience, or poorly connected. Alchemists believed that phlogiston caused fire - we could oversimply their minds by drawing a little node labeled "Phlogiston", and an arrow from this node to their sensory experience of a crackling campfire - but this belief yielded no advance predictions; the link from phlogiston to experience was always configured after the experience, rather than constraining the experience in advance. Or suppose your postmodern English professor teaches you that the famous writer Wulky Wilkinsen is actually a "post-utopian". What does this mean you should expect from his books? Nothing. The belief, if you can call it that, doesn't connect to sensory experience at all. But you had better remember the propositional assertion that "Wulky Wilkinsen" has the "post-utopian" attribute, so you can regurgitate it on the upcoming quiz. Likewise if "post-utopians" show "colonial alienation"; if the quiz asks whether Wulky Wilkinsen shows colonial alienation, you'd better answer yes. The beliefs are connected to each other, though still not connected to any anticipated experience.

We can build up whole networks of beliefs that are connected only to each other - call these "floating" beliefs. It is a uniquely human flaw among animal species, a perversion of Homo sapiens's ability to build more general and flexible belief networks.

The rationalist virtue of empiricism consists of constantly asking which experiences our beliefs predict - or better yet, prohibit. Do you believe that phlogiston is the cause of fire? Then what do you expect to see happen, because of that? Do you believe that Wulky Wilkinsen is a post-utopian? Then what do you expect to see because of that? No, not "colonial alienation"; what experience will happen to you? Do you believe that if a tree falls in the forest, and no one hears it, it still makes a sound? Then what experience must therefore befall you?

It is even better to ask: what experience must not happen to you? Do you believe that elan vital explains the mysterious aliveness of living beings? Then what does this belief not allow to happen - what would definitely falsify this belief? A null answer means that your belief does not constrain experience; it permits anything to happen to you. It floats.

When you argue a seemingly factual question, always keep in mind which difference of anticipation you are arguing about. If you can't find the difference of anticipation, you're probably arguing about labels in your belief network - or even worse, floating beliefs, barnacles on your network. If you don't know what experiences are implied by Wulky Wilkinsen being a post-utopian, you can go on arguing forever. (You can also publish papers forever.)

Above all, don't ask what to believe - ask what to anticipate. Every question of belief should flow from a question of anticipation, and that question of anticipation should be the center of the inquiry. Every guess of belief should begin by flowing to a specific guess of anticipation, and should continue to pay rent in future anticipations. If a belief turns deadbeat, evict it.

Sequence: Mysterious Answers to Mysterious Questions

Belief in Belief

Referenced by: Belief in Belief • Belief as Attire • Religion's Claim to be Non-Disprovable • The Proper Use of Doubt • Your Strength as a Rationalist • Fake Explanations • Fake Causality • Mysterious Answers to Mysterious Questions • Fake Optimization Criteria • Truly Part Of You • Purpose and Pragmatism • Is Reality Ugly? • Feel the Meaning • Replace the Symbol with the Substance • Dissolving the Question • Quantum Explanations • Which Basis Is More Fundamental? • Decoherent Essences • Do Scientists Already Know This Stuff? • Timeless Physics • Possibility and Could-ness • Can Counterfactuals Be True? • Setting Up Metaethics • The Comedy of Behaviorism • No Logical Positivist I • Belief in Intelligence • Raising the Sanity Waterline • Go Forth and Create the Art! • Practical Advice Backed By Deep Theories

Original with comments: Making Beliefs Pay Rent (in Anticipated Experiences)

Belief in Belief

Eliezer Yudkowsky, 29 July 2007 05:49PM

Followup to: Making Beliefs Pay Rent (in Anticipated Experiences)

Carl Sagan once told a parable of a man who comes to us and claims: "There is a dragon in my garage." Fascinating! We reply that we wish to see this dragon - let us set out at once for the garage! "But wait," the claimant says to us, "it is an invisible dragon."

Now as Sagan points out, this doesn't make the hypothesis unfalsifiable. Perhaps we go to the claimant's garage, and although we see no dragon, we hear heavy breathing from no visible source; footprints mysteriously appear on the ground; and instruments show that something in the garage is consuming oxygen and breathing out carbon dioxide.

But now suppose that we say to the claimant, "Okay, we'll visit the garage and see if we can hear heavy breathing," and the claimant quickly says no, it's an inaudible dragon. We propose to measure carbon dioxide in the air, and the claimant says the dragon does not breathe. We propose to toss a bag of flour into the air to see if it outlines an invisible dragon, and the claimant immediately says, "The dragon is permeable to flour."

Carl Sagan used this parable to illustrate the classic moral that poor hypotheses need to do fast footwork to avoid falsification. But I tell this parable to make a different point: The claimant must have an accurate model of the situation somewhere in his mind, because he can anticipate, in advance, exactly which experimental results he'll need to excuse.

Some philosophers have been much confused by such scenarios, asking, "Does the claimant really believe there's a dragon present, or not?" As if the human brain only had enough disk space to represent one belief at a time! Real minds are more tangled than that. As discussed in yesterday's post, there are different types of belief; not all beliefs are direct anticipations. The claimant clearly does not anticipate seeing anything unusual upon opening the garage door; otherwise he wouldn't make advance excuses. It may also be that the claimant's pool of propositional beliefs contains There is a dragon in my garage. It may seem, to a rationalist, that these two beliefs should collide and conflict even though they are of different types. Yet it is a physical fact that you can write "The sky is green!" next to a picture of a blue sky without the paper bursting into flames.

The rationalist virtue of empiricism is supposed to prevent us from this class of mistake. We're supposed to constantly ask our beliefs which experiences they predict, make them pay rent in anticipation. But the dragon-claimant's problem runs deeper, and cannot be cured with such simple advice. It's not exactly difficult to connect belief in a dragon to anticipated experience of the garage. If you believe there's a dragon in your garage, then you can expect to open up the door and see a dragon. If you don't see a dragon, then that means there's no dragon in your garage. This is pretty straightforward. You can even try it with your own garage.

No, this invisibility business is a symptom of something much worse.

Depending on how your childhood went, you may remember a time period when you first began to doubt Santa Claus's existence, but you still believed that you were supposed to believe in Santa Claus, so you tried to deny the doubts. As Daniel Dennett observes, where it is difficult to believe a thing, it is often much easier to believe that you ought to believe it. What does it mean to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green? The statement is confusing; it's not even clear what it would mean to believe it - what exactly would be believed, if you believed. You can much more easily believe that it is proper, that it is good and virtuous and beneficial, to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green. Dennett calls this "belief in belief".

And here things become complicated, as human minds are wont to do - I think even Dennett oversimplifies how this psychology works in practice. For one thing, if you believe in belief, you cannot admit to yourself that you only believe in belief, because it is virtuous to believe, not to believe in belief, and so if you only believe in belief, instead of believing, you are not virtuous. Nobody will admit to themselves, "I don't believe the Ultimate Cosmic Sky is blue and green, but I believe I ought to believe it" - not unless they are unusually capable of acknowledging their own lack of virtue. People don't believe in belief in belief, they just believe in belief.

(Those who find this confusing may find it helpful to study mathematical logic, which trains one to make very sharp distinctions between the proposition P, a proof of P, and a proof that P is provable. There are similarly sharp distinctions between P, wanting P, believing P, wanting to believe P, and believing that you believe P.)

There's different kinds of belief in belief. You may believe in belief explicitly; you may recite in your deliberate stream of consciousness the verbal sentence "It is virtuous to believe that the Ultimate Cosmic Sky is perfectly blue and perfectly green." (While also believing that you believe this, unless you are unusually capable of acknowledging your own lack of virtue.) But there's also less explicit forms of belief in belief. Maybe the dragon-claimant fears the public ridicule that he imagines will result if he publicly confesses he was wrong (although, in fact, a rationalist would congratulate him, and others are more likely to ridicule him if he goes on claiming there's a dragon in his garage). Maybe the dragon-claimant flinches away from the prospect of admitting to himself that there is no dragon, because it conflicts with his self-image as the glorious discoverer of the dragon, who saw in his garage what all others had failed to see.

If all our thoughts were deliberate verbal sentences like philosophers manipulate, the human mind would be a great deal easier for humans to understand. Fleeting mental images, unspoken flinches, desires acted upon without acknowledgement - these account for as much of ourselves as words.

While I disagree with Dennett on some details and complications, I still think that Dennett's notion of belief in belief is the key insight necessary to understand the dragon-claimant. But we need a wider concept of belief, not limited to verbal sentences. "Belief" should include unspoken anticipation-controllers. "Belief in belief" should include unspoken cognitive-behavior-guiders. It is not psychologically realistic to say "The dragon-claimant does not believe there is a dragon in his garage; he believes it is beneficial to believe there is a dragon in his garage." But it is realistic to say the dragon-claimant anticipates as if there is no dragon in his garage, and makes excuses as if he believed in the belief.

You can possess an ordinary mental picture of your garage, with no dragons in it, which correctly predicts your experiences on opening the door, and never once think the verbal phrase There is no dragon in my garage. I even bet it's happened to you - that when you open your garage door or bedroom door or whatever, and expect to see no dragons, no such verbal phrase runs through your mind.

And to flinch away from giving up your belief in the dragon - or flinch away from giving up your self-image as a person who believes in the dragon - it is not necessary to explicitly think I want to believe there's a dragon in my garage. It is only necessary to flinch away from the prospect of admitting you don't believe.

To correctly anticipate, in advance, which experimental results shall need to be excused, the dragon-claimant must (a) possess an accurate anticipation-controlling model somewhere in his mind, and (b) act cognitively to protect either (b1) his free-floating propositional belief in the dragon or (b2) his self-image of believing in the dragon.

If someone believes in their belief in the dragon, and also believes in the dragon, the problem is much less severe. They will be willing to stick their neck out on experimental predictions, and perhaps even agree to give up the belief if the experimental prediction is wrong - although belief in belief can still interfere with this, if the belief itself is not absolutely confident. When someone makes up excuses in advance, it would seem to require that belief, and belief in belief, have become unsynchronized.

Sequence: Mysterious Answers to Mysterious Questions

Making Beliefs Pay Rent (in Anticipated Experiences)

Bayesian Judo

Sequence: Against Doublethink

Singlethink

Referenced by: Bayesian Judo • Professing and Cheering • Belief as Attire • Religion's Claim to be Non-Disprovable • The Proper Use of Doubt • Absence of Evidence Is Evidence of Absence • Update Yourself Incrementally • Scientific Evidence, Legal Evidence, Rational Evidence • Fake Explanations • The Futility of Emergence • Doublethink (Choosing to be Biased) • How to Seem (and Be) Deep • Motivated Stopping and Motivated Continuation • Fake Justification • Fake Optimization Criteria • Mere Messiahs • Resist the Happy Death Spiral • The Fallacy of Gray • Taboo Your Words • Replace the Symbol with the Substance • Qualitatively Confused • Fake Reductionism • Angry Atoms • Zombies! Zombies? • Do Scientists Already Know This Stuff? • The Fear of Common Knowledge • Setting Up Metaethics • The Bedrock of Morality: Arbitrary? • Beyond the Reach of God • Dark Side Epistemology • Prolegomena to a Theory of Fun • The Uses of Fun (Theory) • Seduced by Imagination • No, Really, I've Deceived Myself • Belief in Self-Deception • Moore's Paradox • Don't Believe You'll Self-Deceive

Original with comments: Belief in Belief

Bayesian Judo

Eliezer Yudkowsky, 31 July 2007 05:53AM

You can have some fun with people whose anticipations get out of sync with what they believe they believe.

I was once at a dinner party, trying to explain to a man what I did for a living, when he said: "I don't believe Artificial Intelligence is possible because only God can make a soul."

At this point I must have been divinely inspired, because I instantly responded: "You mean if I can make an Artificial Intelligence, it proves your religion is false?"

He said, "What?"

I said, "Well, if your religion predicts that I can't possibly make an Artificial Intelligence, then, if I make an Artificial Intelligence, it means your religion is false. Either your religion allows that it might be possible for me to build an AI; or, if I build an AI, that disproves your religion."

There was a pause, as the one realized he had just made his hypothesis vulnerable to falsification, and then he said, "Well, I didn't mean that you couldn't make an intelligence, just that it couldn't be emotional in the same way we are."

I said, "So if I make an Artificial Intelligence that, without being deliberately preprogrammed with any sort of script, starts talking about an emotional life that sounds like ours, that means your religion is wrong."

He said, "Well, um, I guess we may have to agree to disagree on this."

I said: "No, we can't, actually. There's a theorem of rationality called Aumann's Agreement Theorem which shows that no two rationalists can agree to disagree. If two people disagree with each other, at least one of them must be doing something wrong."

We went back and forth on this briefly. Finally, he said, "Well, I guess I was really trying to say that I don't think you can make something eternal."

I said, "Well, I don't think so either! I'm glad we were able to reach agreement on this, as Aumann's Agreement Theorem requires." I stretched out my hand, and he shook it, and then he wandered away.

A woman who had stood nearby, listening to the conversation, said to me gravely, "That was beautiful."

"Thank you very much," I said.

Sequence: Mysterious Answers to Mysterious Questions

Belief in Belief

Professing and Cheering

Referenced by: Fake Explanations • Why Are Individual IQ Differences OK? • Decoherence as Projection

Original with comments: Bayesian Judo

Professing and Cheering

Eliezer Yudkowsky, 02 August 2007 07:20AM

I once attended a panel on the topic, "Are science and religion compatible?" One of the women on the panel, a pagan, held forth interminably upon how she believed that the Earth had been created when a giant primordial cow was born into the primordial abyss, who licked a primordial god into existence, whose descendants killed a primordial giant and used its corpse to create the Earth, etc. The tale was long, and detailed, and more absurd than the Earth being supported on the back of a giant turtle. And the speaker clearly knew enough science to know this.

I still find myself struggling for words to describe what I saw as this woman spoke. She spoke with... pride? Self-satisfaction? A deliberate flaunting of herself?

The woman went on describing her creation myth for what seemed like forever, but was probably only five minutes. That strange pride/satisfaction/flaunting clearly had something to do with her knowing that her beliefs were scientifically outrageous. And it wasn't that she hated science; as a panelist she professed that religion and science were compatible. She even talked about how it was quite understandable that the Vikings talked about a primordial abyss, given the land in which they lived - explained away her own religion! - and yet nonetheless insisted this was what she "believed", said with peculiar satisfaction.

I'm not sure that Daniel Dennett's concept of "belief in belief" stretches to cover this event. It was weirder than that. She didn't recite her creation myth with the fanatical faith of someone who needs to reassure herself. She didn't act like she expected us, the audience, to be convinced - or like she needed our belief to validate her.

Dennett, in addition to suggesting belief in belief, has also suggested that much of what is called "religious belief" should really be studied as "religious profession". Suppose an alien anthropologist studied a group of postmodernist English students who all seemingly believed that Wulky Wilkensen was a post-utopian author. The appropriate question may not be "Why do the students all believe this strange belief?" but "Why do they all write this strange sentence on quizzes?" Even if a sentence is essentially meaningless, you can still know when you are supposed to chant the response aloud.

I think Dennett may be slightly too cynical in suggesting that religious profession is just saying the belief aloud - most people are honest enough that, if they say a religious statement aloud, they will also feel obligated to say the verbal sentence into their own stream of consciousness.

But even the concept of "religious profession" doesn't seem to cover the pagan woman's claim to believe in the primordial cow. If you had to profess a religious belief to satisfy a priest, or satisfy a co-religionist - heck, to satisfy your own self-image as a religious person - you would have to pretend to believe much more convincingly than this woman was doing. As she recited her tale of the primordial cow, with that same strange flaunting pride, she wasn't even trying to be persuasive - wasn't even trying to convince us that she took her own religion seriously. I think that's the part that so took me aback. I know people who believe they believe ridiculous things, but when they profess them, they'll spend much more effort to convince themselves that they take their beliefs seriously.

It finally occurred to me that this woman wasn't trying to convince us or even convince herself. Her recitation of the creation story wasn't about the creation of the world at all. Rather, by launching into a five-minute diatribe about the primordial cow, she was cheering for paganism, like holding up a banner at a football game. A banner saying "GO BLUES" isn't a statement of fact, or an attempt to persuade; it doesn't have to be convincing - it's a cheer.

That strange flaunting pride... it was like she was marching naked in a gay pride parade. (Incidentally, I'd have no objection if she had marched naked in a gay pride parade. Lesbianism is not something that truth can destroy.) It wasn't just a cheer, like marching, but an outrageous cheer, like marching naked - believing that she couldn't be arrested or criticized, because she was doing it for her pride parade.

That's why it mattered to her that what she was saying was beyond ridiculous. If she'd tried to make it sound more plausible, it would have been like putting on clothes.

Sequence: Mysterious Answers to Mysterious Questions

Bayesian Judo

Belief as Attire

Referenced by: Belief as Attire • The Proper Use of Doubt • Fake Explanations • Science as Attire • Semantic Stopsigns • My Wild and Reckless Youth • Fake Optimization Criteria • Absolute Authority • Expecting Beauty • Dissolving the Question • Excluding the Supernatural • Disjunctions, Antipredictions, Etc.

Original with comments: Professing and Cheering

Belief as Attire

Eliezer Yudkowsky, 02 August 2007 05:13PM

I have so far distinguished between belief as anticipation-controller, belief in belief, professing and cheering. Of these, we might call anticipation-controlling beliefs "proper beliefs" and the other forms "improper belief". A proper belief can be wrong or irrational, e.g., someone who genuinely anticipates that prayer will cure her sick baby, but the other forms are arguably "not belief at all".

Yet another form of improper belief is belief as group-identification - as a way of belonging. Robin Hanson uses the excellent metaphor of wearing unusual clothing, a group uniform like a priest's vestments or a Jewish skullcap, and so I will call this "belief as attire".

In terms of humanly realistic psychology, the Muslims who flew planes into the World Trade Center undoubtedly saw themselves as heroes defending truth, justice, and the Islamic Way from hideous alien monsters a la the movie Independence Day. Only a very inexperienced nerd, the sort of nerd who has no idea how non-nerds see the world, would say this out loud in an Alabama bar. It is not an American thing to say. The American thing to say is that the terrorists "hate our freedom" and that flying a plane into a building is a "cowardly act". You cannot say the phrases "heroic self-sacrifice" and "suicide bomber" in the same sentence, even for the sake of accurately describing how the Enemy sees the world. The very concept of the courage and altruism of a suicide bomber is Enemy attire - you can tell, because the Enemy talks about it. The cowardice and sociopathy of a suicide bomber is American attire. There are no quote marks you can use to talk about how the Enemy sees the world; it would be like dressing up as a Nazi for Halloween.

Belief-as-attire may help explain how people can be passionate about improper beliefs. Mere belief in belief, or religious professing, would have some trouble creating genuine, deep, powerful emotional effects. Or so I suspect; I confess I'm not an expert here. But my impression is this: People who've stopped anticipating-as-if their religion is true, will go to great lengths to convince themselves they are passionate, and this desperation can be mistaken for passion. But it's not the same fire they had as a child.

On the other hand, it is very easy for a human being to genuinely, passionately, gut-level belong to a group, to cheer for their favorite sports team. (This is the foundation on which rests the swindle of "Republicans vs. Democrats" and analogous false dilemmas in other countries, but that's a topic for another post.) Identifying with a tribe is a very strong emotional force. People will die for it. And once you get people to identify with a tribe, the beliefs which are attire of that tribe will be spoken with the full passion of belonging to that tribe.

Sequence: Mysterious Answers to Mysterious Questions

Professing and Cheering

Focus Your Uncertainty

Referenced by: The Proper Use of Doubt • Fake Explanations • Guessing the Teacher's Password • Science as Attire • The Futility of Emergence • Fake Optimization Criteria • A Sense That More Is Possible

Original with comments: Belief as Attire

Religion's Claim to be Non-Disprovable

Eliezer Yudkowsky, 04 August 2007 03:21AM

The earliest account I know of a scientific experiment is, ironically, the story of Elijah and the priests of Baal.

The people of Israel are wavering between Jehovah and Baal, so Elijah announces that he will conduct an experiment to settle it - quite a novel concept in those days! The priests of Baal will place their bull on an altar, and Elijah will place Jehovah's bull on an altar, but neither will be allowed to start the fire; whichever God is real will call down fire on His sacrifice. The priests of Baal serve as control group for Elijah - the same wooden fuel, the same bull, and the same priests making invocations, but to a false god. Then Elijah pours water on his altar - ruining the experimental symmetry, but this was back in the early days - to signify deliberate acceptance of the burden of proof, like needing a 0.05 significance level. The fire comes down on Elijah's altar, which is the experimental observation. The watching people of Israel shout "The Lord is God!" - peer review.

And then the people haul the 450 priests of Baal down to the river Kishon and slit their throats. This is stern, but necessary. You must firmly discard the falsified hypothesis, and do so swiftly, before it can generate excuses to protect itself. If the priests of Baal are allowed to survive, they will start babbling about how religion is a separate magisterium which can be neither proven nor disproven.

Back in the old days, people actually believed their religions instead of just believing in them. The biblical archaeologists who went in search of Noah's Ark did not think they were wasting their time; they anticipated they might become famous. Only after failing to find confirming evidence - and finding disconfirming evidence in its place - did religionists execute what William Bartley called the retreat to commitment, "I believe because I believe."

Back in the old days, there was no concept of religion being a separate magisterium. The Old Testament is a stream-of-consciousness culture dump: history, law, moral parables, and yes, models of how the universe works. In not one single passage of the Old Testament will you find anyone talking about a transcendent wonder at the complexity of the universe. But you will find plenty of scientific claims, like the universe being created in six days (which is a metaphor for the Big Bang), or rabbits chewing their cud. (Which is a metaphor for...)

Back in the old days, saying the local religion "could not be proven" would have gotten you burned at the stake. One of the core beliefs of Orthodox Judaism is that God appeared at Mount Sinai and said in a thundering voice, "Yeah, it's all true." From a Bayesian perspective that's some darned unambiguous evidence of a superhumanly powerful entity. (Albeit it doesn't prove that the entity is God per se, or that the entity is benevolent - it could be alien teenagers.) The vast majority of religions in human history - excepting only those invented extremely recently - tell stories of events that would constitute completely unmistakable evidence if they'd actually happened. The orthogonality of religion and factual questions is a recent and strictly Western concept. The people who wrote the original scriptures didn't even know the difference.

The Roman Empire inherited philosophy from the ancient Greeks; imposed law and order within its provinces; kept bureaucratic records; and enforced religious tolerance. The New Testament, created during the time of the Roman Empire, bears some traces of modernity as a result. You couldn't invent a story about God completely obliterating the city of Rome (a la Sodom and Gomorrah), because the Roman historians would call you on it, and you couldn't just stone them.

In contrast, the people who invented the Old Testament stories could make up pretty much anything they liked. Early Egyptologists were genuinely shocked to find no trace whatsoever of Hebrew tribes having ever been in Egypt - they weren't expecting to find a record of the Ten Plagues, but they expected to find something. As it turned out, they did find something. They found out that, during the supposed time of the Exodus, Egypt ruled much of Canaan. That's one huge historical error, but if there are no libraries, nobody can call you on it.

The Roman Empire did have libraries. Thus, the New Testament doesn't claim big, showy, large-scale geopolitical miracles as the Old Testament routinely did. Instead the New Testament claims smaller miracles which nonetheless fit into the same framework of evidence. A boy falls down and froths at the mouth; the cause is an unclean spirit; an unclean spirit could reasonably be expected to flee from a true prophet, but not to flee from a charlatan; Jesus casts out the unclean spirit; therefore Jesus is a true prophet and not a charlatan. This is perfectly ordinary Bayesian reasoning, if you grant the basic premise that epilepsy is caused by demons (and that the end of an epileptic fit proves the demon fled).

Not only did religion used to make claims about factual and scientific matters, religion used to make claims about everything. Religion laid down a code of law - before legislative bodies; religion laid down history - before historians and archaeologists; religion laid down the sexual morals - before Women's Lib; religion described the forms of government - before constitutions; and religion answered scientific questions from biological taxonomy to the formation of stars. The Old Testament doesn't talk about a sense of wonder at the complexity of the universe - it was busy laying down the death penalty for women who wore men's clothing, which was solid and satisfying religious content of that era. The modern concept of religion as purely ethical derives from every other area having been taken over by better institutions. Ethics is what's left.

Or rather, people think ethics is what's left. Take a culture dump from 2,500 years ago. Over time, humanity will progress immensely, and pieces of the ancient culture dump will become ever more glaringly obsolete. Ethics has not been immune to human progress - for example, we now frown upon such Bible-approved practices as keeping slaves. Why do people think that ethics is still fair game?

Intrinsically, there's nothing small about the ethical problem with slaughtering thousands of innocent first-born male children to convince an unelected Pharaoh to release slaves who logically could have been teleported out of the country. It should be more glaring than the comparatively trivial scientific error of saying that grasshoppers have four legs. And yet, if you say the Earth is flat, people will look at you like you're crazy. But if you say the Bible is your source of ethics, women will not slap you. Most people's concept of rationality is determined by what they think they can get away with; they think they can get away with endorsing Bible ethics; and so it only requires a manageable effort of self-deception for them to overlook the Bible's moral problems. Everyone has agreed not to notice the elephant in the living room, and this state of affairs can sustain itself for a time.

Maybe someday, humanity will advance further, and anyone who endorses the Bible as a source of ethics will be treated the same way as Trent Lott endorsing Strom Thurmond's presidential campaign. And then it will be said that religion's "true core" has always been genealogy or something.

The idea that religion is a separate magisterium which cannot be proven or disproven is a Big Lie - a lie which is repeated over and over again, so that people will say it without thinking; yet which is, on critical examination, simply false. It is a wild distortion of how religion happened historically, of how all scriptures present their beliefs, of what children are told to persuade them, and of what the majority of religious people on Earth still believe. You have to admire its sheer brazenness, on a par with Oceania has always been at war with Eastasia. The prosecutor whips out the bloody axe, and the defendant, momentarily shocked, thinks quickly and says: "But you can't disprove my innocence by mere evidence - it's a separate magisterium!"

And if that doesn't work, grab a piece of paper and scribble yourself a Get-Out-of-Jail-Free card.

Referenced by: Conservation of Expected Evidence • Update Yourself Incrementally • Fake Explanations • The Lens That Sees Its Flaws • Cached Thoughts • Why Are Individual IQ Differences OK? • Fake Optimization Criteria • Resist the Happy Death Spiral • The Genetic Fallacy • Changing Your Metaethics • Excluding the Supernatural • Crisis of Faith • Dark Side Epistemology • Can Humanism Match Religion's Output? • That Crisis thing seems pretty useful • Atheism = Untheism + Antitheism • Why I'm Staying On Bloggingheads.tv

Original with comments: Religion's Claim to be Non-Disprovable

The Importance of Saying "Oops"

Eliezer Yudkowsky, 05 August 2007 03:17AM

I just finished reading a history of Enron's downfall, The Smartest Guys in the Room, which hereby wins my award for "Least Appropriate Book Title".

An unsurprising feature of Enron's slow rot and abrupt collapse was that the executive players never admitted to having made a large mistake. When catastrophe #247 grew to such an extent that it required an actual policy change, they would say "Too bad that didn't work out - it was such a good idea - how are we going to hide the problem on our balance sheet?" As opposed to, "It now seems obvious in retrospect that it was a mistake from the beginning." As opposed to, "I've been stupid." There was never a watershed moment, a moment of humbling realization, of acknowledging a fundamental problem. After the bankruptcy, Jeff Skilling, the former COO and brief CEO of Enron, declined his own lawyers' advice to take the Fifth Amendment; he testified before Congress that Enron had been a great company.

Not every change is an improvement, but every improvement is necessarily a change. If we only admit small local errors, we will only make small local changes. The motivation for a big change comes from acknowledging a big mistake.

As a child I was raised on equal parts science and science fiction, and from Heinlein to Feynman I learned the tropes of Traditional Rationality: Theories must be bold and expose themselves to falsification; be willing to commit the heroic sacrifice of giving up your own ideas when confronted with contrary evidence; play nice in your arguments; try not to deceive yourself; and other fuzzy verbalisms.

A traditional rationalist upbringing tries to produce arguers who will concede to contrary evidence eventually - there should be some mountain of evidence sufficient to move you. This is not trivial; it distinguishes science from religion. But there is less focus on speed, on giving up the fight as quickly as possible, integrating evidence efficiently so that it only takes a minimum of contrary evidence to destroy your cherished belief.

I was raised in Traditional Rationality, and thought myself quite the rationalist. I switched to Bayescraft (Laplace/Jaynes/Tversky/Kahneman) in the aftermath of... well, it's a long story. Roughly, I switched because I realized that Traditional Rationality's fuzzy verbal tropes had been insufficient to prevent me from making a large mistake.

After I had finally and fully admitted my mistake, I looked back upon the path that had led me to my Awful Realization. And I saw that I had made a series of small concessions, minimal concessions, grudgingly conceding each millimeter of ground, realizing as little as possible of my mistake on each occasion, admitting failure only in small tolerable nibbles. I could have moved so much faster, I realized, if I had simply screamed "OOPS!"

And I thought: I must raise the level of my game.

There is a powerful advantage to admitting you have made a large mistake. It's painful. It can also change your whole life.

It is important to have the watershed moment, the moment of humbling realization. To acknowledge a fundamental problem, not divide it into palatable bite-size mistakes.

Do not indulge in drama and become proud of admitting errors. It is surely superior to get it right the first time. But if you do make an error, better by far to see it all at once. Even hedonically, it is better to take one large loss than many small ones. The alternative is stretching out the battle with yourself over years. The alternative is Enron.

Since then I have watched others making their own series of minimal concessions, grudgingly conceding each millimeter of ground; never confessing a global mistake where a local one will do; always learning as little as possible from each error. What they could fix in one fell swoop voluntarily, they transform into tiny local patches they must be argued into. Never do they say, after confessing one mistake, I've been a fool. They do their best to minimize their embarrassment by saying I was right in principle, or It could have worked, or I still want to embrace the true essence of whatever-I'm-attached-to. Defending their pride in this passing moment, they ensure they will again make the same mistake, and again need to defend their pride.

Better to swallow the entire bitter pill in one terrible gulp.

Sequence: Letting Go

Feeling Rational

The Crackpot Offer

Referenced by: The Crackpot Offer • If Many-Worlds Had Come First • A Genius for Destruction • Fighting a Rearguard Action Against the Truth • Crisis of Faith • The Sacred Mundane • That Crisis thing seems pretty useful • Unspeakable Morality

Original with comments: The Importance of Saying "Oops"

Focus Your Uncertainty

Eliezer Yudkowsky, 05 August 2007 08:49PM

Will bond yields go up, or down, or remain the same? If you're a TV pundit and your job is to explain the outcome after the fact, then there's no reason to worry. No matter which of the three possibilities comes true, you'll be able to explain why the outcome perfectly fits your pet market theory . There's no reason to think of these three possibilities as somehow opposed to one another, as exclusive, because you'll get full marks for punditry no matter which outcome occurs.

But wait! Suppose you're a novice TV pundit, and you aren't experienced enough to make up plausible explanations on the spot. You need to prepare remarks in advance for tomorrow's broadcast, and you have limited time to prepare. In this case, it would be helpful to know which outcome will actually occur - whether bond yields will go up, down, or remain the same - because then you would only need to prepare one set of excuses.

Alas, no one can possibly foresee the future. What are you to do? You certainly can't use "probabilities". We all know from school that "probabilities" are little numbers that appear next to a word problem, and there aren't any little numbers here. Worse, you feel uncertain. You don't remember feeling uncertain while you were manipulating the little numbers in word problems. College classes teaching math are nice clean places, therefore math itself can't apply to life situations that aren't nice and clean. You wouldn't want to inappropriately transfer thinking skills from one context to another. Clearly, this is not a matter for "probabilities".

Nonetheless, you only have 100 minutes to prepare your excuses. You can't spend the entire 100 minutes on "up", and also spend all 100 minutes on "down", and also spend all 100 minutes on "same". You've got to prioritize somehow.

If you needed to justify your time expenditure to a review committee, you would have to spend equal time on each possibility. Since there are no little numbers written down, you'd have no documentation to justify spending different amounts of time. You can hear the reviewers now: And why, Mr. Finkledinger, did you spend exactly 42 minutes on excuse #3? Why not 41 minutes, or 43? Admit it - you're not being objective! You're playing subjective favorites!

But, you realize with a small flash of relief, there's no review committee to scold you. This is good, because there's a major Federal Reserve announcement tomorrow, and it seems unlikely that bond prices will remain the same. You don't want to spend 33 precious minutes on an excuse you don't anticipate needing.

Your mind keeps drifting to the explanations you use on television, of why each event plausibly fits your market theory. But it rapidly becomes clear that plausibility can't help you here - all three events are plausible. Fittability to your pet market theory doesn't tell you how to divide your time. There's an uncrossable gap between your 100 minutes of time, which are conserved; versus your ability to explain how an outcome fits your theory, which is unlimited.

And yet... even in your uncertain state of mind, it seems that you anticipate the three events differently; that you expect to need some excuses more than others. And - this is the fascinating part - when you think of something that makes it seem more likely that bond prices will go up, then you feel less likely to need an excuse for bond prices going down or remaining the same.

It even seems like there's a relation between how much you anticipate each of the three outcomes, and how much time you want to spend preparing each excuse. Of course the relation can't actually be quantified. You have 100 minutes to prepare your speech, but there isn't 100 of anything to divide up in this anticipation business. (Although you do work out that, if some particular outcome occurs, then your utility function is logarithmic in time spent preparing the excuse.)

Still... your mind keeps coming back to the idea that anticipation is limited, unlike excusability, but like time to prepare excuses. Maybe anticipation should be treated as a conserved resource, like money. Your first impulse is to try to get more anticipation, but you soon realize that, even if you get more anticiptaion, you won't have any more time to prepare your excuses. No, your only course is to allocate your limited supply of anticipation as best you can.

You're pretty sure you weren't taught anything like that in your statistics courses. They didn't tell you what to do when you felt so terribly uncertain. They didn't tell you what to do when there were no little numbers handed to you. Why, even if you tried to use numbers, you might end up using any sort of numbers at all - there's no hint what kind of math to use, if you should be using math! Maybe you'd end up using pairs of numbers, right and left numbers, which you'd call DS for Dexter-Sinister... or who knows what else? (Though you do have only 100 minutes to spend preparing excuses.)

If only there were an art of focusing your uncertainty - of squeezing as much anticipation as possible into whichever outcome will actually happen!

But what could we call an art like that? And what would the rules be like?

Sequence: Mysterious Answers to Mysterious Questions

Belief as Attire

The Virtue of Narrowness

Referenced by: Fake Explanations • Mysterious Answers to Mysterious Questions • Fake Optimization Criteria • Absolute Authority

Original with comments: Focus Your Uncertainty

The Proper Use of Doubt

Eliezer Yudkowsky, 06 August 2007 08:29PM

Once, when I was holding forth upon the Way, I remarked upon how most organized belief systems exist to flee from doubt. A listener replied to me that the Jesuits must be immune from this criticism, because they practice organized doubt: their novices, he said, are told to doubt Christianity; doubt the existence of God; doubt if their calling is real; doubt that they are suitable for perpetual vows of chastity and poverty. And I said: Ah, but they're supposed to overcome these doubts, right? He said: No, they are to doubt that perhaps their doubts may grow and become stronger.

Googling failed to confirm or refute these allegations. (If anyone in the audience can help, I'd be much obliged.) But I find this scenario fascinating, worthy of discussion, regardless of whether it is true or false of Jesuits. If the Jesuits practiced deliberate doubt, as described above, would they therefore be virtuous as rationalists?

I think I have to concede that the Jesuits, in the (possibly hypothetical) scenario above, would not properly be described as "fleeing from doubt". But the (possibly hypothetical) conduct still strikes me as highly suspicious. To a truly virtuous rationalist, doubt should not be scary. The conduct described above sounds to me like a program of desensitization for something very scary, like exposing an arachnophobe to spiders under carefully controlled conditions.

But even so, they are encouraging their novices to doubt - right? Does it matter if their reasons are flawed? Is this not still a worthy deed unto a rationalist?

All curiosity seeks to annihilate itself; there is no curiosity that does not want an answer. But if you obtain an answer, if you satisfy your curiosity, then the glorious mystery will no longer be mysterious.

In the same way, every doubt exists in order to annihilate some particular belief. If a doubt fails to destroy its target, the doubt has died unfulfilled - but that is still a resolution, an ending, albeit a sadder one. A doubt that neither destroys itself nor destroys its target might as well have never existed at all. It is the resolution of doubts, not the mere act of doubting, which drives the ratchet of rationality forward.

Every improvement is a change, but not every change is an improvement. Every rationalist doubts, but not all doubts are rational. Wearing doubts doesn't make you a rationalist any more than wearing a white medical lab coat makes you a doctor.

A rational doubt comes into existence for a specific reason - you have some specific justification to suspect the belief is wrong. This reason in turn, implies an avenue of investigation which will either destroy the targeted belief, or destroy the doubt. This holds even for highly abstract doubts, like "I wonder if there might be a simpler hypothesis which also explains this data." In this case you investigate by trying to think of simpler hypotheses. As this search continues longer and longer without fruit, you will think it less and less likely that the next increment of computation will be the one to succeed. Eventually the cost of searching will exceed the expected benefit, and you'll stop searching. At which point you can no longer claim to be usefully doubting. A doubt that is not investigated might as well not exist. Every doubt exists to destroy itself, one way or the other. An unresolved doubt is a null-op; it does not turn the wheel, neither forward nor back.

If you really believe a religion (not just believe in it), then why would you tell your novices to consider doubts that must die unfulfilled? It would be like telling physics students to painstakingly doubt that the 20th-century revolution might have been a mistake, and that Newtonian mechanics was correct all along. If you don't really doubt something, why would you pretend that you do?

Because we all want to be seen as rational - and doubting is widely believed to be a virtue of a rationalist. But it is not widely understood that you need a particular reason to doubt, or that an unresolved doubt is a null-op. Instead people think it's about modesty, a submissive demeanor, maintaining the tribal status hierarchy - almost exactly the same problem as with humility, on which I have previously written. Making a great public display of doubt to convince yourself that you are a rationalist, will do around as much good as wearing a lab coat.

To avoid professing doubts, remember:

- A rational doubt exists to destroy its target belief, and if it does not destroy its target it dies unfulfilled.

- A rational doubt arises from some specific reason the belief might be wrong.

- An unresolved doubt is a null-op.

- An uninvestigated doubt might as well not exist.

- You should not be proud of mere doubting, although you can justly be proud when you have just finished tearing a cherished belief to shreds.

- Though it may take courage to face your doubts, never forget that to an ideal mind doubt would not be scary in the first place.

Sequence: Letting Go

Just Lose Hope Already

You Can Face Reality

Referenced by: Conjunction Controversy (Or, How They Nail It Down) • Avoiding Your Belief's Real Weak Points • Two Cult Koans • Cultish Countercultishness • Trust in Math • No Safe Defense, Not Even Science • Against Devil's Advocacy • My Kind of Reflection • The Level Above Mine • Crisis of Faith • Growing Up is Hard • The Sin of Underconfidence • "Outside View!" as Conversation-Halter

Original with comments: The Proper Use of Doubt

The Virtue of Narrowness

Eliezer Yudkowsky, 07 August 2007 05:57PM

What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world.
-- Twelve Virtues of Rationality

Within their own professions, people grasp the importance of narrowness; a car mechanic knows the difference between a carburetor and a radiator, and would not think of them both as "car parts". A hunter-gatherer knows the difference between a lion and a panther. A janitor does not wipe the floor with window cleaner, even if the bottles look similar to one who has not mastered the art.

Outside their own professions, people often commit the misstep of trying to broaden a word as widely as possible, to cover as much territory as possible. Is it not more glorious, more wise, more impressive, to talk about all the apples in the world? How much loftier it must be to explain human thought in general, without being distracted by smaller questions, such as how humans invent techniques for solving a Rubik's Cube. Indeed, it scarcely seems necessary to consider specific questions at all; isn't a general theory a worthy enough accomplishment on its own?

It is the way of the curious to lift up one pebble from among a million pebbles on the shore, and see something new about it, something interesting, something different. You call these pebbles "diamonds", and ask what might be special about them - what inner qualities they might have in common, beyond the glitter you first noticed. And then someone else comes along and says: "Why not call this pebble a diamond too? And this one, and this one?" They are enthusiastic, and they mean well. For it seems undemocratic and exclusionary and elitist and unholistic to call some pebbles "diamonds", and others not. It seems... narrow-minded... if you'll pardon the phrase. Hardly open, hardly embracing, hardly communal.

You might think it poetic, to give one word many meanings, and thereby spread shades of connotation all around. But even poets, if they are good poets, must learn to see the world precisely. It is not enough to compare love to a flower. Hot jealous unconsummated love is not the same as the love of a couple married for decades. If you need a flower to symbolize jealous love, you must go into the garden, and look, and make subtle distinctions - find a flower with a heady scent, and a bright color, and thorns. Even if your intent is to shade meanings and cast connotations, you must keep precise track of exactly which meanings you shade and connote.

It is a necessary part of the rationalist's art - or even the poet's art! - to focus narrowly on unusual pebbles which possess some special quality. And look at the details which those pebbles - and those pebbles alone! - share among each other. This is not a sin.

It is perfectly all right for modern evolutionary biologists to explain just the patterns of living creatures, and not the "evolution" of stars or the "evolution" of technology. Alas, some unfortunate souls use the same word "evolution" to cover the naturally selected patterns of replicating life, and the strictly accidental structure of stars, and the intelligently configured structure of technology. And as we all know, if people use the same word, it must all be the same thing. You should automatically generalize anything you think you know about biological evolution to technology. Anyone who tells you otherwise must be a mere pointless pedant. It couldn't possibly be that your abysmal ignorance of modern evolutionary theory is so total that you can't tell the difference between a carburetor and a radiator. That's unthinkable. No, the other guy - you know, the one who's studied the math - is just too dumb to see the connections.

And what could be more virtuous than seeing connections? Surely the wisest of all human beings are the New Age gurus who say "Everything is connected to everything else." If you ever say this aloud, you should pause, so that everyone can absorb the sheer shock of this Deep Wisdom.

There is a trivial mapping between a graph and its complement. A fully connected graph, with an edge between every two vertices, conveys the same amount of information as a graph with no edges at all. The important graphs are the ones where some things are not connected to some other things.

When the unenlightened ones try to be profound, they draw endless verbal comparisons between this topic, and that topic, which is like this, which is like that; until their graph is fully connected and also totally useless. The remedy is specific knowledge and in-depth study. When you understand things in detail, you can see how they are not alike, and start enthusiastically subtracting edges off your graph.

Likewise, the important categories are the ones that do not contain everything in the universe. Good hypotheses can only explain some possible outcomes, and not others.

It was perfectly all right for Isaac Newton to explain just gravity, just the way things fall down - and how planets orbit the Sun, and how the Moon generates the tides - but not the role of money in human society or how the heart pumps blood. Sneering at narrowness is rather reminiscent of ancient Greeks who thought that going out and actually looking at things was manual labor, and manual labor was for slaves.

As Plato put it (in The Republic, Book VII):

"If anyone should throw back his head and learn something by staring at the varied patterns on a ceiling, apparently you would think that he was contemplating with his reason, when he was only staring with his eyes... I cannot but believe that no study makes the soul look on high except that which is concerned with real being and the unseen. Whether he gape and stare upwards, or shut his mouth and stare downwards, if it be things of the senses that he tries to learn something about, I declare he never could learn, for none of these things admit of knowledge: I say his soul is looking down, not up, even if he is floating on his back on land or on sea!"

Many today make a similar mistake, and think that narrow concepts are as lowly and unlofty and unphilosophical as, say, going out and looking at things - an endeavor only suited to the underclass. But rationalists - and also poets - need narrow words to express precise thoughts; they need categories which include only some things, and exclude others. There's nothing wrong with focusing your mind, narrowing your categories, excluding possibilities, and sharpening your propositions. Really, there isn't! If you make your words too broad, you end up with something that isn't true and doesn't even make good poetry.

And DON'T EVEN GET ME STARTED on people who think Wikipedia is an "Artificial Intelligence", the invention of LSD was a "Singularity" or that corporations are "superintelligent"!

Sequence: Mysterious Answers to Mysterious Questions

Focus Your Uncertainty

Your Strength as a Rationalist

Referenced by: Original Seeing • Evolutionary Psychology • Resist the Happy Death Spiral • The Fallacy of Gray • Expecting Beauty • Building Something Smarter • Recursive Self-Improvement • Seduced by Imagination • Informers and Persuaders • On Things that are Awesome

Original with comments: The Virtue of Narrowness

You Can Face Reality

Eliezer Yudkowsky, 09 August 2007 01:46AM

What is true is already so.
Owning up to it doesn't make it worse.
Not being open about it doesn't make it go away.
And because it's true, it is what is there to be interacted with.
Anything untrue isn't there to be lived.
People can stand what is true,
for they are already enduring it.
-- Eugene Gendlin

(Hat tip to Stephen Omohundro.)

Sequence: Letting Go

The Proper Use of Doubt

The Meditation on Curiosity

Referenced by: Say Not "Complexity" • The Meditation on Curiosity • Is Morality Given? • The Fear of Common Knowledge • Crisis of Faith • That Crisis thing seems pretty useful

Original with comments: You Can Face Reality

The Apocalypse Bet

Eliezer Yudkowsky, 09 August 2007 05:23PM

A problem with betting on engineered superplagues, physics disasters, nanotechnological warfare, or intelligence explosions of both Friendly and unFriendly type, is that all these events are likely to disrupt settlement of trades (to put it mildly). It's not easy to sell a bet that pays off only if the prediction market ceases to exist.

And yet everyone still wants to know the year, month, and day of the Singularity. Even I want to know, I'm just professionally aware that the knowledge is not available.

This morning, I saw that someone had launched yet another poll on "when the Singularity will occur". Just a raw poll, mind you, not a prediction market. I was thinking of how completely and utterly worthless this poll was, and how a prediction market might be slightly less than completely worthless, when it occurred to me how to structure the bet - bet that "settlement of trades will be disrupted / the resources gambled will become worthless, no later than time T".

Suppose you think that gold will become worthless on April 27th, 2020 at between four and four-thirty in the morning. I, on the other hand, think this event will not occur until 2030. We can sign a contract in which I pay you one ounce of gold per year from 2010 to 2020, and then you pay me two ounces of gold per year from 2020 to 2030. If gold becomes worthless when you say, you will have profited; if gold becomes worthlesss when I say, I will have profited. We can have a prediction market on a generic apocalypse, in which participants who believe in an earlier apocalypse are paid by believers in a later apocalypse, until they pass the date of their prediction, at which time the flow reverses with interest. I don't see any way to distinguish between apocalypses, but we can ask the participants why they were willing to bet, and probably receive a decent answer.

I would be quite interested in seeing what such a market had to say. And if the predicted date was hovering around 2080, I would pick up as much of that free money as I dared.

Original with comments: The Apocalypse Bet

Your Strength as a Rationalist

Eliezer Yudkowsky, 11 August 2007 12:21AM

(The following happened to me in an IRC chatroom, long enough ago that I was still hanging around in IRC chatrooms. Time has fuzzed the memory and my report may be imprecise.)

So there I was, in an IRC chatroom, when someone reports that a friend of his needs medical advice. His friend says that he's been having sudden chest pains, so he called an ambulance, and the ambulance showed up, but the paramedics told him it was nothing, and left, and now the chest pains are getting worse. What should his friend do?

I was confused by this story. I remembered reading about homeless people in New York who would call ambulances just to be taken someplace warm, and how the paramedics always had to take them to the emergency room, even on the 27th iteration. Because if they didn't, the ambulance company could be sued for lots and lots of money. Likewise, emergency rooms are legally obligated to treat anyone, regardless of ability to pay. (And the hospital absorbs the costs, which are enormous, so hospitals are closing their emergency rooms... It makes you wonder what's the point of having economists if we're just going to ignore them.) So I didn't quite understand how the described events could have happened. Anyone reporting sudden chest pains should have been hauled off by an ambulance instantly.

And this is where I fell down as a rationalist. I remembered several occasions where my doctor would completely fail to panic at the report of symptoms that seemed, to me, very alarming. And the Medical Establishment was always right. Every single time. I had chest pains myself, at one point, and the doctor patiently explained to me that I was describing chest muscle pain, not a heart attack. So I said into the IRC channel, "Well, if the paramedics told your friend it was nothing, it must really be nothing - they'd have hauled him off if there was the tiniest chance of serious trouble."

Thus I managed to explain the story within my existing model, though the fit still felt a little forced...

Later on, the fellow comes back into the IRC chatroom and says his friend made the whole thing up. Evidently this was not one of his more reliable friends.

I should have realized, perhaps, that an unknown acquaintance of an acquaintance in an IRC channel might be less reliable than a published journal article. Alas, belief is easier than disbelief; we believe instinctively, but disbelief requires a conscious effort.

So instead, by dint of mighty straining, I forced my model of reality to explain an anomaly that never actually happened. And I knew how embarrassing this was. I knew that the usefulness of a model is not what it can explain, but what it can't. A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation.

Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge.

We are all weak, from time to time; the sad part is that I could have been stronger. I had all the information I needed to arrive at the correct answer, I even noticed the problem, and then I ignored it. My feeling of confusion was a Clue, and I threw my Clue away.

I should have paid more attention to that sensation of still feels a little forced. It's one of the most important feelings a truthseeker can have, a part of your strength as a rationalist. It is a design flaw in human cognition that this sensation manifests as a quiet strain in the back of your mind, instead of a wailing alarm siren and a glowing neon sign reading "EITHER YOUR MODEL IS FALSE OR THIS STORY IS WRONG."

Sequence: Mysterious Answers to Mysterious Questions

The Virtue of Narrowness

Absence of Evidence Is Evidence of Absence

Sequence: Noticing Confusion

Absence of Evidence Is Evidence of Absence

Referenced by: Absence of Evidence Is Evidence of Absence • Conservation of Expected Evidence • Hindsight Devalues Science • Fake Explanations • Guessing the Teacher's Password • Science as Attire • Fake Causality • The Futility of Emergence • Positive Bias: Look Into the Dark • How to Convince Me That 2 + 2 = 3 • Avoiding Your Belief's Real Weak Points • How to Seem (and Be) Deep • An Alien God • Fake Optimization Criteria • Asch's Conformity Experiment • Dissolving the Question • Where Experience Confuses Physicists • Faster Than Science • Setting Up Metaethics • Entangled Truths, Contagious Lies • Belief in Intelligence

Original with comments: Your Strength as a Rationalist

I Defy the Data!

Eliezer Yudkowsky, 11 August 2007 09:33PM

One of the great weaknesses of Science is this mistaken idea that if an experiment contradicts the dominant theory, we should throw out the theory instead of the experiment.

Experiments can go awry. They can contain design flaws. They can be deliberately corrupted. They can be unconsciously corrupted. They can be selectively reported. Most of all, 1 time in 20 they can be "statistically significant" by sheer coincidence, and there are a lot of experiments out there.

Unfortunately, Science has this notion that you can never go against an honestly obtained experimental result. So, when someone obtains an experimental result that contradicts the standard model, researchers are faced with a dilemma for resolving their cognitive dissonance: they either have to immediately throw away the standard model, or else attack the experiment - accuse the researchers of dishonesty, or flawed design, or conflict of interest...

Someone once presented me with a new study on the effects of intercessory prayer (that is, people praying for patients who are not told about the prayer), which showed 50% of the prayed-for patients achieving success at in-vitro fertilization, versus 25% of the control group. I liked this claim. It had a nice large effect size. Claims of blatant impossible effects are much more pleasant to deal with than claims of small impossible effects that are "statistically significant".

So I cheerfully said: "I defy the data."

My original phrasing was actually "I deny the data". Nonetheless I said it outright, without apology, and with deliberate insolence. I am keeping my theory; your experiment is wrong.

If an experimental result contradicts the Standard Model, this is an important fact. It needs to be openly acknowledged. An experiment that makes traditionalists want to discard the data - or even an experiment that makes traditionalists very skeptical of the data - should be a high priority for replication. An experiment worth defying should command attention!

But it is not socially acceptable to say, "The hell with your experimental falsification, I'm keeping my theory." So the data has to be defied covertly - by character assassination of the researchers, by sly innuendos, by dire hints of controversy. The data has to be dismissed, excused away, swept under a rug, silently into the dark, because you can't admit you're defying the data. This is not a good way of focusing attention on an anomalous result. This is not a good way to ensure funding for replication attempts.

It would be much better if science had a standard procedure for saying, "I defy the data!" It would be clearly understood that this was a bold act, and someone else in the audience might stand up and say, "Wait a minute, is that data really worth defying?" If a major figure in the field said "I defy the data!", this would be sufficient justification on grant proposals for why the result urgently needed replication. Scientists could say, "I'm holding my breath, waiting for replication," rather than having to take sides immediately in the character-assassination controversy.

Maybe you could even get the media to report that the experiment has been "published but defied". Then the replication, or failure to replicate, would be news. The replicators could get their names in the newspaper, and the negative result could be published in a major journal. If you want replications done, you'll have to offer some incentive.

I would also suggest that when an experiment is defied, the replication must pre-declare a minimum effect size, and attain significance of p<0.01. In extreme cases where claims have been made and shot down before, p<0.001.

Oh, and the prayer study? Soon enough we heard that it had been retracted and was probably fraudulent. But I didn't say fraud. I didn't speculate on how the results might have been obtained. That would have been dismissive. I just stuck my neck out, and nakedly, boldly, without excuses, defied the data.

Addendum: I should have spelled this out explicitly: You can defy the data on one experiment. You can't defy the data on multiple experiments. At that point you either have to relinquish the theory or dismiss the data - point to a design flaw, or refer to an even larger body of experiments that failed to replicate the result, or accuse the researchers of a deliberate hoax, et cetera. But you should not turn around and argue that the theory and the experiment are actually compatible. Why didn't you think of that before you defied the data? Defying the data admits that the data is not compatible with your theory; it sticks your neck way out, so your head can be easily chopped off.

Referenced by: Update Yourself Incrementally • Conjunction Controversy (Or, How They Nail It Down) • Psychic Powers • Shut up and do the impossible! • Logical or Connectionist AI?

Original with comments: I Defy the Data!

Absence of Evidence Is Evidence of Absence

Eliezer Yudkowsky, 12 August 2007 08:34PM

From Robyn Dawes's Rational Choice in an Uncertain World:

Post-hoc fitting of evidence to hypothesis was involved in a most grievous chapter in United States history: the internment of Japanese-Americans at the beginning of the Second World War. When California governor Earl Warren testified before a congressional hearing in San Francisco on February 21, 1942, a questioner pointed out that there had been no sabotage or any other type of espionage by the Japanese-Americans up to that time. Warren responded, "I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed... I believe we are just being lulled into a false sense of security."

Consider Warren's argument from a Bayesian perspective. When we see evidence, hypotheses that assigned a higher likelihood to that evidence, gain probability at the expense of hypotheses that assigned a lower likelihood to the evidence. This is a phenomenon of relative likelihoods and relative probabilities. You can assign a high likelihood to the evidence and still lose probability mass to some other hypothesis, if that other hypothesis assigns a likelihood that is even higher.

Warren seems to be arguing that, given that we see no sabotage, this confirms that a Fifth Column exists. You could argue that a Fifth Column might delay its sabotage. But the likelihood is still higher that the absence of a Fifth Column would perform an absence of sabotage.

Let E stand for the observation of sabotage, H1 for the hypothesis of a Japanese-American Fifth Column, and H2 for the hypothesis that no Fifth Column exists. Whatever the likelihood that a Fifth Column would do no sabotage, the probability P(E|H1), it cannot be as large as the likelihood that no Fifth Column does no sabotage, the probability P(E|H2). So observing a lack of sabotage increases the probability that no Fifth Column exists.

A lack of sabotage doesn't prove that no Fifth Column exists. Absence of proof is not proof of absence. In logic, A->B, "A implies B", is not equivalent to ~A->~B, "not-A implies not-B".

But in probability theory, absence of evidence is always evidence of absence. If E is a binary event and P(H|E) > P(H), "seeing E increases the probability of H"; then P(H|~E) < P(H), "failure to observe E decreases the probability of H". P(H) is a weighted mix of P(H|E) and P(H|~E), and necessarily lies between the two. If any of this sounds at all confusing, see An Intuitive Explanation of Bayesian Reasoning.

Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely to produce the signs. The absence of an observation may be strong evidence of absence or very weak evidence of absence, depending on how likely the cause is to produce the observation. The absence of an observation that is only weakly permitted (even if the alternative hypothesis does not allow it at all), is very weak evidence of absence (though it is evidence nonetheless). This is the fallacy of "gaps in the fossil record" - fossils form only rarely; it is futile to trumpet the absence of a weakly permitted observation when many strong positive observations have already been recorded. But if there are no positive observations at all, it is time to worry; hence the Fermi Paradox.

Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge. The strength of a model is not what it can explain, but what it can't, for only prohibitions constrain anticipation. If you don't notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.

Sequence: Mysterious Answers to Mysterious Questions

Your Strength as a Rationalist

Conservation of Expected Evidence

Sequence: Noticing Confusion

Your Strength as a Rationalist

Hindsight bias

Referenced by: Conservation of Expected Evidence • Fake Explanations • Conjunction Controversy (Or, How They Nail It Down) • Fake Optimization Criteria • When Science Can't Help • Frequentist Statistics are Frequently Subjective • Demands for Particular Proof: Appendices

Original with comments: Absence of Evidence Is Evidence of Absence

Conservation of Expected Evidence

Eliezer Yudkowsky, 13 August 2007 03:55PM

Followup to: Absence of Evidence Is Evidence of Absence.

Friedrich Spee von Langenfeld, a priest who heard the confessions of condemned witches, wrote in 1631 the Cautio Criminalis ('prudence in criminal cases') in which he bitingly described the decision tree for condemning accused witches: If the witch had led an evil and improper life, she was guilty; if she had led a good and proper life, this too was a proof, for witches dissemble and try to appear especially virtuous. After the woman was put in prison: if she was afraid, this proved her guilt; if she was not afraid, this proved her guilt, for witches characteristically pretend innocence and wear a bold front. Or on hearing of a denunciation of witchcraft against her, she might seek flight or remain; if she ran, that proved her guilt; if she remained, the devil had detained her so she could not get away.

Spee acted as confessor to many witches; he was thus in a position to observe every branch of the accusation tree, that no matter what the accused witch said or did, it was held a proof against her. In any individual case, you would only hear one branch of the dilemma. It is for this reason that scientists write down their experimental predictions in advance.

But you can't have it both ways - as a matter of probability theory, not mere fairness. The rule that "absence of evidence is evidence of absence" is a special case of a more general law, which I would name Conservation of Expected Evidence: The expectation of the posterior probability, after viewing the evidence, must equal the prior probability.

P(H) = P(H)
P(H) = P(H,E) + P(H,~E)
P(H) = P(H|E)*P(E) + P(H|~E)*P(~E)

Therefore, for every expectation of evidence, there is an equal and opposite expectation of counterevidence.

If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction. If you're very confident in your theory, and therefore anticipate seeing an outcome that matches your hypothesis, this can only provide a very small increment to your belief (it is already close to 1); but the unexpected failure of your prediction would (and must) deal your confidence a huge blow. On average, you must expect to be exactly as confident as when you started out. Equivalently, the mere expectation of encountering evidence - before you've actually seen it - should not shift your prior beliefs. (Again, if this is not intuitively obvious, see An Intuitive Explanation of Bayesian Reasoning.)

So if you claim that "no sabotage" is evidence for the existence of a Japanese-American Fifth Column, you must conversely hold that seeing sabotage would argue against a Fifth Column. If you claim that "a good and proper life" is evidence that a woman is a witch, then an evil and improper life must be evidence that she is not a witch. If you argue that God, to test humanity's faith, refuses to reveal His existence, then the miracles described in the Bible must argue against the existence of God.

Doesn't quite sound right, does it? Pay attention to that feeling of this seems a little forced, that quiet strain in the back of your mind. It's important.

For a true Bayesian, it is impossible to seek evidence that confirms a theory. There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before. You can only ever seek evidence to test a theory, not to confirm it.

This realization can take quite a load off your mind. You need not worry about how to interpret every possible experimental result to confirm your theory. You needn't bother planning how to make any given iota of evidence confirm your theory, because you know that for every expectation of evidence, there is an equal and oppositive expectation of counterevidence. If you try to weaken the counterevidence of a possible "abnormal" observation, you can only do it by weakening the support of a "normal" observation, to a precisely equal and opposite degree. It is a zero-sum game. No matter how you connive, no matter how you argue, no matter how you strategize, you can't possibly expect the resulting game plan to shift your beliefs (on average) in a particular direction.

You might as well sit back and relax while you wait for the evidence to come in.

...human psychology is so screwed up.

Sequence: Mysterious Answers to Mysterious Questions

Absence of Evidence Is Evidence of Absence

Hindsight bias

Sequence: Against Rationalization

Knowing About Biases Can Hurt People

Update Yourself Incrementally

Referenced by: Update Yourself Incrementally • Hindsight bias • Hindsight Devalues Science • Fake Explanations • Fake Causality • Rationalization • The Meditation on Curiosity • Fake Optimization Criteria • Rational vs. Scientific Ev-Psych • Crisis of Faith • Serious Stories • That Crisis thing seems pretty useful • Drawing Two Aces

Original with comments: Conservation of Expected Evidence

Update Yourself Incrementally

Eliezer Yudkowsky, 14 August 2007 02:56PM

Politics is the mind-killer. Debate is war, arguments are soldiers. There is the temptation to search for ways to interpret every possible experimental result to confirm your theory, like securing a citadel against every possible line of attack. This you cannot do. It is mathematically impossible. For every expectation of evidence, there is an equal and opposite expectation of counterevidence.

But it's okay if your cherished belief isn't perfectly defended. If the hypothesis is that the coin comes up heads 95% of the time, then one time in twenty you will see what looks like contrary evidence. This is okay. It's normal. It's even expected, so long as you've got nineteen supporting observations for every contrary one. A probabilistic model can take a hit or two, and still survive, so long as the hits don't keep on coming in.

Yet it is widely believed, especially in the court of public opinion, that a true theory can have no failures and a false theory no successes.

You find people holding up a single piece of what they conceive to be evidence, and claiming that their theory can 'explain' it, as though this were all the support that any theory needed. Apparently a false theory can have no supporting evidence; it is impossible for a false theory to fit even a single event. Thus, a single piece of confirming evidence is all that any theory needs.

It is only slightly less foolish to hold up a single piece of probabilistic counterevidence as disproof, as though it were impossible for a correct theory to have even a slight argument against it. But this is how humans have argued for ages and ages, trying to defeat all enemy arguments, while denying the enemy even a single shred of support. People want their debates to be one-sided; they are accustomed to a world in which their preferred theories have not one iota of antisupport. Thus, allowing a single item of probabilistic counterevidence would be the end of the world.

I just know someone in the audience out there is going to say, "But you can't concede even a single point if you want to win debates in the real world! If you concede that any counterarguments exist, the Enemy will harp on them over and over - you can't let the Enemy do that! You'll lose! What could be more viscerally terrifying than that?"

Whatever. Rationality is not for winning debates, it is for deciding which side to join. If you've already decided which side to argue for, the work of rationality is done within you, whether well or poorly. But how can you, yourself, decide which side to argue? If choosing the wrong side is viscerally terrifying, even just a little viscerally terrifying, you'd best integrate all the evidence.

Rationality is not a walk, but a dance. On each step in that dance your foot should come down in exactly the correct spot, neither to the left nor to the right. Shifting belief upward with each iota of confirming evidence. Shifting belief downward with each iota of contrary evidence. Yes, down. Even with a correct model, if it is not an exact model, you will sometimes need to revise your belief down.

If an iota or two of evidence happens to countersupport your belief, that's okay. It happens, sometimes, with probabilistic evidence for non-exact theories. (If an exact theory fails, you are in trouble!) Just shift your belief downward a little - the probability, the odds ratio, or even a nonverbal weight of credence in your mind. Just shift downward a little, and wait for more evidence. If the theory is true, supporting evidence will come in shortly, and the probability will climb again. If the theory is false, you don't really want it anyway.

The problem with using black-and-white, binary, qualitative reasoning is that any single observation either destroys the theory or it does not. When not even a single contrary observation is allowed, it creates cognitive dissonance and has to be argued away. And this rules out incremental progress; it rules out correct integration of all the evidence. Reasoning probabilistically, we realize that on average, a correct theory will generate a greater weight of support than countersupport. And so you can, without fear, say to yourself: "This is gently contrary evidence, I will shift my belief downward". Yes, down. It does not destroy your cherished theory. That is qualitative reasoning; think quantitatively.

For every expectation of evidence, there is an equal and opposite expectation of counterevidence. On every occasion, you must, on average, anticipate revising your beliefs downward as much as you anticipate revising them upward. If you think you already know what evidence will come in, then you must already be fairly sure of your theory - probability close to 1 - which doesn't leave much room for the probability to go further upward. And however unlikely it seems that you will encounter disconfirming evidence, the resulting downward shift must be large enough to precisely balance the anticipated gain on the other side. The weighted mean of your expected posterior probability must equal your prior probability.

How silly is it, then, to be terrified of revising your probability downward, if you're bothering to investigate a matter at all? On average, you must anticipate as much downward shift as upward shift from every individual observation.

It may perhaps happen that an iota of antisupport comes in again, and again and again, while new support is slow to trickle in. You may find your belief drifting downward and further downward. Until, finally, you realize from which quarter the winds of evidence are blowing against you. In that moment of realization, there is no point in constructing excuses. In that moment of realization, you have already relinquished your cherished belief. Yay! Time to celebrate! Pop a champagne bottle or send out for pizza! You can't become stronger by keeping the beliefs you started with, after all.

Sequence: Against Rationalization

Conservation of Expected Evidence

One Argument Against An Army

Referenced by: One Argument Against An Army • LA-602 vs. RHIC Review • Crisis of Faith • Singletons Rule OK • That Crisis thing seems pretty useful • Demands for Particular Proof: Appendices

Original with comments: Update Yourself Incrementally

One Argument Against An Army

Eliezer Yudkowsky, 15 August 2007 06:39PM

Followup to: Update Yourself Incrementally

Yesterday I talked about a style of reasoning in which not a single contrary argument is allowed, with the result that every non-supporting observation has to be argued away. Today I suggest that when people encounter a contrary argument, they prevent themselves from downshifting their confidence by rehearsing already-known support.

Suppose the country of Freedonia is debating whether its neighbor, Sylvania, is responsible for a recent rash of meteor strikes on its cities. There are several pieces of evidence suggesting this: the meteors struck cities close to the Sylvanian border; there was unusual activity in the Sylvanian stock markets before the strikes; and the Sylvanian ambassador Trentino was heard muttering about "heavenly vengeance".

Someone comes to you and says: "I don't think Sylvania is responsible for the meteor strikes. They have trade with us of billions of dinars annually." "Well," you reply, "the meteors struck cities close to Sylvania, there was suspicious activity in their stock market, and their ambassador spoke of heavenly vengeance afterward." Since these three arguments outweigh the first, you keep your belief that Sylvania is responsible - you believe rather than disbelieve, qualitatively. Clearly, the balance of evidence weighs against Sylvania.

Then another comes to you and says: "I don't think Sylvania is responsible for the meteor strikes. Directing an asteroid strike is really hard. Sylvania doesn't even have a space program." You reply, "But the meteors struck cities close to Sylvania, and their investors knew it, and the ambassador came right out and admitted it!" Again, these three arguments outweigh the first (by three arguments against one argument), so you keep your belief that Sylvania is responsible.

Indeed, your convictions are strengthened. On two separate occasions now, you have evaluated the balance of evidence, and both times the balance was tilted against Sylvania by a ratio of 3-to-1.

You encounter further arguments by the pro-Sylvania traitors - again, and again, and a hundred times again - but each time the new argument is handily defeated by 3-to-1. And on every occasion, you feel yourself becoming more confident that Sylvania was indeed responsible, shifting your prior according to the felt balance of evidence.

The problem, of course, is that by rehearsing arguments you already knew, you are double-counting the evidence This would be a grave sin even if you double-counted all the evidence. (Imagine a scientist who does an experiment with 50 subjects and fails to obtain statistically significant results, so he counts all the data twice.)

But to selectively double-count only some evidence is sheer farce. I remember seeing a cartoon as a child, where a villain was dividing up loot using the following algorithm: "One for you, one for me. One for you, one-two for me. One for you, one-two-three for me."

As I emphasized yesterday, even if a cherished belief is true, a rationalist may sometimes need to downshift the probability while integrating all the evidence. Yes, the balance of support may still favor your cherished belief. But you still have to shift the probability down - yes, down - from whatever it was before you heard the contrary evidence. It does no good to rehearse supporting arguments, because you have already taken those into account.

And yet it does appear to me that when people are confronted by a new counterargument, they search for a justification not to downshift their confidence, and of course they find supporting arguments they already know. I have to keep constant vigilance not to do this myself! It feels as natural as parrying a sword-strike with a handy shield.

With the right kind of wrong reasoning, a handful of support - or even a single argument - can stand off an army of contradictions.

Sequence: Against Rationalization

Update Yourself Incrementally

The Bottom Line

Referenced by: What Evidence Filtered Evidence? • Avoiding Your Belief's Real Weak Points • The Meditation on Curiosity • Priming and Contamination • Resist the Happy Death Spiral • LA-602 vs. RHIC Review • The Genetic Fallacy • Fundamental Doubts • Fighting a Rearguard Action Against the Truth • Crisis of Faith • Informers and Persuaders • That Crisis thing seems pretty useful

Original with comments: One Argument Against An Army

Hindsight bias

Eliezer Yudkowsky, 16 August 2007 09:58PM

Hindsight bias is when people who know the answer vastly overestimate its predictability or obviousness, compared to the estimates of subjects who must guess without advance knowledge. Hindsight bias is sometimes called the I-knew-it-all-along effect.

Fischhoff and Beyth (1975) presented students with historical accounts of unfamiliar incidents, such as a conflict between the Gurkhas and the British in 1814. Given the account as background knowledge, five groups of students were asked what they would have predicted as the probability for each of four outcomes: British victory, Gurkha victory, stalemate with a peace settlement, or stalemate with no peace settlement. Four experimental groups were respectively told that these four outcomes were the historical outcome. The fifth, control group was not told any historical outcome. In every case, a group told an outcome assigned substantially higher probability to that outcome, than did any other group or the control group.

Hindsight bias matters in legal cases, where a judge or jury must determine whether a defendant was legally negligent in failing to foresee a hazard (Sanchiro 2003). In an experiment based on an actual legal case, Kamin and Rachlinski (1995) asked two groups to estimate the probability of flood damage caused by blockage of a city-owned drawbridge. The control group was told only the background information known to the city when it decided not to hire a bridge watcher. The experimental group was given this information, plus the fact that a flood had actually occurred. Instructions stated the city was negligent if the foreseeable probability of flooding was greater than 10%. 76% of the control group concluded the flood was so unlikely that no precautions were necessary; 57% of the experimental group concluded the flood was so likely that failure to take precautions was legally negligent. A third experimental group was told the outcome andalso explicitly instructed to avoid hindsight bias, which made no difference: 56% concluded the city was legally negligent.

Viewing history through the lens of hindsight, we vastly underestimate the cost of effective safety precautions. In 1986, the Challenger exploded for reasons traced to an O-ring losing flexibility at low temperature. There were warning signs of a problem with the O-rings. But preventing the Challenger disaster would have required, not attending to the problem with the O-rings, but attending to every warning sign which seemed as severe as the O-ring problem, without benefit of hindsight. It could have been done, but it would have required a general policy much more expensive than just fixing the O-Rings.

Shortly after September 11th 2001, I thought to myself, and now someone will turn up minor intelligence warnings of something-or-other, and then the hindsight will begin. Yes, I'm sure they had some minor warnings of an al Qaeda plot, but they probably also had minor warnings of mafia activity, nuclear material for sale, and an invasion from Mars.

Because we don't see the cost of a general policy, we learn overly specific lessons. After September 11th, the FAA prohibited box-cutters on airplanes - as if the problem had been the failure to take this particular "obvious" precaution. We don't learn the general lesson: the cost of effective caution is very high because you must attend to problems that are not as obvious now as past problems seem in hindsight.

The test of a model is how much probability it assigns to the observed outcome. Hindsight bias systematically distorts this test; we think our model assigned much more probability than it actually did. Instructing the jury doesn't help. You have to write down your predictions in advance. Or as Fischhoff (1982) put it:

When we attempt to understand past events, we implicitly test the hypotheses or rules we use both to interpret and to anticipate the world around us. If, in hindsight, we systematically underestimate the surprises that the past held and holds for us, we are subjecting those hypotheses to inordinately weak tests and, presumably, finding little reason to change them.

Fischhoff, B. 1982. For those condemned to study the past: Heuristics and biases in hindsight. In Kahneman et. al. 1982: 332–351.

Fischhoff, B., and Beyth, R. 1975. I knew it would happen: Remembered probabilities of once-future things. Organizational Behavior and Human Performance, 13: 1-16.

Kamin, K. and Rachlinski, J. 1995. Ex Post â‰  Ex Ante: Determining Liability in Hindsight. Law and Human Behavior, 19(1): 89-104.

Sanchiro, C. 2003. Finding Error. Mich. St. L. Rev. 1189.

Sequence: Mysterious Answers to Mysterious Questions

Conservation of Expected Evidence

Hindsight Devalues Science

Sequence: Noticing Confusion

Absence of Evidence Is Evidence of Absence

Hindsight Devalues Science

Referenced by: Hindsight Devalues Science • Fake Causality • The Futility of Emergence • Making History Available • Why is the Future So Absurd? • Occam's Razor • We Change Our Minds Less Often Than We Think • Priming and Contamination • Illusion of Transparency: Why No One Understands You • Fake Optimization Criteria • The Hidden Complexity of Wishes • Mere Messiahs • Affective Death Spirals • Buy Now Or Forever Hold Your Peace • Distinct Configurations • Qualitative Strategies of Friendliness • The True Prisoner's Dilemma

Original with comments: Hindsight bias

Hindsight Devalues Science

Eliezer Yudkowsky, 17 August 2007 07:39PM

This excerpt from Meyers's Exploring Social Psychology is worth reading in entirety. Cullen Murphy, editor of The Atlantic, said that the social sciences turn up "no ideas or conclusions that can't be found in [any] encyclopedia of quotations... Day after day social scientists go out into the world. Day after day they discover that people's behavior is pretty much what you'd expect."

Of course, the "expectation" is all hindsight. (Hindsight bias: Subjects who know the actual answer to a question assign much higher probabilities they "would have" guessed for that answer, compared to subjects who must guess without knowing the answer.)

The historian Arthur Schlesinger, Jr. dismissed scientific studies of WWII soldiers' experiences as "ponderous demonstrations" of common sense. For example:

1. Better educated soldiers suffered more adjustment problems than less educated soldiers. (Intellectuals were less prepared for battle stresses than street-smart people.)

2. Southern soldiers coped better with the hot South Sea Island climate than Northern soldiers. (Southerners are more accustomed to hot weather.)

3. White privates were more eager to be promoted to noncommissioned officers than Black privates. (Years of oppression take a toll on achievement motivation.)

4. Southern Blacks preferred Southern to Northern White officers (because Southern officers were more experienced and skilled in interacting with Blacks).

5. As long as the fighting continued, soldiers were more eager to return home than after the war ended. (During the fighting, soldiers knew they were in mortal danger.)

How many of these findings do you think you could have predicted in advance? 3 out of 5? 4 out of 5? Are there any cases where you would have predicted the opposite - where your model takes a hit? Take a moment to think before continuing...

In this demonstration (from Paul Lazarsfeld by way of Meyers), all of the findings above are the opposite of what was actually found. How many times did you think your model took a hit? How many times did you admit you would have been wrong? That's how good your model really was. The measure of your strength as a rationalist is your ability to be more confused by fiction than by reality.

Unless, of course, I reversed the results again. What do you think?

Do your thought processes at this point, where you really don't know the answer, feel different from the thought processes you used to rationalize either side of the "known" answer?

Daphna Baratz exposed college students to pairs of supposed findings, one true ("In prosperous times people spend a larger portion of their income than during a recession") and one the truth's opposite. In both sides of the pair, students rated the supposed finding as what they "would have predicted". Perfectly standard hindsight bias.

Which leads people to think they have no need for science, because they "could have predicted" that.

(Just as you would expect, right?)

Hindsight will lead us to systematically undervalue the surprisingness of scientific findings, especially the discoveries we understand - the ones that seem real to us, the ones we can retrofit into our models of the world. If you understand neurology or physics and read news in that topic, then you probably underestimate the surprisingness of findings in those fields too. This unfairly devalues the contribution of the researchers; and worse, will prevent you from noticing when you are seeing evidence that doesn't fit what you really would have expected.

We need to make a conscious effort to be shocked enough.

Sequence: Mysterious Answers to Mysterious Questions

Hindsight bias

Fake Explanations

Sequence: Noticing Confusion

Hindsight bias

Positive Bias: Look Into the Dark

Referenced by: Fake Explanations • Guessing the Teacher's Password • Fake Causality • The Futility of Emergence • Why is the Future So Absurd? • Fake Optimization Criteria • Beautiful Math • Words as Mental Paintbrush Handles • Heat vs. Motion • Can You Prove Two Particles Are Identical? • On Being Decoherent • Logical or Connectionist AI?

Original with comments: Hindsight Devalues Science

Scientific Evidence, Legal Evidence, Rational Evidence

Eliezer Yudkowsky, 19 August 2007 05:36AM

Suppose that your good friend, the police commissioner, tells you in strictest confidence that the crime kingpin of your city is Wulky Wilkinsen. As a rationalist, are you licensed to believe this statement? Put it this way: if you go ahead and mess around with Wulky's teenage daughter, I'd call you foolhardy. Since it is prudent to act as if Wulky has a substantially higher-than-default probability of being a crime boss, the police commissioner's statement must have been strong Bayesian evidence.

Our legal system will not imprison Wulky on the basis of the police commissioner's statement. It is not admissible as legal evidence. Maybe if you locked up every person accused of being a crime boss by a police commissioner, you'd initially catch a lot of crime bosses, plus some people that a police commissioner didn't like. Power tends to corrupt: over time, you'd catch fewer and fewer real crime bosses (who would go to greater lengths to ensure anonymity) and more and more innocent victims (unrestrained power attracts corruption like honey attracts flies).

This does not mean that the police commissioner's statement is not rational evidence. It still has a lopsided likelihood ratio, and you'd still be a fool to mess with Wulky's teenager daughter. But on a social level, in pursuit of a social goal, we deliberately define "legal evidence" to include only particular kinds of evidence, such as the police commissioner's own observations on the night of April 4th. All legal evidence should ideally be rational evidence, but not the other way around. We impose special, strong, additional standards before we anoint rational evidence as "legal evidence".

As I write this sentence at 8:33pm, Pacific time, on August 18th 2007, I am wearing white socks. As a rationalist, are you licensed to believe the previous statement? Yes. Could I testify to it in court? Yes. Is it a scientific statement? No, because there is no experiment you can perform yourself to verify it. Science is made up of generalizations which apply to many particular instances, so that you can run new real-world experiments which test the generalization, and thereby verify for yourself that the generalization is true, without having to trust anyone's authority. Science is the publicly reproducible knowledge of humankind.

Like a court system, science as a social process is made up of fallible humans. We want a protected pool of beliefs that are especially reliable. And we want social rules that encourage the generation of such knowledge. So we impose special, strong, additional standards before we canonize rational knowledge as "scientific knowledge", adding it to the protected belief pool.

Is a rationalist licensed to believe in the historical existence of Alexander the Great? Yes. We have a rough picture of ancient Greece, untrustworthy but better than maximum entropy. But we are dependent on authorities such as Plutarch; we cannot discard Plutarch and verify everything for ourselves. Historical knowledge is not scientific knowledge.

Is a rationalist licensed to believe that the Sun will rise on September 18th, 2007? Yes - not with absolute certainty, but that's the way to bet. (Pedants: interpret this as the Earth's rotation and orbit remaining roughly constant relative to the Sun.) Is this statement, as I write this essay on August 18th 2007, a scientific belief?

It may seem perverse to deny the adjective "scientific" to statements like "The Sun will rise on September 18th, 2007." If Science could not make predictions about future events - events which have not yet happened - then it would be useless; it could make no prediction in advance of experiment. The prediction that the Sun will rise is, definitely, an extrapolation from scientific generalizations. It is based upon models of the Solar System which you could test for yourself by experiment.

But imagine that you're constructing an experiment to verify prediction #27, in a new context, of an accepted theory Q. You may not have any concrete reason to suspect the belief is wrong; you just want to test it in a new context. It seems dangerous to say, before running the experiment, that there is a "scientific belief" about the result. There is a "conventional prediction" or "theory Q's prediction". But if you already know the "scientific belief" about the result, why bother to run the experiment?

You begin to see, I hope, why I identify Science with generalizations, rather than the history of any one experiment. A historical event happens once; generalizations apply over many events. History is not reproducible; scientific generalizations are.

Is my definition of "scientific knowledge" true? That is not a well-formed question. The special standards we impose upon science are pragmatic choices. Nowhere upon the stars or the mountains is it written that p<0.05 shall be the standard for scientific publication. Many now argue that 0.05 is too weak, and that it would be useful to lower it to 0.01 or 0.001.

Perhaps future generations, acting on the theory that science is the public, reproducible knowledge of humankind, will only label as "scientific" papers published in an open-access journal. If you charge for access to the knowledge, is it part of the knowledge of humankind? Can we trust a result if people must pay to criticize it? Is it really science?

The question "Is it really science?" is ill-formed. Is a $20,000/year closed-access journal really Bayesian evidence? As with the police commissioner's private assurance that Wulky is the kingpin, I think we must answer "Yes." But should the closed-access journal be further canonized as "science"? Should we allow it into the special, protected belief pool? For myself, I think science would be better served by the dictum that only open knowledge counts as the public, reproducible knowledge pool of humankind.

Referenced by: Is Molecular Nanotechnology "Scientific"? • Rational vs. Scientific Ev-Psych • Newcomb's Problem and Regret of Rationality • To Spread Science, Keep It Secret • Decoherence is Falsifiable and Testable • Many Worlds, One Best Guess • Faster Than Science • Why Quantum? • The Genetic Fallacy • Changing Your Metaethics • Eutopia is Scary • 3 Levels of Rationality Verification • You're Entitled to Arguments, But Not (That Particular) Proof

Original with comments: Scientific Evidence, Legal Evidence, Rational Evidence

Is Molecular Nanotechnology "Scientific"?

Eliezer Yudkowsky, 20 August 2007 04:11AM

Prerequisite / Read this first: Scientific Evidence, Legal Evidence, Rational Evidence

Consider the statement "It is physically possible to construct diamondoid nanomachines which repair biological cells." Some people will tell you that molecular nanotechnology is "pseudoscience" because it has not been verified by experiment - no one has ever seen a nanofactory, so how can believing in their possibility be scientific?

Drexler, I think, would reply that his extrapolations of diamondoid nanomachines are based on standard physics, which is to say, scientific generalizations. Therefore, if you say that nanomachines cannot work, you must be inventing new physics. Or to put it more sharply: If you say that a simulation of a molecular gear is inaccurate, if you claim that atoms thus configured would behave differently from depicted, then either you know a flaw in the simulation algorithm or you're inventing your own laws of physics.

My own sympathies, I confess, are with Drexler. And not just because you could apply the same argument of "I've never seen it, therefore it can't happen" to my own field of Artificial Intelligence.

What about the Wright Brothers' attempt to build a non-biological heavier-than-air powered flying machine? Was that "pseudoscience"? No one had ever seen one before. Wasn't "all flying machines crash" a generalization true over all previous observations? Wouldn't it be scientific to extend this generalization to predict future experiments?

"Flying machines crash" is a qualitative, imprecise, verbal, surface-level generalization. If you have a quantitative theory of aerodynamics which can calculate precisely how previous flying machines crashed, that same theory of aerodynamics would predict the Wright Flyer will fly (and how high, at what speed). Deep quantitative generalizations take strict precedence over verbal surface generalizations. Only deep laws possess the absolute universality and stability of physics. Perhaps there are no new quarks under the Sun, but on higher levels of organization, new things happen all the time.

"No one has ever seen a non-biological nanomachine" is a verbalish surface-level generalization, which can hardly overrule the precise physical models used to simulate a molecular gear.

And yet... I still would not say that "It's possible to construct a nanofactory" is a scientific belief. This belief will not become scientific until someone actually constructs a nanofactory. Just because something is the best extrapolation from present generalizations, doesn't make it true. We have not done an atom-by-atom calculation for the synthesis and behavior of an entire nanofactory; the argument for nanofactories is based on qualitative, abstract reasoning. Such reasoning, even from the best available current theories, sometimes goes wrong. Not always, but sometimes.

The argument for "it's possible to construct a nanofactory" is based on the protected belief pool of science. But it does not, itself, meet the special strong standards required to ceremonially add a belief to the protected belief pool.

Yet if, on a whim, you decide to make a strong positive assertion that nanomachines are impossible, you are being irrational. You are even being "unscientific". An ungrounded whimsical assertion that tomorrow the Sun will not rise is "unscientific", because you have needlessly contradicted the best extrapolation from current scientific knowledge.

In the nanotechnology debate, we see once again the severe folly of thinking that everything which is not science is pseudoscience - as if Nature is prohibited from containing any truths except those already verified by surface observations of scientific experiments. It is a fallacy of the excluded middle.

Of course you could try to criticize the feasibility of diamondoid nanotechnology from within the known laws of physics. That could be argued. It wouldn't have the just plain silly quality of "Nanotech is pseudoscience because no one's ever seen a nanotech." Drexler used qualitative, abstract reasoning from known science; perhaps his argument has a hidden flaw according to known science.

For now, "diamondoid nanosystems are possible" is merely a best guess. It is merely based on qualitative, abstract, approximate, potentially fallible reasoning from beliefs already in the protected belief pool of science. Such a guess is not reliable enough itself to be added to the protected belief pool. It is merely rational.

Referenced by: Science as Attire • Absurdity Heuristic, Absurdity Bias • Hug the Query

Original with comments: Is Molecular Nanotechnology "Scientific"?

Fake Explanations

Eliezer Yudkowsky, 20 August 2007 09:13PM

Once upon a time, there was an instructor who taught physics students. One day she called them into her class, and showed them a wide, square plate of metal, next to a hot radiator. The students each put their hand on the plate, and found the side next to the radiator cool, and the distant side warm. And the instructor said, Why do you think this happens? Some students guessed convection of air currents, and others guessed strange metals in the plate. They devised many creative explanations, none stooping so low as to say "I don't know" or "This seems impossible."

And the answer was that before the students entered the room, the instructor turned the plate around.

Consider the student who frantically stammers, "Eh, maybe because of the heat conduction and so?" I ask: is this answer a proper belief? The words are easily enough professed - said in a loud, emphatic voice. But do the words actually control anticipation?

Ponder that innocent little phrase, "because of", which comes before "heat conduction". Ponder some of the other things we could put after it. We could say, for example, "Because of phlogiston", or "Because of magic."

"Magic!" you cry. "That's not a scientific explanation!" Indeed, the phrases "because of heat conduction" and "because of magic" are readily recognized as belonging to different literary genres. "Heat conduction" is something that Spock might say on Star Trek, whereas "magic" would be said by Giles in Buffy the Vampire Slayer.

However, as Bayesians, we take no notice of literary genres. For us, the substance of a model is the control it exerts on anticipation. If you say "heat conduction", what experience does that lead you to anticipate? Under normal circumstances, it leads you to anticipate that, if you put your hand on the side of the plate near the radiator, that side will feel warmer than the opposite side. If "because of heat conduction" can also explain the radiator-adjacent side feeling cooler, then it can explain pretty much anything.

And as we all know by this point (I do hope), if you are equally good at explaining any outcome, you have zero knowledge. "Because of heat conduction", used in such fashion, is a disguised hypothesis of maximum entropy. It is anticipation-isomorphic to saying "magic". It feels like an explanation, but it's not.

Supposed that instead of guessing, we measured the heat of the metal plate at various points and various times. Seeing a metal plate next to the radiator, we would ordinarily expect the point temperatures to satisfy an equilibrium of the diffusion equation with respect to the boundary conditions imposed by the environment. You might not know the exact temperature of the first point measured, but after measuring the first points - I'm not physicist enough to know how many would be required - you could take an excellent guess at the rest.

A true master of the art of using numbers to constrain the anticipation of material phenomena - a "physicist" - would take some measurements and say, "This plate was in equilibrium with the environment two and a half minutes ago, turned around, and is now approaching equilibrium again."

The deeper error of the students is not simply that they failed to constrain anticipation. Their deeper error is that they thought they were doing physics. They said the phrase "because of", followed by the sort of words Spock might say on Star Trek, and thought they thereby entered the magisterium of science.

Not so. They simply moved their magic from one literary genre to another.

Sequence: Mysterious Answers to Mysterious Questions

Hindsight Devalues Science

Guessing the Teacher's Password

Referenced by: Guessing the Teacher's Password • Science as Attire • Fake Causality • Mysterious Answers to Mysterious Questions • The Futility of Emergence • Say Not "Complexity" • My Wild and Reckless Youth • "Science" as Curiosity-Stopper • Occam's Razor • Fake Optimization Criteria • Disputing Definitions • Fake Reductionism • The Beauty of Settled Science • To Spread Science, Keep It Secret • Where Experience Confuses Physicists • Is Fairness Arbitrary? • Complexity and Intelligence • Recognizing Intelligence • Go Forth and Create the Art!

Original with comments: Fake Explanations

Guessing the Teacher's Password

Eliezer Yudkowsky, 22 August 2007 03:40AM

Followup to: Fake Explanations

When I was young, I read popular physics books such as Richard Feynman's QED: The Strange Theory of Light and Matter. I knew that light was waves, sound was waves, matter was waves. I took pride in my scientific literacy, when I was nine years old.

When I was older, and I began to read the Feynman Lectures on Physics, I ran across a gem called "the wave equation". I could follow the equation's derivation, but, looking back, I couldn't see its truth at a glance. So I thought about the wave equation for three days, on and off, until I saw that it was embarrassingly obvious. And when I finally understood, I realized that the whole time I had accepted the honest assurance of physicists that light was waves, sound was waves, matter was waves, I had not had the vaguest idea of what the word "wave" meant to a physicist.

There is an instinctive tendency to think that if a physicist says "light is made of waves", and the teacher says "What is light made of?", and the student says "Waves!", the student has made a true statement. That's only fair, right? We accept "waves" as a correct answer from the physicist; wouldn't it be unfair to reject it from the student? Surely, the answer "Waves!" is either true or false, right?

Which is one more bad habit to unlearn from school. Words do not have intrinsic definitions. If I hear the syllables "bea-ver" and think of a large rodent, that is a fact about my own state of mind, not a fact about the syllables "bea-ver". The sequence of syllables "made of waves" (or "because of heat conduction") is not a hypothesis, it is a pattern of vibrations traveling through the air, or ink on paper. It can associate to a hypothesis in someone's mind, but it is not, of itself, right or wrong. But in school, the teacher hands you a gold star for saying "made of waves", which must be the correct answer because the teacher heard a physicist emit the same sound-vibrations. Since verbal behavior (spoken or written) is what gets the gold star, students begin to think that verbal behavior has a truth-value. After all, either light is made of waves, or it isn't, right?

And this leads into an even worse habit. Suppose the teacher presents you with a confusing problem involving a metal plate next to a radiator; the far side feels warmer than the side next to the radiator. The teacher asks "Why?" If you say "I don't know", you have no chance of getting a gold star - it won't even count as class participation. But, during the current semester, this teacher has used the phrases "because of heat convection", "because of heat conduction", and "because of radiant heat". One of these is probably what the teacher wants. You say, "Eh, maybe because of heat conduction?"

This is not a hypothesis about the metal plate. This is not even a proper belief. It is an attempt to guess the teacher's password.

Even visualizing the symbols of the diffusion equation (the math governing heat conduction) doesn't mean you've formed a hypothesis about the metal plate. This is not school; we are not testing your memory to see if you can write down the diffusion equation. This is Bayescraft; we are scoring your anticipations of experience. If you use the diffusion equation, by measuring a few points with a thermometer and then trying to predict what the thermometer will say on the next measurement, then it is definitely connected to experience. Even if the student just visualizes something flowing, and therefore holds a match near the cooler side of the plate to try to measure where the heat goes, then this mental image of flowing-ness connects to experience; it controls anticipation.

If you aren't using the diffusion equation - putting in numbers and getting out results that control your anticipation of particular experiences - then the connection between map and territory is severed as though by a knife. What remains is not a belief, but a verbal behavior.

In the school system, it's all about verbal behavior, whether written on paper or spoken aloud. Verbal behavior gets you a gold star or a failing grade. Part of unlearning this bad habit is becoming consciously aware of the difference between an explanation and a password.

Does this seem too harsh? When you're faced by a confusing metal plate, can't "Heat conduction?" be a first step toward finding the answer? Maybe, but only if you don't fall into the trap of thinking that you are looking for a password. What if there is no teacher to tell you that you failed? Then you may think that "Light is wakalixes" is a good explanation, that "wakalixes" is the correct password. It happened to me when I was nine years old - not because I was stupid, but because this is what happens by default. This is how human beings think, unless they are trained not to fall into the trap. Humanity stayed stuck in holes like this for thousands of years.

Maybe, if we drill students that words don't count, only anticipation-controllers, the student will not get stuck on "Heat conduction? No? Maybe heat convection? That's not it either?" Maybe then, thinking the phrase "Heat conduction" will lead onto a genuinely helpful path, like:

- "Heat conduction?"

- But that's only a phrase - what does it mean?

- The diffusion equation?

- But those are only symbols - how do I apply them?

- What does applying the diffusion equation lead me to anticipate?

- It sure doesn't lead me to anticipate that the side of a metal plate farther away from a radiator would feel warmer.

- I notice that I am confused. Maybe the near side just feels cooler, because it's made of more insulative material and transfers less heat to my hand? I'll try measuring the temperature...

- Okay, that wasn't it. Can I try to verify whether the diffusion equation holds true of this metal plate, at all? Is heat flowing the way it usually does, or is something else going on?

- I could hold a match to the plate and try to measure how heat spreads over time...

If we are not strict about "Eh, maybe because of heat conduction?" being a fake explanation, the student will very probably get stuck on some wakalixes-password. This happens by default, it happened to the whole human species for thousands of years.

(This post is part of the sequence Mysterious Answers to Mysterious Questions.)

Sequence: Mysterious Answers to Mysterious Questions

Fake Explanations

Science as Attire

Referenced by: Science as Attire • Fake Causality • The Futility of Emergence • Say Not "Complexity" • "Science" as Curiosity-Stopper • Conjunction Controversy (Or, How They Nail It Down) • Double Illusion of Transparency • An Alien God • Evolutions Are Stupid (But Work Anyway) • Fake Optimization Criteria • Artificial Addition • Truly Part Of You • Taboo Your Words • Perpetual Motion Beliefs • Searching for Bayes-Structure • Fake Reductionism • Joy in Discovery • The Beauty of Settled Science • Angry Atoms • Identity Isn't In Specific Atoms • The So-Called Heisenberg Uncertainty Principle • Quantum Non-Realism • Principles of Disagreement • The Gift We Give To Tomorrow • Does Your Morality Care What You Think? • Setting Up Metaethics • Building Something Smarter • Living By Your Own Strength • Against Maturity • Go Forth and Create the Art!

Original with comments: Guessing the Teacher's Password

Science as Attire

Eliezer Yudkowsky, 23 August 2007 05:10AM

Prerequisites: Fake Explanations, Belief As Attire

The preview for the X-Men movie has a voice-over saying: "In every human being... there is the genetic code... for mutation." Apparently you can acquire all sorts of neat abilities by mutation. The mutant Storm, for example, has the ability to throw lightning bolts.

I beg you, dear reader, to consider the biological machinery necessary to generate electricity; the biological adaptations necessary to avoid being harmed by electricity; and the cognitive circuitry required for finely tuned control of lightning bolts. If we actually observed any organism acquiring these abilities in one generation, as the result of mutation, it would outright falsify the neo-Darwinian model of natural selection. It would be worse than finding rabbit fossils in the pre-Cambrian. If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.

The X-Men comics use terms like "evolution", "mutation", and "genetic code", purely to place themselves in what they conceive to be the literary genre of science. The part that scares me is wondering how many people, especially in the media, understand science only as a literary genre.

I encounter people who very definitely believe in evolution, who sneer at the folly of creationists. And yet they have no idea of what the theory of evolutionary biology permits and prohibits. They'll talk about "the next step in the evolution of humanity", as if natural selection got here by following a plan. Or even worse, they'll talk about something completely outside the domain of evolutionary biology, like an improved design for computer chips, or corporations splitting, or humans uploading themselves into computers, and they'll call that "evolution". If evolutionary biology could cover that, it could cover anything.

Probably an actual majority of the people who believe in evolution use the phrase "because of evolution" because they want to be part of the scientific in-crowd - belief as scientific attire, like wearing a lab coat. If the scientific in-crowd instead used the phrase "because of intelligent design", they would just as cheerfully use that instead - it would make no difference to their anticipation-controllers. Saying "because of evolution" instead of "because of intelligent design" does not, for them, prohibit Storm. Its only purpose, for them, is to identify with a tribe.

I encounter people who are quite willing to entertain the notion of dumber-than-human Artificial Intelligence, or even mildly smarter-than-human Artificial Intelligence. Introduce the notion of strongly superhuman Artificial Intelligence, and they'll suddenly decide it's "pseudoscience". It's not that they think they have a theory of intelligence which lets them calculate a theoretical upper bound on the power of an optimization process. Rather, they associate strongly superhuman AI to the literary genre of apocalyptic literature; whereas an AI running a small corporation associates to the literary genre of Wired magazine. They aren't speaking from within a model of cognition. They don't realize they need a model. They don't realize that science is about models. Their devastating critiques consist purely of comparisons to apocalyptic literature, rather than, say, known laws which prohibit such an outcome. They understand science only as a literary genre, or in-group to belong to. The attire doesn't look to them like a lab coat; this isn't the football team they're cheering for.

Is there anything in science that you are proud of believing, and yet you do not use the belief professionally? You had best ask yourself which future experiences your belief prohibits from happening to you. That is the sum of what you have assimilated and made a true part of yourself. Anything else is probably passwords or attire.

Sequence: Mysterious Answers to Mysterious Questions

Guessing the Teacher's Password

Fake Causality

Referenced by: My Wild and Reckless Youth • An Alien God • No Evolutions for Corporations or Nanodevices • Guardians of Ayn Rand • Zen and the Art of Rationality • Fake Reductionism • Angry Atoms • The Psychological Unity of Humankind • Is That Your True Rejection? • High Status and Stupidity: Why?

Original with comments: Science as Attire

Fake Causality

Eliezer Yudkowsky, 23 August 2007 06:12PM

Followup to: Fake Explanations, Guessing the Teacher's Password

Phlogiston was the 18 century's answer to the Elemental Fire of the Greek alchemists. Ignite wood, and let it burn. What is the orangey-bright "fire" stuff? Why does the wood transform into ash? To both questions, the 18th-century chemists answered, "phlogiston".

...and that was it, you see, that was their answer: "Phlogiston."

Phlogiston escaped from burning substances as visible fire. As the phlogiston escaped, the burning substances lost phlogiston and so became ash, the "true material". Flames in enclosed containers went out because the air became saturated with phlogiston, and so could not hold any more. Charcoal left little residue upon burning because it was nearly pure phlogiston.

Of course, one didn't use phlogiston theory to predict the outcome of a chemical transformation. You looked at the result first, then you used phlogiston theory to explain it. It's not that phlogiston theorists predicted a flame would extinguish in a closed container; rather they lit a flame in a container, watched it go out, and then said, "The air must have become saturated with phlogiston." You couldn't even use phlogiston theory to say what you ought not to see; it could explain everything.

This was an earlier age of science. For a long time, no one realized there was a problem. Fake explanations don't feel fake. That's what makes them dangerous.

Modern research suggests that humans think about cause and effect using something like the directed acyclic graphs (DAGs) of Bayes nets. Because it rained, the sidewalk is wet; because the sidewalk is wet, it is slippery:

[Rain] -> [Sidewalk wet] -> [Sidewalk slippery]

From this we can infer - or, in a Bayes net, rigorously calculate in probabilities - that when the sidewalk is slippery, it probably rained; but if we already know that the sidewalk is wet, learning that the sidewalk is slippery tells us nothing more about whether it rained.

Why is fire hot and bright when it burns?

["Phlogiston"] -> [Fire hot and bright]

It feels like an explanation. It's represented using the same cognitive data format. But the human mind does not automatically detect when a cause has an unconstraining arrow to its effect. Worse, thanks to hindsight bias, it may feel like the cause constrains the effect, when it was merely fitted to the effect.

Interestingly, our modern understanding of probabilistic reasoning about causality can describe precisely what the phlogiston theorists were doing wrong. One of the primary inspirations for Bayesian networks was noticing the problem of double-counting evidence if inference resonates between an effect and a cause. For example, let's say that I get a bit of unreliable information that the sidewalk is wet. This should make me think it's more likely to be raining. But, if it's more likely to be raining, doesn't that make it more likely that the sidewalk is wet? And wouldn't that make it more likely that the sidewalk is slippery? But if the sidewalk is slippery, it's probably wet; and then I should again raise my probability that it's raining...

Judea Pearl uses the metaphor of an algorithm for counting soldiers in a line. Suppose you're in the line, and you see two soldiers next to you, one in front and one in back. That's three soldiers. So you ask the soldier next to you, "How many soldiers do you see?" He looks around and says, "Three". So that's a total of six soldiers. This, obviously, is not how to do it.

A smarter way is to ask the soldier in front of you, "How many soldiers forward of you?" and the soldier in back, "How many soldiers backward of you?" The question "How many soldiers forward?" can be passed on as a message without confusion. If I'm at the front of the line, I pass the message "1 soldier forward", for myself. The person directly in back of me gets the message "1 soldier forward", and passes on the message "2 soldiers forward" to the soldier behind him. At the same time, each soldier is also getting the message "N soldiers backward" from the soldier behind them, and passing it on as "N+1 soldiers backward" to the soldier in front of them. How many soldiers in total? Add the two numbers you receive, plus one for yourself: that is the total number of soldiers in line.

The key idea is that every soldier must separately track the two messages, the forward-message and backward-message, and add them together only at the end. You never add any soldiers from the backward-message you receive to the forward-message you pass back. Indeed, the total number of soldiers is never passed as a message - no one ever says it aloud.

An analogous principle operates in rigorous probabilistic reasoning about causality. If you learn something about whether it's raining, from some source other than observing the sidewalk to be wet, this will send a forward-message from [rain] to [sidewalk wet] and raise our expectation of the sidewalk being wet. If you observe the sidewalk to be wet, this sends a backward-message to our belief that it is raining, and this message propagates from [rain] to all neighboring nodes except the [sidewalk wet] node. We count each piece of evidence exactly once; no update message ever "bounces" back and forth. The exact algorithm may be found in Judea Pearl's classic "Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference".

So what went wrong in phlogiston theory? When we observe that fire is hot, the [fire] node can send a backward-evidence to the ["phlogiston"] node, leading us to update our beliefs about phlogiston. But if so, we can't count this as a successful forward-prediction of phlogiston theory. The message should go in only one direction, and not bounce back.

Alas, human beings do not use a rigorous algorithm for updating belief networks. We learn about parent nodes from observing children, and predict child nodes from beliefs about parents. But we don't keep rigorously separate books for the backward-message and forward-message. We just remember that phlogiston is hot, which causes fire to be hot. So it seems like phlogiston theory predicts the hotness of fire. Or, worse, it just feels like phlogiston makes the fire hot.

Until you notice that no advance predictions are being made, the non-constraining causal node is not labeled "fake". It's represented the same way as any other node in your belief network. It feels like a fact, like all the other facts you know: Phlogiston makes the fire hot.

A properly designed AI would notice the problem instantly. This wouldn't even require special-purpose code, just correct bookkeeping of the belief network. (Sadly, we humans can't rewrite our own code, the way a properly designed AI could.)

Speaking of "hindsight bias" is just the nontechnical way of saying that humans do not rigorously separate forward and backward messages, allowing forward messages to be contaminated by backward ones.

Those who long ago went down the path of phlogiston were not trying to be fools. No scientist deliberately wants to get stuck in a blind alley. Are there any fake explanations in your mind? If there are, I guarantee they're not labeled "fake explanation", so polling your thoughts for the "fake" keyword will not turn them up.

Thanks to hindsight bias, it's also not enough to check how well your theory "predicts" facts you already know. You've got to predict for tomorrow, not yesterday. It's the only way a messy human mind can be guaranteed of sending a pure forward message.

Sequence: Mysterious Answers to Mysterious Questions

Science as Attire

Semantic Stopsigns

Referenced by: Mysterious Answers to Mysterious Questions • The Futility of Emergence • Positive Bias: Look Into the Dark • Say Not "Complexity" • Occam's Razor • We Change Our Minds Less Often Than We Think • Priming and Contamination • The Logical Fallacy of Generalization from Fictional Evidence • Illusion of Transparency: Why No One Understands You • Fake Optimization Criteria • Leaky Generalizations • Affective Death Spirals • Neural Categories • Conditional Independence, and Naive Bayes • If You Demand Magic, Magic Won't Help • Heat vs. Motion • Zombies! Zombies? • Where Experience Confuses Physicists • The Conscious Sorites Paradox • Passing the Recursive Buck • Dreams of AI Design • Complexity and Intelligence • Raising the Sanity Waterline

Original with comments: Fake Causality

Semantic Stopsigns

Eliezer Yudkowsky, 24 August 2007 07:29PM

And the child asked:

Q: Where did this rock come from?
A: I chipped it off the big boulder, at the center of the village.
Q: Where did the boulder come from?
A: It probably rolled off the huge mountain that towers over our village.
Q: Where did the mountain come from?
A: The same place as all stone: it is the bones of Ymir, the primordial giant.
Q: Where did the primordial giant, Ymir, come from?
A: From the great abyss, Ginnungagap.
Q: Where did the great abyss, Ginnungagap, come from?
A: Never ask that question.

Consider the seeming paradox of the First Cause. Science has traced events back to the Big Bang, but why did the Big Bang happen? It's all well and good to say that the zero of time begins at the Big Bang - that there is nothing before the Big Bang in the ordinary flow of minutes and hours. But saying this presumes our physical law, which itself appears highly structured; it calls out for explanation. Where did the physical laws come from? You could say that we're all a computer simulation, but then the computer simulation is running on some other world's laws of physics - where did those laws of physics come from?

At this point, some people say, "God!"

What could possibly make anyone, even a highly religious person, think this even helped answer the paradox of the First Cause? Why wouldn't you automatically ask, "Where did God come from?" Saying "God is uncaused" or "God created Himself" leaves us in exactly the same position as "Time began with the Big Bang." We just ask why the whole metasystem exists in the first place, or why some events but not others are allowed to be uncaused.

My purpose here is not to discuss the seeming paradox of the First Cause, but to ask why anyone would think "God!" could resolve the paradox. Saying "God!" is a way of belonging to a tribe, which gives people a motive to say it as often as possible - some people even say it for questions like "Why did this hurricane strike New Orleans?" Even so, you'd hope people would notice that on the particular puzzle of the First Cause, saying "God!" doesn't help. It doesn't make the paradox seem any less paradoxical even if true. How could anyone not notice this?

Jonathan Wallace suggested that "God!" functions as a semantic stopsign - that it isn't a propositional assertion, so much as a cognitive traffic signal: do not think past this point. Saying "God!" doesn't so much resolve the paradox, as put up a cognitive traffic signal to halt the obvious continuation of the question-and-answer chain.

Of course you'd never do that, being a good and proper atheist, right? But "God!" isn't the only semantic stopsign, just the obvious first example.

The transhuman technologies - molecular nanotechnology, advanced biotech, genetech, Artificial Intelligence, et cetera - pose tough policy questions. What kind of role, if any, should a government take in supervising a parent's choice of genes for their child? Could parents deliberately choose genes for schizophrenia? If enhancing a child's intelligence is expensive, should governments help ensure access, to prevent the emergence of a cognitive elite? You can propose various institutions to answer these policy questions - for example, that private charities should provide financial aid for intelligence enhancement - but the obvious next question is, "Will this institution be effective?" If we rely on product liability lawsuits to prevent corporations from building harmful nanotech, will that really work?

I know someone whose answer to every one of these questions is "Liberal democracy!" That's it. That's his answer. If you ask the obvious question of "How well have liberal democracies performed, historically, on problems this tricky?" or "What if liberal democracy does something stupid?" then you're an autocrat, or libertopian, or otherwise a very very bad person. No one is allowed to question democracy.

I once called this kind of thinking "the divine right of democracy". But it is more precise to say that "Democracy!" functioned for him as a semantic stopsign. If anyone had said to him "Turn it over to the Coca-Cola corporation!", he would have asked the obvious next questions: "Why? What will the Coca-Cola corporation do about it? Why should we trust them? Have they done well in the past on equally tricky problems?"

Or suppose that someone says "Mexican-Americans are plotting to remove all the oxygen in Earth's atmosphere." You'd probably ask, "Why would they do that? Don't Mexican-Americans have to breathe too? Do Mexican-Americans even function as a unified conspiracy?" If you don't ask these obvious next questions when someone says, "Corporations are plotting to remove Earth's oxygen," then "Corporations!" functions for you as a semantic stopsign.

Be careful here not to create a new generic counterargument against things you don't like - "Oh, it's just a stopsign!" No word is a stopsign of itself; the question is whether a word has that effect on a particular person. Having strong emotions about something doesn't qualify it as a stopsign. I'm not exactly fond of terrorists or fearful of private property; that doesn't mean "Terrorists!" or "Capitalism!" are cognitive traffic signals unto me. (The word "intelligence" did once have that effect on me, though no longer.) What distinguishes a semantic stopsign is failure to consider the obvious next question.

(This post is part of the sequence Mysterious Answers to Mysterious Questions.)

Sequence: Mysterious Answers to Mysterious Questions

Fake Causality

Mysterious Answers to Mysterious Questions

Referenced by: Mysterious Answers to Mysterious Questions • Explain/Worship/Ignore? • "Science" as Curiosity-Stopper • Applause Lights • Rationality and the English Language • A Priori • An Alien God • Fake Optimization Criteria • Fake Reductionism • Quantum Non-Realism • Many Worlds, One Best Guess • Do Scientists Already Know This Stuff? • Timeless Identity • Passing the Recursive Buck • The Moral Void • Crisis of Faith • Lawful Creativity • Raising the Sanity Waterline • That Crisis thing seems pretty useful • Logical Rudeness • Conversation Halters

Original with comments: Semantic Stopsigns

Mysterious Answers to Mysterious Questions

Eliezer Yudkowsky, 25 August 2007 10:27PM

Imagine looking at your hand, and knowing nothing of cells, nothing of biochemistry, nothing of DNA. You've learned some anatomy from dissection, so you know your hand contains muscles; but you don't know why muscles move instead of lying there like clay. Your hand is just... stuff... and for some reason it moves under your direction. Is this not magic?

"The animal body does not act as a thermodynamic engine ... consciousness teaches every individual that they are, to some extent, subject to the direction of his will. It appears therefore that animated creatures have the power of immediately applying to certain moving particles of matter within their bodies, forces by which the motions of these particles are directed to produce derived mechanical effects... The influence of animal or vegetable life on matter is infinitely beyond the range of any scientific inquiry hitherto entered on. Its power of directing the motions of moving particles, in the demonstrated daily miracle of our human free-will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concurrence of atoms... Modern biologists were coming once more to the acceptance of something and that was a vital principle."
-- Lord Kelvin

This was the theory of vitalism; that the mysterious difference between living matter and non-living matter was explained by an elan vital or vis vitalis. Elan vital infused living matter and caused it to move as consciously directed. Elan vital participated in chemical transformations which no mere non-living particles could undergo - Wöhler's later synthesis of urea, a component of urine, was a major blow to the vitalistic theory because it showed that mere chemistry could duplicate a product of biology.

Calling "elan vital" an explanation, even a fake explanation like phlogiston, is probably giving it too much credit. It functioned primarily as a curiosity-stopper. You said "Why?" and the answer was "Elan vital!"

When you say "Elan vital!", it feels like you know why your hand moves. You have a little causal diagram in your head that says ["Elan vital!"] -> [hand moves]. But actually you know nothing you didn't know before. You don't know, say, whether your hand will generate heat or absorb heat, unless you have observed the fact already; if not, you won't be able to predict it in advance. Your curiosity feels sated, but it hasn't been fed. Since you can say "Why? Elan vital!" to any possible observation, it is equally good at explaining all outcomes, a disguised hypothesis of maximum entropy, etcetera.

But the greater lesson lies in the vitalists' reverence for the elan vital, their eagerness to pronounce it a mystery beyond all science. Meeting the great dragon Unknown, the vitalists did not draw their swords to do battle, but bowed their necks in submission. They took pride in their ignorance, made biology into a sacred mystery, and thereby became loath to relinquish their ignorance when evidence came knocking.

The Secret of Life was infinitely beyond the reach of science! Not just a little beyond, mind you, but infinitely beyond! Lord Kelvin sure did get a tremendous emotional kick out of not knowing something.

But ignorance exists in the map, not in the territory. If I am ignorant about a phenomenon, that is a fact about my own state of mind, not a fact about the phenomenon itself. A phenomenon can seem mysterious to some particular person. There are no phenomena which are mysterious of themselves. To worship a phenomenon because it seems so wonderfully mysterious, is to worship your own ignorance.

Vitalism shared with phlogiston the error of encapsulating the mystery as a substance. Fire was mysterious, and the phlogiston theory encapsulated the mystery in a mysterious substance called "phlogiston". Life was a sacred mystery, and vitalism encapsulated the sacred mystery in a mysterious substance called "elan vital". Neither answer helped concentrate the model's probability density - make some outcomes easier to explain than others. The "explanation" just wrapped up the question as a small, hard, opaque black ball.

In a comedy written by Moliere, a physician explains the power of a soporific by saying that it contains a "dormitive potency". Same principle. It is a failure of human psychology that, faced with a mysterious phenomenon, we more readily postulate mysterious inherent substances than complex underlying processes.

But the deeper failure is supposing that an answer can be mysterious. If a phenomenon feels mysterious, that is a fact about our state of knowledge, not a fact about the phenomenon itself. The vitalists saw a mysterious gap in their knowledge, and postulated a mysterious stuff that plugged the gap. In doing so, they mixed up the map with the territory. All confusion and bewilderment exist in the mind, not in encapsulated substances.

This is the ultimate and fully general explanation for why, again and again in humanity's history, people are shocked to discover that an incredibly mysterious question has a non-mysterious answer. Mystery is a property of questions, not answers.

Therefore I call theories such as vitalism mysterious answers to mysterious questions.

These are the signs of mysterious answers to mysterious questions:

- First, the explanation acts as a curiosity-stopper rather than an anticipation-controller.

- Second, the hypothesis has no moving parts - the model is not a specific complex mechanism, but a blankly solid substance or force. The mysterious substance or mysterious force may be said to be here or there, to cause this or that; but the reason why the mysterious force behaves thus is wrapped in a blank unity.

- Third, those who proffer the explanation cherish their ignorance; they speak proudly of how the phenomenon defeats ordinary science or is unlike merely mundane phenomena.

- Fourth, even after the answer is given, the phenomenon is still a mystery and possesses the same quality of wonderful inexplicability that it had at the start.

Sequence: Mysterious Answers to Mysterious Questions

Semantic Stopsigns

The Futility of Emergence

Referenced by: The Futility of Emergence • My Wild and Reckless Youth • Failing to Learn from History • Making History Available • Explain/Worship/Ignore? • "Science" as Curiosity-Stopper • The Lens That Sees Its Flaws • No One Knows What Science Doesn't Know • An Alien God • Fake Optimization Criteria • Artificial Addition • Uncritical Supercriticality • Feel the Meaning • Wrong Questions • Joy in the Merely Real • Zombies! Zombies? • Where Experience Confuses Physicists • The Conscious Sorites Paradox • The Born Probabilities • Quantum Non-Realism • Do Scientists Already Know This Stuff? • Timeless Beauty • A Premature Word on AI • Passing the Recursive Buck • Grasping Slippery Things • The Gift We Give To Tomorrow • Changing Your Metaethics • Setting Up Metaethics • "Arbitrary" • When Anthropomorphism Became Stupid • Three Fallacies of Teleology • Dreams of AI Design • Excluding the Supernatural • Raised in Technophilia • The Sheer Folly of Callow Youth • On Doing the Impossible • Building Something Smarter • Living By Your Own Strength • Nonperson Predicates • Unteachable Excellence • Raising the Sanity Waterline • Atheism = Untheism + Antitheism • Why I'm Staying On Bloggingheads.tv

Original with comments: Mysterious Answers to Mysterious Questions

The Futility of Emergence

Eliezer Yudkowsky, 26 August 2007 10:10PM

Prerequisites: Belief in Belief, Fake Explanations, Fake Causality, Mysterious Answers to Mysterious Questions

The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a limb, and name some current theory which I deem analogously flawed?

I name emergence or emergent phenomena - usually defined as the study of systems whose high-level behaviors arise or "emerge" from the interaction of many low-level elements. (Wikipedia: "The way complex systems and patterns arise out of a multiplicity of relatively simple interactions".) Taken literally, that description fits every phenomenon in our universe above the level of individual quarks, which is part of the problem. Imagine pointing to a market crash and saying "It's not a quark!" Does that feel like an explanation? No? Then neither should saying "It's an emergent phenomenon!"

It's the noun "emergence" that I protest, rather than the verb "emerges from". There's nothing wrong with saying "X emerges from Y", where Y is some specific, detailed model with internal moving parts. "Arises from" is another legitimate phrase that means exactly the same thing: Gravity arises from the curvature of spacetime, according to the specific mathematical model of General Relativity. Chemistry arises from interactions between atoms, according to the specific model of quantum electrodynamics.

Now suppose I should say that gravity is explained by "arisence" or that chemistry is an "arising phenomenon", and claim that as my explanation.

The phrase "emerges from" is acceptable, just like "arises from" or "is caused by" are acceptable, if the phrase precedes some specific model to be judged on its own merits.

However, this is not the way "emergence" is commonly used. "Emergence" is commonly used as an explanation in its own right.

I have lost track of how many times I have heard people say, "Intelligence is an emergent phenomenon!" as if that explained intelligence. This usage fits all the checklist items for a mysterious answer to a mysterious question. What do you know, after you have said that intelligence is "emergent"? You can make no new predictions. You do not know anything about the behavior of real-world minds that you did not know before. It feels like you believe a new fact, but you don't anticipate any different outcomes. Your curiosity feels sated, but it has not been fed. The hypothesis has no moving parts - there's no detailed internal model to manipulate. Those who proffer the hypothesis of "emergence" confess their ignorance of the internals, and take pride in it; they contrast the science of "emergence" to other sciences merely mundane.

And even after the answer of "Why? Emergence!" is given, the phenomenon is still a mystery and possesses the same sacred impenetrability it had at the start.

A fun exercise is to eliminate the adjective "emergent" from any sentence in which it appears, and see if the sentence says anything different:

- Before: Human intelligence is an emergent product of neurons firing.

- After: Human intelligence is a product of neurons firing.

- Before: The behavior of the ant colony is the emergent outcome of the interactions of many individual ants.

- After: The behavior of the ant colony is the outcome of the interactions of many individual ants.

- Even better: A colony is made of ants. We can successfully predict some aspects of colony behavior using models that include only individual ants, without any global colony variables, showing that we understand how those colony behaviors arise from ant behaviors.

Another fun exercise is to replace the word "emergent" with the old word, the explanation that people had to use before emergence was invented:

- Before: Life is an emergent phenomenon.

- After: Life is a magical phenomenon.

- Before: Human intelligence is an emergent product of neurons firing.

- After: Human intelligence is a magical product of neurons firing.

Does not each statement convey exactly the same amount of knowledge about the phenomenon's behavior? Does not each hypothesis fit exactly the same set of outcomes?

"Emergence" has become very popular, just as saying "magic" used to be very popular. "Emergence" has the same deep appeal to human psychology, for the same reason. "Emergence" is such a wonderfully easy explanation, and it feels good to say it; it gives you a sacred mystery to worship. Emergence is popular because it is the junk food of curiosity. You can explain anything using emergence, and so people do just that; for it feels so wonderful to explain things. Humans are still humans, even if they've taken a few science classes in college. Once they find a way to escape the shackles of settled science, they get up to the same shenanigans as their ancestors, dressed up in the literary genre of "science" but still the same species psychology.

Sequence: Mysterious Answers to Mysterious Questions

Mysterious Answers to Mysterious Questions

Say Not "Complexity"

Referenced by: Positive Bias: Look Into the Dark • Say Not "Complexity" • Double Illusion of Transparency • Fake Optimization Criteria • Do Scientists Already Know This Stuff? • A Premature Word on AI • Ghosts in the Machine • Surface Analogies and Deep Causes • The Gift We Give To Tomorrow • Setting Up Metaethics • Dreams of AI Design • Worse Than Random • Selling Nonapples • Raising the Sanity Waterline • ESR's comments on some EY:OB/LW posts

Original with comments: The Futility of Emergence

Positive Bias: Look Into the Dark

Eliezer Yudkowsky, 28 August 2007 03:55AM

I am teaching a class, and I write upon the blackboard three numbers: 2-4-6. "I am thinking of a rule," I say, "which governs sequences of three numbers. The sequence 2-4-6, as it so happens, obeys this rule. Each of you will find, on your desk, a pile of index cards. Write down a sequence of three numbers on a card, and I'll mark it "Yes" for fits the rule, or "No" for not fitting the rule. Then you can write down another set of three numbers and ask whether it fits again, and so on. When you're confident that you know the rule, write down the rule on a card. You can test as many triplets as you like."

Here's the record of one student's guesses:

4, 6, 2             No
4, 6, 8             Yes
10, 12, 14       Yes

At this point the student wrote down his guess at the rule. What do you think the rule is? Would you have wanted to test another triplet, and if so, what would it be? Take a moment to think before continuing.

The challenge above is based on a classic experiment due to Peter Wason, the 2-4-6 task. Although subjects given this task typically expressed high confidence in their guesses, only 21% of the subjects successfully guessed the experimenter's real rule, and replications since then have continued to show success rates of around 20%.

The study was called "On the failure to eliminate hypotheses in a conceptual task" (Quarterly Journal of Experimental Psychology, 12: 129-140, 1960). Subjects who attempt the 2-4-6 task usually try to generate positive examples, rather than negative examples - they apply the hypothetical rule to generate a representative instance, and see if it is labeled "Yes".

Thus, someone who forms the hypothesis "numbers increasing by two" will test the triplet 8-10-12, hear that it fits, and confidently announce the rule. Someone who forms the hypothesis X-2X-3X will test the triplet 3-6-9, discover that it fits, and then announce that rule.

In every case the actual rule is the same: the three numbers must be in ascending order.

But to discover this, you would have to generate triplets that shouldn't fit, such as 20-23-26, and see if they are labeled "No". Which people tend not to do, in this experiment. In some cases, subjects devise, "test", and announce rules far more complicated than the actual answer.

This cognitive phenomenon is usually lumped in with "confirmation bias". However, it seems to me that the phenomenon of trying to test positive rather than negative examples, ought to be distinguished from the phenomenon of trying to preserve the belief you started with. "Positive bias" is sometimes used as a synonym for "confirmation bias", and fits this particular flaw much better.

It once seemed that phlogiston theory could explain a flame going out in an enclosed box (the air became saturated with phlogiston and no more could be released), but phlogiston theory could just as well have explained the flame not going out. To notice this, you have to search for negative examples instead of positive examples, look into zero instead of one; which goes against the grain of what experiment has shown to be human instinct.

For by instinct, we human beings only live in half the world.

One may be lectured on positive bias for days, and yet overlook it in-the-moment. Positive bias is not something we do as a matter of logic, or even as a matter of emotional attachment. The 2-4-6 task is "cold", logical, not affectively "hot". And yet the mistake is sub-verbal, on the level of imagery, of instinctive reactions. Because the problem doesn't arise from following a deliberate rule that says "Only think about positive examples", it can't be solved just by knowing verbally that "We ought to think about both positive and negative examples." Which example automatically pops into your head? You have to learn, wordlessly, to zag instead of zig. You have to learn to flinch toward the zero, instead of away from it.

I have been writing for quite some time now on the notion that the strength of a hypothesis is what it can't explain, not what it can - if you are equally good at explaining any outcome, you have zero knowledge. So to spot an explanation that isn't helpful, it's not enough to think of what it does explain very well - you also have to search for results it couldn't explain, and this is the true strength of the theory.

So I said all this, and then yesterday, I challenged the usefulness of "emergence" as a concept. One commenter cited superconductivity and ferromagnetism as examples of emergence. I replied that non-superconductivity and non-ferromagnetism were also examples of emergence, which was the problem. But be it far from me to criticize the commenter! Despite having read extensively on "confirmation bias", I didn't spot the "gotcha" in the 2-4-6 task the first time I read about it. It's a subverbal blink-reaction that has to be retrained. I'm still working on it myself.

So much of a rationalist's skill is below the level of words. It makes for challenging work in trying to convey the Art through blog posts. People will agree with you, but then, in the next sentence, do something subdeliberative that goes in the opposite direction. Not that I'm complaining! A major reason I'm posting here is to observe what my words haven't conveyed.

Are you searching for positive examples of positive bias right now, or sparing a fraction of your search on what positive bias should lead you to not see? Did you look toward light or darkness?

Sequence: Mysterious Answers to Mysterious Questions

Say Not "Complexity"

My Wild and Reckless Youth

Sequence: Noticing Confusion

Hindsight Devalues Science

Referenced by: Occam's Razor • We Change Our Minds Less Often Than We Think • Priming and Contamination • The Tragedy of Group Selectionism • Fake Optimization Criteria • Extensions and Intensions • Contaminated by Optimism • Crisis of Faith • Growing Up is Hard • That Crisis thing seems pretty useful • Privileging the Hypothesis

Original with comments: Positive Bias: Look Into the Dark

Say Not "Complexity"

Eliezer Yudkowsky, 29 August 2007 04:22AM

Once upon a time...

This is a story from when I first met Marcello, with whom I would later work for a year on AI theory; but at this point I had not yet accepted him as my apprentice. I knew that he competed at the national level in mathematical and computing olympiads, which sufficed to attract my attention for a closer look; but I didn't know yet if he could learn to think about AI.

I had asked Marcello to say how he thought an AI might discover how to solve a Rubik's Cube. Not in a preprogrammed way, which is trivial, but rather how the AI itself might figure out the laws of the Rubik universe and reason out how to exploit them. How would an AI invent for itself the concept of an "operator", or "macro", which is the key to solving the Rubik's Cube?

At some point in this discussion, Marcello said: "Well, I think the AI needs complexity to do X, and complexity to do Y -"

And I said, "Don't say 'complexity'."

Marcello said, "Why not?"

I said, "Complexity should never be a goal in itself. You may need to use a particular algorithm that adds some amount of complexity, but complexity for the sake of complexity just makes things harder." (I was thinking of all the people whom I had heard advocating that the Internet would "wake up" and become an AI when it became "sufficiently complex".)

And Marcello said, "But there's got to be some amount of complexity that does it."

I closed my eyes briefly, and tried to think of how to explain it all in words. To me, saying 'complexity' simply felt like the wrong move in the AI dance. No one can think fast enough to deliberate, in words, about each sentence of their stream of consciousness; for that would require an infinite recursion. We think in words, but our stream of consciousness is steered below the level of words, by the trained-in remnants of past insights and harsh experience...

I said, "Did you read A Technical Explanation of Technical Explanation?"

"Yes," said Marcello.

"Okay," I said, "saying 'complexity' doesn't concentrate your probability mass."

"Oh," Marcello said, "like 'emergence'. Huh. So... now I've got to think about how X might actually happen..."

That was when I thought to myself, "Maybe this one is teachable."

Complexity is not a useless concept. It has mathematical definitions attached to it, such as Kolmogorov complexity, and Vapnik-Chervonenkis complexity. Even on an intuitive level, complexity is often worth thinking about - you have to judge the complexity of a hypothesis and decide if it's "too complicated" given the supporting evidence, or look at a design and try to make it simpler.

But concepts are not useful or useless of themselves. Only usages are correct or incorrect. In the step Marcello was trying to take in the dance, he was trying to explain something for free, get something for nothing. It is an extremely common misstep, at least in my field. You can join a discussion on Artificial General Intelligence and watch people doing the same thing, left and right, over and over again - constantly skipping over things they don't understand, without realizing that's what they're doing.

In an eyeblink it happens: putting a non-controlling causal node behind something mysterious, a causal node that feels like an explanation but isn't. The mistake takes place below the level of words. It requires no special character flaw; it is how human beings think by default, since the ancient times.

What you must avoid is skipping over the mysterious part; you must linger at the mystery to confront it directly. There are many words that can skip over mysteries, and some of them would be legitimate in other contexts - "complexity", for example. But the essential mistake is that skip-over, regardless of what causal node goes behind it. The skip-over is not a thought, but a microthought. You have to pay close attention to catch yourself at it. And when you train yourself to avoid skipping, it will become a matter of instinct, not verbal reasoning. You have to feel which parts of your map are still blank, and more importantly, pay attention to that feeling.

I suspect that in academia there is a huge pressure to sweep problems under the rug so that you can present a paper with the appearance of completeness. You'll get more kudos for a seemingly complete model that includes some "emergent phenomena", versus an explicitly incomplete map where the label says "I got no clue how this part works" or "then a miracle occurs". A journal may not even accept the latter paper, since who knows but that the unknown steps are really where everything interesting happens? And yes, it sometimes happens that all the non-magical parts of your map turn out to also be non-important. That's the price you sometimes pay, for entering into terra incognita and trying to solve problems incrementally. But that makes it even more important to know when you aren't finished yet. Mostly, people don't dare to enter terra incognita at all, for the deadly fear of wasting their time.

And if you're working on a revolutionary AI startup, there is an even huger pressure to sweep problems under the rug; or you will have to admit to yourself that you don't know how to build an AI yet, and your current life-plans will come crashing down in ruins around your ears. But perhaps I am over-explaining, since skip-over happens by default in humans; if you're looking for examples, just watch people discussing religion or philosophy or spirituality or any science in which they were not professionally trained.

Marcello and I developed a convention in our AI work: when we ran into something we didn't understand, which was often, we would say "magic" - as in, "X magically does Y" - to remind ourselves that here was an unsolved problem, a gap in our understanding. It is far better to say "magic", than "complexity" or "emergence"; the latter words create an illusion of understanding. Wiser to say "magic", and leave yourself a placeholder, a reminder of work you will have to do later.

Sequence: Mysterious Answers to Mysterious Questions

The Futility of Emergence

Positive Bias: Look Into the Dark

Referenced by: "Science" as Curiosity-Stopper • Fake Optimization Criteria • Artificial Addition • Fake Utility Functions • Do Scientists Already Know This Stuff? • A Premature Word on AI • Ghosts in the Machine • The Comedy of Behaviorism • Dreams of AI Design • Complexity and Intelligence

Original with comments: Say Not "Complexity"

My Wild and Reckless Youth

Eliezer Yudkowsky, 30 August 2007 01:52AM

It is said that parents do all the things they tell their children not to do, which is how they know not to do them.

Long ago, in the unthinkably distant past, I was a devoted Traditional Rationalist, conceiving myself skilled according to that kind, yet I knew not the Way of Bayes. When the young Eliezer was confronted with a mysterious-seeming question, the precepts of Traditional Rationality did not stop him from devising a Mysterious Answer. It is, by far, the most embarrassing mistake I made in my life, and I still wince to think of it.

What was my mysterious answer to a mysterious question? This I will not describe, for it would be a long tale and complicated. I was young, and a mere Traditional Rationalist who knew not the teachings of Tversky and Kahneman. I knew about Occam's Razor, but not the conjunction fallacy. I thought I could get away with thinking complicated thoughts myself, in the literary style of the complicated thoughts I read in science books, not realizing that correct complexity is only possible when every step is pinned down overwhelmingly. Today, one of the chief pieces of advice I give to aspiring young rationalists is "Do not attempt long chains of reasoning or complicated plans."

Nothing more than this need be said: Even after I invented my "answer", the phenomenon was still a mystery unto me, and possessed the same quality of wondrous impenetrability that it had at the start.

Make no mistake, that younger Eliezer was not stupid. All the errors of which the young Eliezer was guilty, are still being made today by respected scientists in respected journals. It would have taken a subtler skill to protect him, than ever he was taught as a Traditional Rationalist.

Indeed, the young Eliezer diligently and painstakingly followed the injunctions of Traditional Rationality in the course of going astray.

As a Traditional Rationalist, the young Eliezer was careful to ensure that his Mysterious Answer made a bold prediction of future experience. Namely, I expected future neurologists to discover that neurons were exploiting quantum gravity, a la Sir Roger Penrose. This required neurons to maintain a certain degree of quantum coherence, which was something you could look for, and find or not find. Either you observe that or you don't, right?

But my hypothesis made no retrospective predictions. According to Traditional Science, retrospective predictions don't count - so why bother making them? To a Bayesian, on the other hand, if a hypothesis does not today have a favorable likelihood ratio over "I don't know", it raises the question of why you today believe anything more complicated than "I don't know". But I knew not the Way of Bayes, so I was not thinking about likelihood ratios or focusing probability density. I had Made a Falsifiable Prediction; was this not the Law?

As a Traditional Rationalist, the young Eliezer was careful not to believe in magic, mysticism, carbon chauvinism, or anything of that sort. I proudly professed of my Mysterious Answer, "It is just physics like all the rest of physics!" As if you could save magic from being a cognitive isomorph of magic, by calling it quantum gravity. But I knew not the Way of Bayes, and did not see the level on which my idea was isomorphic to magic. I gave my allegiance to physics, but this did not save me; what does probability theory know of allegiances? I avoided everything that Traditional Rationality told me was forbidden, but what was left was still magic.

Beyond a doubt, my allegiance to Traditional Rationality helped me get out of the hole I dug myself into. If I hadn't been a Traditional Rationalist, I would have been completely screwed. But Traditional Rationality still wasn't enough to get it right. It just led me into different mistakes than the ones it had explicitly forbidden.

When I think about how my younger self very carefully followed the rules of Traditional Rationality in the course of getting the answer wrong, it sheds light on the question of why people who call themselves "rationalists" do not rule the world. You need one whole hell of a lot of rationality before it does anything but lead you into new and interesting mistakes.

Traditional Rationality is taught as an art, rather than a science; you read the biography of famous physicists describing the lessons life taught them, and you try to do what they tell you to do. But you haven't lived their lives, and half of what they're trying to describe is an instinct that has been trained into them.

The way Traditional Rationality is designed, it would have been acceptable for me to spend 30 years on my silly idea, so long as I succeeded in falsifying it eventually, and was honest with myself about what my theory predicted, and accepted the disproof when it arrived, et cetera. This is enough to let the Ratchet of Science click forward, but it's a little harsh on the people who waste 30 years of their lives. Traditional Rationality is a walk, not a dance. It's designed to get you to the truth eventually, and gives you all too much time to smell the flowers along the way.

Traditional Rationalists can agree to disagree. Traditional Rationality doesn't have the ideal that thinking is an exact art in which there is only one correct probability estimate given the evidence. In Traditional Rationality, you're allowed to guess, and then test your guess. But experience has taught me that if you don't know, and you guess, you'll end up being wrong.

The Way of Bayes is also an imprecise art, at least the way I'm holding forth upon it. These blog posts are still fumbling attempts to put into words lessons that would be better taught by experience. But at least there's underlying math, plus experimental evidence from cognitive psychology on how humans actually think. Maybe that will be enough to cross the stratospherically high threshold required for a discipline that lets you actually get it right, instead of just constraining you into interesting new mistakes.

Sequence: Mysterious Answers to Mysterious Questions

Positive Bias: Look Into the Dark

Failing to Learn from History

Referenced by: Failing to Learn from History • Zombies! Zombies? • The Born Probabilities • When Science Can't Help • Science Isn't Strict Enough • A Premature Word on AI • Against Devil's Advocacy • No Universally Compelling Arguments • Rebelling Within Nature • Could Anything Be Right? • The Meaning of Right • Contaminated by Optimism • My Childhood Death Spiral • My Best and Worst Mistake • The Sheer Folly of Callow Youth

Original with comments: My Wild and Reckless Youth

Failing to Learn from History

Eliezer Yudkowsky, 30 August 2007 08:22PM

Continuation of: My Wild and Reckless Youth

Once upon a time, in my wild and reckless youth, when I knew not the Way of Bayes, I gave a Mysterious Answer to a mysterious-seeming question. Many failures occurred in sequence, but one mistake stands out as most critical: My younger self did not realize that solving a mystery should make it feel less confusing. I was trying to explain a Mysterious Phenomenon - which to me meant providing a cause for it, fitting it into an integrated model of reality. Why should this make the phenomenon less Mysterious, when that is its nature? I was trying to explain the Mysterious Phenomenon, not render it (by some impossible alchemy) into a mundane phenomenon, a phenomenon that wouldn't even call out for an unusual explanation in the first place.

As a Traditional Rationalist, I knew the historical tales of astrologers and astronomy, of alchemists and chemistry, of vitalists and biology. But the Mysterious Phenomenon was not like this. It was something new, something stranger, something more difficult, something that ordinary science had failed to explain for centuries -

- as if stars and matter and life had not been mysteries for hundreds of years and thousands of years, from the dawn of human thought right up until science finally solved them -

We learn about astronomy and chemistry and biology in school, and it seems to us that these matters have always been the proper realm of science, that they have never been mysterious. When science dares to challenge a new Great Puzzle, the children of that generation are skeptical, for they have never seen science explain something that feels mysterious to them. Science is only good for explaining scientific subjects, like stars and matter and life.

I thought the lesson of history was that astrologers and alchemists and vitalists had an innate character flaw, a tendency toward mysterianism, which led them to come up with mysterious explanations for non-mysterious subjects. But surely, if a phenomenon really was very weird, a weird explanation might be in order?

It was only afterward, when I began to see the mundane structure inside the mystery, that I realized whose shoes I was standing in. Only then did I realize how reasonable vitalism had seemed at the time, how surprising and embarrassing had been the universe's reply of, "Life is mundane, and does not need a weird explanation."

We read history but we don't live it, we don't experience it. If only I had personally postulated astrological mysteries and then discovered Newtonian mechanics, postulated alchemical mysteries and then discovered chemistry, postulated vitalistic mysteries and then discovered biology. I would have thought of my Mysterious Answer and said to myself: No way am I falling for that again.

Sequence: Mysterious Answers to Mysterious Questions

My Wild and Reckless Youth

Making History Available

Referenced by: Making History Available • Why is the Future So Absurd? • Artificial Addition • Expecting Beauty • Angry Atoms • Heat vs. Motion • Living By Your Own Strength • Eutopia is Scary

Original with comments: Failing to Learn from History

Making History Available

Eliezer Yudkowsky, 31 August 2007 07:52PM

Followup to: Failing to Learn from History

There is a habit of thought which I call the logical fallacy of generalization from fictional evidence, which deserves a blog post in its own right, one of these days. Journalists who, for example, talk about the Terminator movies in a report on AI, do not usually treat Terminator as a prophecy or fixed truth. But the movie is recalled - is available - as if it were an illustrative historical case. As if the journalist had seen it happen on some other planet, so that it might well happen here. More on this in Section 6 of this paper.

There is an inverse error to generalizing from fictional evidence: failing to be sufficiently moved by historical evidence. The trouble with generalizing from fictional evidence is that it is fiction - it never actually happened. It's not drawn from the same distribution as this, our real universe; fiction differs from reality in systematic ways. But history has happened, and should be available.

In our ancestral environment, there were no movies; what you saw with your own eyes was true. Is it any wonder that fictions we see in lifelike moving pictures have too great an impact on us? Conversely, things that really happened, we encounter as ink on paper; they happened, but we never saw them happen. We don't remember them happening to us.

The inverse error is to treat history as mere story, process it with the same part of your mind that handles the novels you read. You may say with your lips that it is "truth", rather than "fiction", but that doesn't mean you are being moved as much as you should be. Many biases involve being insufficiently moved by dry, abstract information.

Once upon a time, I gave a Mysterious Answer to a mysterious question, not realizing that I was making exactly the same mistake as astrologers devising mystical explanations for the stars, or alchemists devising magical properties of matter, or vitalists postulating an opaque "elan vital" to explain all of biology.

When I finally realized whose shoes I was standing in, there was a sudden shock of unexpected connection with the past. I realized that the invention and destruction of vitalism - which I had only read about in books - had actually happened to real people, who experienced it much the same way I experienced the invention and destruction of my own mysterious answer. And I also realized that if I had actually experienced the past - if I had lived through past scientific revolutions myself, rather than reading about them in history books - I probably would not have made the same mistake again. I would not have come up with another mysterious answer; the first thousand lessons would have hammered home the moral.

So (I thought), to feel sufficiently the force of history, I should try to approximate the thoughts of an Eliezer who had lived through history - I should try to think as if everything I read about in history books, had actually happened to me. (With appropriate reweighting for the availability bias of history books - I should remember being a thousand peasants for every ruler.) I should immerse myself in history, imagine living through eras I only saw as ink on paper.

Why should I remember the Wright Brothers' first flight? I was not there. But as a rationalist, could I dare to not remember, when the event actually happened? Is there so much difference between seeing an event through your eyes - which is actually a causal chain involving reflected photons, not a direct connection - and seeing an event through a history book? Photons and history books both descend by causal chains from the event itself.

I had to overcome the false amnesia of being born at a particular time. I had to recall - make available - all the memories, not just the memories which, by mere coincidence, belonged to myself and my own era.

The Earth became older, of a sudden.

To my former memory, the United States had always existed - there was never a time when there was no United States. I had not remembered, until that time, how the Roman Empire rose, and brought peace and order, and lasted through so many centuries, until I forgot that things had ever been otherwise; and yet the Empire fell, and barbarians overran my city, and the learning that I had possessed was lost. The modern world became more fragile to my eyes; it was not the first modern world.

So many mistakes, made over and over and over again, because I did not remember making them, in every era I never lived...

And to think, people sometimes wonder if overcoming bias is important.

Don't you remember how many times your biases have killed you? You don't? I've noticed that sudden amnesia often follows a fatal mistake. But take it from me, it happened. I remember; I wasn't there.

So the next time you doubt the strangeness of the future, remember how you were born in a hunter-gatherer tribe ten thousand years ago, when no one knew of Science at all. Remember how you were shocked, to the depths of your being, when Science explained the great and terrible sacred mysteries that you once revered so highly. Remember how you once believed that you could fly by eating the right mushrooms, and then you accepted with disappointment that you would never fly, and then you flew. Remember how you had always thought that slavery was right and proper, and then you changed your mind. Don't imagine how you could have predicted the change, for that is amnesia. Remember that, in fact, you did not guess. Remember how, century after century, the world changed in ways you did not guess.

Maybe then you will be less shocked by what happens next.

Sequence: Mysterious Answers to Mysterious Questions

Failing to Learn from History

Explain/Worship/Ignore?

Referenced by: Stranger Than History • Human Evil and Muddled Thinking

Original with comments: Making History Available

Stranger Than History

Eliezer Yudkowsky, 01 September 2007 06:57PM

Suppose I told you that I knew for a fact that the following statements were true:

- If you paint yourself a certain exact color between blue and green, it will reverse the force of gravity on you and cause you to fall upward.

- In the future, the sky will be filled by billions of floating black spheres. Each sphere will be larger than all the zeppelins that have ever existed put together. If you offer a sphere money, it will lower a male prostitute out of the sky on a bungee cord.

- Your grandchildren will think it is not just foolish, but evil, to put thieves in jail instead of spanking them.

You'd think I was crazy, right?

Now suppose it were the year 1901, and you had to choose between believing those statements I have just offered, and believing statements like the following:

- There is an absolute speed limit on how fast two objects can seem to be traveling relative to each other, which is exactly 670616629.2 miles per hour. If you hop on board a train going almost this fast and fire a gun out the window, the fundamental units of length change around, so it looks to you like the bullet is speeding ahead of you, but other people see something different. Oh, and time changes around too.

- In the future, there will be a superconnected global network of billions of adding machines, each one of which has more power than all pre-1901 adding machines put together. One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.

- Your grandchildren will think it is not just foolish, but evil, to say that someone should not be President of the United States because she is black.

Based on a comment of Robin's: "I wonder if one could describe in enough detail a fictional story of an alternative reality, a reality that our ancestors could not distinguish from the truth, in order to make it very clear how surprising the truth turned out to be."

Referenced by: Absurdity Heuristic, Absurdity Bias • Why is the Future So Absurd? • The Logical Fallacy of Generalization from Fictional Evidence • Timeless Physics • Engelbart: Insufficiently Recursive

Original with comments: Stranger Than History

Explain/Worship/Ignore?

Eliezer Yudkowsky, 02 September 2007 08:01PM

Followup to: Semantic Stopsigns, Mysterious Answers to Mysterious Questions

As our tribe wanders through the grasslands, searching for fruit trees and prey, it happens every now and then that water pours down from the sky.

"Why does water sometimes fall from the sky?" I ask the bearded wise man of our tribe.

He thinks for a moment, this question having never occurred to him before, and then says, "From time to time, the sky spirits battle, and when they do, their blood drips from the sky."

"Where do the sky spirits come from?" I ask.

His voice drops to a whisper. "From the before time. From the long long ago."

When it rains, and you don't know why, you have several options. First, you could simply not ask why - not follow up on the question, or never think of the question in the first place. This is the Ignore command, which the bearded wise man originally selected. Second, you could try to devise some sort of explanation, the Explain command, as the bearded man did in response to your first question. Third, you could enjoy the sensation of mysteriousness - the Worship command.

Now, as you are bound to notice from this story, each time you select Explain, the best-case scenario is that you get an explanation, such as "sky spirits". But then this explanation itself is subject to the same dilemma - Explain, Worship, or Ignore? Each time you hit Explain, science grinds for a while, returns an explanation, and then another dialog box pops up. As good rationalists, we feel duty-bound to keep hitting Explain, but it seems like a road that has no end.

You hit Explain for life, and get chemistry; you hit Explain for chemistry, and get atoms; you hit Explain for atoms, and get electrons and nuclei; you hit Explain for nuclei, and get quantum chromodynamics and quarks; you hit Explain for how the quarks got there, and get back the Big Bang...

We can hit Explain for the Big Bang, and wait while science grinds through its process, and maybe someday it will return a perfectly good explanation. But then that will just bring up another dialog box. So, if we continue long enough, we must come to a special dialog box, a new option, an Explanation That Needs No Explanation, a place where the chain ends - and this, maybe, is the only explanation worth knowing.

There - I just hit Worship.

Never forget that there are many more ways to worship something than lighting candles around an altar.

If I'd said, "Huh, that does seem paradoxical. I wonder how the apparent paradox is resolved?" then I would have hit Explain, which does sometimes take a while to produce an answer.

And if the whole issue seems to you unimportant, or irrelevant, or if you'd rather put off thinking about it until tomorrow, than you have hit Ignore.

Select your option wisely.

Sequence: Mysterious Answers to Mysterious Questions

Making History Available

"Science" as Curiosity-Stopper

Referenced by: A Priori • No One Knows What Science Doesn't Know • Joy in the Merely Real • Quantum Non-Realism • Einstein's Superpowers • Where Recursive Justification Hits Bottom • The Meaning of Right • Crisis of Faith • That Crisis thing seems pretty useful

Original with comments: Explain/Worship/Ignore?

"Science" as Curiosity-Stopper

Eliezer Yudkowsky, 03 September 2007 08:04PM

Followup to: Semantic Stopsigns, Mysterious Answers to Mysterious Questions, Say Not 'Complexity'

Imagine that I, in full view of live television cameras, raised my hands and chanted abracadabra and caused a brilliant light to be born, flaring in empty space beyond my outstretched hands. Imagine that I committed this act of blatant, unmistakeable sorcery under the full supervision of James Randi and all skeptical armies. Most people, I think, would be fairly curious as to what was going on.

But now suppose instead that I don't go on television. I do not wish to share the power, nor the truth behind it. I want to keep my sorcery secret. And yet I also want to cast my spells whenever and wherever I please. I want to cast my brilliant flare of light so that I can read a book on the train - without anyone becoming curious. Is there a spell that stops curiosity?

Yes indeed! Whenever anyone asks "How did you do that?", I just say "Science!"

It's not a real explanation, so much as a curiosity-stopper. It doesn't tell you whether the light will brighten or fade, change color in hue or saturation, and it certainly doesn't tell you how to make a similar light yourself. You don't actually know anything more than you knew before I said the magic word. But you turn away, satisfied that nothing unusual is going on.

Better yet, the same trick works with a standard light switch.

Flip a switch and a light bulb turns on. Why?

In school, one is taught that the password to the light bulb is "Electricity!" By now, I hope, you're wary of marking the light bulb "understood" on such a basis. Does saying "Electricity!" let you do calculations that will control your anticipation of experience? There is, at the least, a great deal more to learn. (Physicists should ignore this paragraph and substitute a problem in evolutionary theory, where the substance of the theory is again in calculations that few people know how to perform.)

If you thought the light bulb was scientifically inexplicable, it would seize the entirety of your attention. You would drop whatever else you were doing, and focus on that light bulb.

But what does the phrase "scientifically explicable" mean? It means that someone else knows how the light bulb works. When you are told the light bulb is "scientifically explicable", you don't know more than you knew earlier; you don't know whether the light bulb will brighten or fade. But because someone else knows, it devalues the knowledge in your eyes. You become less curious.

Since this is an econblog, someone out there is bound to say, "If the light bulb were unknown to science, you could gain fame and fortune by investigating it." But I'm not talking about greed. I'm not talking about career ambition. I'm talking about the raw emotion of curiosity - the feeling of being intrigued. Why should your curiosity be diminished because someone else, not you, knows how the light bulb works? Is this not spite? It's not enough for you to know; other people must also be ignorant, or you won't be happy?

There are goods that knowledge may serve besides curiosity, such as the social utility of technology. For these instrumental goods, it matters whether some other entity in local space already knows. But for my own curiosity, why should it matter?

Besides, consider the consequences if you permit "Someone else knows the answer" to function as a curiosity-stopper. One day you walk into your living room and see a giant green elephant, seemingly hovering in midair, surrounded by an aura of silver light.

"What the heck?" you say.

And a voice comes from above the elephant, saying, "SOMEONE ELSE ALREADY KNOWS WHY THIS ELEPHANT IS HERE."

"Oh," you say, "in that case, never mind," and walk on to the kitchen.

I don't know the grand unified theory for this universe's laws of physics. I also don't know much about human anatomy with the exception of the brain. I couldn't point out on my body where my kidneys are, and I can't recall offhand what my liver does. (I am not proud of this. Alas, with all the math I need to study, I'm not likely to learn anatomy anytime soon.)

Should I, so far as curiosity is concerned, be more intrigued by my ignorance of the ultimate laws of physics, than the fact that I don't know much about what goes on inside my own body?

If I raised my hands and cast a light spell, you would be intrigued. Should you be any less intrigued by the very fact that I raised my hands? When you raise your arm and wave a hand around, this act of will is coordinated by (among other brain areas) your cerebellum. I bet you don't know how the cerebellum works. I know a little - though only the gross details, not enough to perform calculations... but so what? What does that matter, if you don't know? Why should there be a double standard of curiosity for sorcery and hand motions?

Look at yourself in the mirror. Do you know what you're looking at? Do you know what looks out from behind your eyes? Do you know what you are? Some of that answer, Science knows, and some of it Science does not. But why should that distinction matter to your curiosity, if you don't know?

Do you know how your knees work? Do you know how your shoes were made? Do you know why your computer monitor glows? Do you know why water is wet?

The world around you is full of puzzles. Prioritize, if you must. But do not complain that cruel Science has emptied the world of mystery. With reasoning such as that, I could get you to overlook an elephant in your living room.

Sequence: Mysterious Answers to Mysterious Questions

Explain/Worship/Ignore?

Applause Lights

Referenced by: The Lens That Sees Its Flaws • No One Knows What Science Doesn't Know • Feel the Meaning • Fake Reductionism • Joy in the Merely Real • Many Worlds, One Best Guess • Grasping Slippery Things

Original with comments: "Science" as Curiosity-Stopper

Absurdity Heuristic, Absurdity Bias

Eliezer Yudkowsky, 05 September 2007 03:20AM

Followup to: Stranger Than History, Robin's post What Evidence Ease of Imagination?

I've been pondering lately the notion of "absurdity" - wondering what exactly goes on in people's minds when they utter the adjective "absurd" or the objection "Absurd!"

If there is an absurdity heuristic, it would seem, at first glance, to be the mirror image of the well-known representativeness heuristic. The less X resembles Y, or the more X violates typicality assumptions of Y, the less probable that X is the product, explanation, or outcome of Y. A sequence of events is less probable when it involves an egg unscrambling itself, water flowing upward, machines thinking or dead people coming back to life. Since human psychology is not a pure structure of quantitative probabilities, it is easy to imagine that the absurdity heuristic is separate from the representativeness heuristic - implemented by separate absurdity-detecting brainware.

I suspect people may also be more sensitive to "absurdity" that invalidates a plan or indicates cheating. Consider the difference between "I saw a little blue man yesterday, walking down the street" versus "I'm going to jump off this cliff and a little blue man will catch me on the way down" or "If you give me your wallet, a little blue man will bring you a pot of gold." (I'm thinking, in particular, about how projections of future technology are often met by the objection, "That's absurd!", and how the objection seems more violent than usual in this case.)

As Robin observed, a heuristic is not necessarily a bias. The vast majority of objects do not fall upward. And yet helium balloons are an exception. When are exceptions predictable?

I can think of three major circumstances where the absurdity heuristic gives rise to an absurdity bias:

The first case is when we have information about underlying laws which should override surface reasoning. If you know why most objects fall, and you can calculate how fast they fall, then your calculation that a helium balloon should rise at such-and-such a rate, ought to strictly override the absurdity of an object falling upward. If you can do deep calculations, you have no need for qualitative surface reasoning. But we may find it hard to attend to mere calculations in the face of surface absurdity, until we see the balloon rise.

(In 1913, Lee de Forest was accused of fraud for selling stock in an impossible endeavor, the Radio Telephone Company: "De Forest has said in many newspapers and over his signature that it would be possible to transmit human voice across the Atlantic before many years. Based on these absurd and deliberately misleading statements, the misguided public...has been persuaded to purchase stock in his company...")

The second case is a generalization of the first - attending to surface absurdity in the face of abstract information that ought to override it. If people cannot accept that studies show that marginal spending on medicine has zero net effect, because it seems absurd - violating the surface rule that "medicine cures" - then I would call this "absurdity bias". There are many reasons that people may fail to attend to abstract information or integrate it incorrectly. I think it worth distinguishing cases where the failure arises from absurdity detectors going off.

The third case is when the absurdity heuristic simply doesn't work - the process is not stable in its surface properties over the range of extrapolation - and yet people use it anyway. The future is usually "absurd" - it is unstable in its surface rules over fifty-year intervals.

This doesn't mean that anything can happen. Of all the events in the 20th century that would have been "absurd" by the standards of the 19th century, not a single one - to the best of our knowledge - violated the law of conservation of energy, which was known in 1850. Reality is not up for grabs; it works by rules even more precise than the ones we believe in instinctively.

The point is not that you can say anything you like about the future and no one can contradict you; but, rather, that the particular practice of crying "Absurd!" has historically been an extremely poor heuristic for predicting the future. Over the last few centuries, the absurdity heuristic has done worse than maximum entropy - ruled out the actual outcomes as being far too absurd to be considered. You would have been better off saying "I don't know".

Referenced by: Availability • Why is the Future So Absurd? • My Strange Beliefs • Lawful Creativity

Original with comments: Absurdity Heuristic, Absurdity Bias

Availability

Eliezer Yudkowsky, 06 September 2007 06:55AM

The availability heuristic is judging the frequency or probability of an event, by the ease with which examples of the event come to mind.

A famous 1978 study by Lichtenstein, Slovic, Fischhoff, Layman, and Combs, "Judged Frequency of Lethal Events", studied errors in quantifying the severity of risks, or judging which of two dangers occurred more frequently. Subjects thought that accidents caused about as many deaths as disease; thought that homicide was a more frequent cause of death than suicide. Actually, diseases cause about 16 times as many deaths as accidents, and suicide is twice as frequent as homicide.

An obvious hypothesis to account for these skewed beliefs is that murders are more likely to be talked about than suicides - thus, someone is more likely to recall hearing about a murder than hearing about a suicide. Accidents are more dramatic than diseases - perhaps this makes people more likely to remember, or more likely to recall, an accident. In 1979, a followup study by Combs and Slovic showed that the skewed probability judgments correlated strongly (.85 and .89) with skewed reporting frequencies in two newspapers. This doesn't disentangle whether murders are more available to memory because they are more reported-on, or whether newspapers report more on murders because murders are more vivid (hence also more remembered). But either way, an availability bias is at work.

Selective reporting is one major source of availability biases. In the ancestral environment, much of what you knew, you experienced yourself; or you heard it directly from a fellow tribe-member who had seen it. There was usually at most one layer of selective reporting between you, and the event itself. With today's Internet, you may see reports that have passed through the hands of six bloggers on the way to you - six successive filters. Compared to our ancestors, we live in a larger world, in which far more happens, and far less of it reaches us - a much stronger selection effect, which can create much larger availability biases.

In real life, you're unlikely to ever meet Bill Gates. But thanks to selective reporting by the media, you may be tempted to compare your life success to his - and suffer hedonic penalties accordingly. The objective frequency of Bill Gates is 0.00000000015, but you hear about him much more often. Conversely, 19% of the planet lives on less than $1/day, and I doubt that one fifth of the blog posts you read are written by them.

Using availability seems to give rise to an absurdity bias; events that have never happened, are not recalled, and hence deemed to have probability zero. When no flooding has recently occurred (and yet the probabilities are still fairly calculable), people refuse to buy flood insurance even when it is heavily subsidized and priced far below an actuarially fair value. Kunreuther et. al. (1993) suggests underreaction to threats of flooding may arise from "the inability of individuals to conceptualize floods that have never occurred... Men on flood plains appear to be very much prisoners of their experience... Recently experienced floods appear to set an upward bound to the size of loss with which managers believe they ought to be concerned."

Burton et. al. (1978) report that when dams and levees are built, they reduce the frequency of floods, and thus apparently create a false sense of security, leading to reduced precautions. While building dams decreases the frequency of floods, damage per flood is afterward so much greater that average yearly damage increases.

The wise would extrapolate from a memory of small hazards to the possibility of large hazards. Instead, past experience of small hazards seems to set a perceived upper bound on risk. A society well-protected against minor hazards takes no action against major risks, building on flood plains once the regular minor floods are eliminated. A society subject to regular minor hazards treats those minor hazards as an upper bound on the size of the risks, guarding against regular minor floods but not occasional major floods.

Memory is not always a good guide to probabilities in the past, let alone the future.

Burton, I., Kates, R. and White, G. 1978. Environment as Hazard. New York: Oxford University Press.

Combs, B. and Slovic, P. 1979. Causes of death: Biased newspaper coverage and biased judgments. Journalism Quarterly, 56: 837-843.

Kunreuther, H., Hogarth, R. and Meszaros, J. 1993. Insurer ambiguity and market failure. Journal of Risk and Uncertainty, 7: 71-87.

Lichtenstein, S., Slovic, P., Fischhoff, B., Layman, M. and Combs, B. 1978. Judged Frequency of Lethal Events. Journal of Experimental Psychology: Human Learning and Memory, 4(6), November: 551-78.

Referenced by: Why is the Future So Absurd? • Doublethink (Choosing to be Biased) • Planning Fallacy • We Change Our Minds Less Often Than We Think • The Logical Fallacy of Generalization from Fictional Evidence • My Strange Beliefs • A Premature Word on AI • Abstracted Idealized Dynamics • Dunbar's Function

Original with comments: Availability

Why is the Future So Absurd?

Eliezer Yudkowsky, 07 September 2007 08:42AM

Followup to: Stranger than History, Absurdity Heuristic / Absurdity Bias

Why is the future more absurd than people seem to expect? (That is: Why, historically, has the future so often turned out to be more "absurd" than people seem to have expected?)

One obvious reason is hindsight bias. Hindsight does not just cause people to severely underestimate how much they would have been surprised. Hindsight also leads people to overestimate how much attention they would have paid to the key factors, the factors that turned out to be important. As R. H. Tawney put it:

"Historians give an appearance of inevitability to an existing order by dragging into prominence the forces which have triumphed and thrusting into the background those which they have swallowed up."

When people look at historical changes and think "I could have predicted X" or "You could have predicted X if you looked at factors 1, 2, and 3"; then they forget that people did not, in fact, predict X, perhaps because they were distracted by factors 4 through 117. People read history books, see coherent narratives, and think that's how Time works. Underestimating the surprise of the present, they overestimate the predictability of the future.

I suspect that a major factor contributing to absurdity bias is that, when we look over history, we see changes away from absurd conditions such as everyone being a peasant farmer and women not having the vote, toward normal conditions like a majority middle class and equal rights. When people look at history, they see a series of normalizations. They learn the rule, "The future grows ever less absurd over time."

Perhaps one way to comprehend the bizarreness of the future would be to try and imagine historical changes occurring in reverse - how absurd would it be if all your electrical appliances suddenly disappeared, or you were transformed into a peasant farmer? Even if the future is nicer than the past, it will feel at least that absurd.

The correspondence bias of social psychology may also play a role in how we fail to learn from history - or so my own experience suggests. When we read about the strange behaviors of people in other eras, we may see them as people with a disposition to that strange behavior, rather than properly comprehending the strangeness of the times. In the 16th century, one popular entertainment was setting a cat on fire. If you think to yourself "What horrible people they must be!" then you have, to the same extent, diminished your appreciation of what horrible times they lived in.

We see at least some social and technological changes during our own lifetime. We do have some experience of genuine future shock. Why wouldn't this be enough to extrapolate forward?

According to Ray Kurzweil's thesis of accelerating change, our intuitions about the future are linear - we expect around as much change as occurred in the past - but technological change feeds on itself, and therefore has a positive second derivative. We should expect more technological change in the future than we have seen in the past, and insofar as technology drives cultural change, we should expect more cultural change too.

Or that, in my opinion, is the strongest version of Kurzweil's theory that can be put forward. Kurzweil dwells on Moore's Law and smoothly predictable exponential curves, but this seems to me both iffy and unnecessary. A curve does not need to be smooth or exponential to have a positive second derivative. And our cultural sensitivity to, say, computing power, is probably logarithmic anyway, obeying Weber's Law - a 20% increase in computing power probably feels the same whether it's from 1MHz to 1.2MHz, or 2GHz to 2.4GHz. In which case, people extrapolating the future "linearly" should get it pretty much correct.

But if you pull back and view the last few millennia, not just the last few decades, the strength of the core idea becomes obvious - technology change does feed on itself and therefore does speed up.

I would actually question Kurzweil's assertion that people extrapolate the past linearly into the future. Kurzweil may be too optimistic here. As discussed earlier, dwellers on flood plains do not extrapolate from small floods to large floods; instead, small floods set a perceived upper bound on risk. I suspect that when people try to visualize the strangeness of the future, they focus on a single possible change, of no greater magnitude than the largest single change they remember in their own lifetime.

The real future is not composed of single developments, but many developments together. Even if one change can pass the futurism filter, to suppose three absurdities simultaneously - never mind twenty - would entirely overload the absurdity meter. This may also explain why future projections get wronger and wronger as they go further out. People seem to imagine futures that are minimally counterintuitive, with one or two interesting changes to make a good story, rather than a realistic number of changes that would overload their extrapolation abilities.

What other biases could lead us to underestimate the absurdity of the future?

Referenced by: The Logical Fallacy of Generalization from Fictional Evidence • Bind Yourself to Reality • Where Philosophy Meets Science • The First World Takeover • Surprised by Brains • Eutopia is Scary

Original with comments: Why is the Future So Absurd?

Anchoring and Adjustment

Eliezer Yudkowsky, 07 September 2007 09:33PM

Suppose I spin a Wheel of Fortune device as you watch, and it comes up pointing to 65. Then I ask: Do you think the percentage of African countries in the UN is above or below this number? What do you think is the percentage of African countries in the UN? Take a moment to consider these two questions yourself, if you like, and please don't Google.

Also, try to guess, within 5 seconds, the value of the following arithmetical expression. 5 seconds. Ready? Set... Go!

1 x 2 x 3 x 4 x 5 x 6 x 7 x 8

Tversky and Kahneman (1974) recorded the estimates of subjects who saw the Wheel of Fortune showing various numbers. The median estimate of subjects who saw the wheel show 65 was 45%; the median estimate of subjects who saw 10 was 25%.

The current theory for this and similar experiments is that subjects take the initial, uninformative number as their starting point or anchor; and then they adjust upward or downward from their starting estimate until they reached an answer that "sounded plausible"; and then they stopped adjusting. This typically results in under-adjustment from the anchor - more distant numbers could also be "plausible", but one stops at the first satisfying-sounding answer.

Similarly, students shown "1 x 2 x 3 x 4 x 5 x 6 x 7 x 8" made a median estimate of 512, while students shown "8 x 7 x 6 x 5 x 4 x 3 x 2 x 1" made a median estimate of 2,250. The motivating hypothesis was that students would try to multiply (or guess-combine) the first few factors of the product, then adjust upward. In both cases the adjustments were insufficient, relative to the true value of 40,320; but the first set of guesses were much more insufficient because they started from a lower anchor.

Tversky and Kahneman report that offering payoffs for accuracy did not reduce the anchoring effect.

Strack and Mussweiler (1997) asked for the year Einstein first visited the United States. Completely implausible anchors, such as 1215 or 1992, produced anchoring effects just as large as more plausible anchors such as 1905 or 1939.

There are obvious applications in, say, salary negotiations, or buying a car. I won't suggest that you exploit it, but watch out for exploiters.

And: Watch yourself thinking, and try to notice when you are adjusting a figure in search of an estimate.

Debiasing manipulations for anchoring have generally proved not very effective. I would suggest these two: First, if the initial guess sounds implausible, try to throw it away entirely and come up with a new estimate, rather than sliding from the anchor. But this in itself may not be sufficient - subjects instructed to avoid anchoring still seem to do so (Quattrone et. al. 1981). So second, even if you are trying the first method, try also to think of an anchor in the opposite direction - an anchor that is clearly too small or too large, instead of too large or too small - and dwell on it briefly.

Quattrone, G.A., Lawrence, C.P., Finkel, S.E., & Andrus, D.C. (1981). Explorations in anchoring: The effects of prior range, anchor extremity, and suggestive hints. Manuscript, Stanford University.

Strack, F. & Mussweiler, T. (1997). Explaining the enigmatic anchoring effect: Mechanisms of selective accessibility. Journal of Personality and Social Psychology, 73, 437-446.

Tversky, A. and Kahneman, D. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185:1124--1131.

Sequence: Seeing with Fresh Eyes

Priming and Contamination

Referenced by: Occam's Razor • We Change Our Minds Less Often Than We Think • Priming and Contamination • Do We Believe Everything We're Told? • The Logical Fallacy of Generalization from Fictional Evidence • Self-Anchoring • Engelbart: Insufficiently Recursive

Original with comments: Anchoring and Adjustment

The Crackpot Offer

Eliezer Yudkowsky, 08 September 2007 02:32PM

When I was very young - I think thirteen or maybe fourteen - I thought I had found a disproof of Cantor's Diagonal Argument, a famous theorem which demonstrates that the real numbers outnumber the rational numbers. Ah, the dreams of fame and glory that danced in my head!

My idea was that since each whole number can be decomposed into a bag of powers of 2, it was possible to map the whole numbers onto the set of subsets of whole numbers simply by writing out the binary expansion. 13, for example, 1101, would map onto {0, 2, 3}. It took a whole week before it occurred to me that perhaps I should apply Cantor's Diagonal Argument to my clever construction, and of course it found a counterexample - the binary number ...1111, which does not correspond to any finite whole number.

So I found this counterexample, and saw that my attempted disproof was false, along with my dreams of fame and glory.

I was initially a bit disappointed.

The thought went through my mind: "I'll get that theorem eventually! Someday I'll disprove Cantor's Diagonal Argument, even though my first try failed!" I resented the theorem for being obstinately true, for depriving me of my fame and fortune, and I began to look for other disproofs.

And then I realized something. I realized that I had made a mistake, and that, now that I'd spotted my mistake, there was absolutely no reason to suspect the strength of Cantor's Diagonal Argument any more than other major theorems of mathematics.

I saw then very clearly that I was being offered the opportunity to become a math crank, and to spend the rest of my life writing angry letters in green ink to math professors. (I'd read a book once about math cranks.)

I did not wish this to be my future, so I gave a small laugh, and let it go. I waved Cantor's Diagonal Argument on with all good wishes, and I did not question it again.

And I don't remember, now, if I thought this at the time, or if I thought it afterward... but what a terribly unfair test to visit upon a child of thirteen. That I had to be that rational, already, at that age, or fail.

The smarter you are, the younger you may be, the first time you have what looks to you like a really revolutionary idea. I was lucky in that I saw the mistake myself; that it did not take another mathematician to point it out to me, and perhaps give me an outside source to blame. I was lucky in that the disproof was simple enough for me to understand. Maybe I would have recovered eventually, otherwise. I've recovered from much worse, as an adult. But if I had gone wrong that early, would I ever have developed that skill?

I wonder how many people writing angry letters in green ink were thirteen when they made that first fatal misstep. I wonder how many were promising minds before then.

I made a mistake. That was all. I was not really right, deep down; I did not win a moral victory; I was not displaying ambition or skepticism or any other wondrous virtue; it was not a reasonable error; I was not half right or even the tiniest fraction right. I thought a thought I would never have thought if I had been wiser, and that was all there ever was to it.

If I had been unable to admit this to myself, if I had reinterpreted my mistake as virtuous, if I had insisted on being at least a little right for the sake of pride, then I would not have let go. I would have gone on looking for a flaw in the Diagonal Argument. And, sooner or later, I might have found one.

Until you admit you were wrong, you cannot get on with your life; your self-image will still be bound to the old mistake.

Whenever you are tempted to hold on to a thought you would never have thought if you had been wiser, you are being offered the opportunity to become a crackpot - even if you never write any angry letters in green ink. If no one bothers to argue with you, or if you never tell anyone your idea, you may still be a crackpot. It's the clinging that defines it.

It's not true. It's not true deep down. It's not half-true or even a little true. It's nothing but a thought you should never have thought. Not every cloud has a silver lining. Human beings make mistakes, and not all of them are disguised successes. Human beings make mistakes; it happens, that's all. Say "oops", and get on with your life.

Sequence: Letting Go

The Importance of Saying "Oops"

Just Lose Hope Already

Referenced by: If Many-Worlds Had Come First

Original with comments: The Crackpot Offer

Radical Honesty

Eliezer Yudkowsky, 10 September 2007 06:09AM

I recently ran across this interesting article about Radical Honesty, a movement founded by a psychotherapist named Brad Blanton who suggests that we should kick our addiction to lying and just tell the complete truth all the time. I also like this quote from the Wikipedia article on Radical Honesty: "The significant majority of participants in the Radical Honesty workshops report dramatic changes in their lives after taking the course, though they are not always comfortable and positive." The movement visibly suffers from having been founded by a psychotherapist - it's more about the amazing happiness that absolute truth-telling can bring to your relationships (!!) rather than such rationalist values as seeking truth by teaching yourself a habit of honesty, or not wishing to deceive others because it infringes on their autonomy.

I once suggested a notion called "Crocker's Rules", which was the mirror image of Radical Honesty - rather than telling the whole truth to other people, you would strive to always allow others to tell you the complete truth without being offended.

Crocker's Rules didn't give you the right to say anything offensive, but other people could say potentially offensive things to you, and it was your responsibility not to be offended. This was surprisingly hard to explain to people; many people would read the careful explanation and hear, "Crocker's Rules mean you can say offensive things to other people."

I was initially a bit suspicious of Blanton's movement - it seemed like the mirror-image that so many people misinterpreted, the option of saying offensive things to other people. But Blanton makes it not only optional, but mandatory to speak your mind - a far greater inconvenience than Crocker's Rules would ever impose on anyone.

Crocker's Rules didn't catch on. Maybe it was too hard to tell the difference between someone delivering a slap in the face, and someone deliberately invoking Crocker's Rules - you don't want to miss a real clue to real hostility because of your acceptance; you wouldn't want to not believe a true fact, even if the true fact is that someone else hates you. And third parties may assume the truthteller is an offensive person no matter how much the receiver disclaims offense - they may assume the receiver is "just being polite", or that requesting honesty does not excuse its offensiveness.

Will Blanton's Rules ever catch on? I worry that Radical Honesty would selectively disadvantage rationalists in human relationships. Broadcasting your opinions is much easier when you can deceive yourself about anything you'd feel uncomfortable saying to others. I wonder whether practitioners of Radical Honesty tend to become more adept at self-deception, as they stop being able to tell white lies or admit private thoughts to themselves. I have taken a less restrictive kind of honesty upon myself - to avoid statements that are literally false - and I know that this becomes more and more difficult, more and more of a disadvantage, as I deceive myself less and less.

I suspect that the neural circuits that we use to lie to others, also censor our own thoughts. Honesty to others is important unto a rationalist, even one who is seeking a strictly selfish advantage in finding truth only for themselves. If there were a Bayesian Order, would its practitioners take a vow of Radical Honesty?

I think that if there is ever a vow of honesty among rationalists, it will be restricted in scope. Normally, perhaps, you would avoid making statements that were literally false, and be ready to accept brutal honesty from anyone who first said "Crocker's Rules". Maybe you would be Radically Honest, but only with others who had taken a vow of Radical Honesty, and who understood the trust required to tell someone the truth.

Maybe Radical Honesty would be reserved for matters sacred unto a rationalist? In some domains this is already the case. We believe that scientists should always tell the whole truth about science. It's one thing to lie in everyday life, lie to your boss, lie to the police, lie to your lover; but whoever lies in a journal article is guilty of utter heresy and will be excommunicated.

I wonder what it would be like to have anyone in the world, even a single person, who you could absolutely trust. Or what it would be like for there to be anyone in the world, even a single person, whom you had to tell all your thoughts, without possibility of concealment.

Original with comments: Radical Honesty

We Don't Really Want Your Participation

Eliezer Yudkowsky, 10 September 2007 07:53PM

At the Singularity Summit yesterday, several speakers alleged that we should "reach out" to artists and poets to encourage their participation in the Singularity dialogue. So at the end of one such session, a woman went up to the audience microphone and said:

"I am an artist. I want to participate. What should I do?"

And there was a brief, frozen silence.

I wanted to leap up and say:

No, no, I'm afraid you've misunderstood. We're just calling for greater participation by artists. We can get plenty of credit for being enlightened just by issuing the call. If we really cared what artists thought, we would find some artists and ask them questions, not call for artists to participate. We don't actually want to hear from artists. We think your opinions are stupid.

And if she'd asked me afterward, my real answer would have been:

You are not an artist, you are a human being; art is only one facet in which you express your humanity. Your reactions to the Singularity should arise from your entire self. It's perfectly all right to have a boringly normal and nonunique reaction like "I'm afraid," or "I don't think we should do this," or "I want to help, where do I send the check?" The right answer is not always unusual. Your natural reaction does not need to be unique, and that's why you don't need to try to come up with an "artist's viewpoint" on the Singularity. I would call on you to participate as a human being, not an artist. If your artistry has something to say, it will express itself naturally in your responses, without you needing to make a conscious effort to say something artist-like.

But I didn't say any of this, of course. It would have been indecorous.

And while we're on the subject, I would feel rather patronized - like a dog commanded to perform a trick - if someone presented me with a painting and said, "Say something mathematical!"

Referenced by: Applause Lights • Rationality and the English Language • A Case Study of Motivated Continuation

Original with comments: We Don't Really Want Your Participation

Applause Lights

Eliezer Yudkowsky, 11 September 2007 06:31PM

Followup to: Semantic Stopsigns, We Don't Really Want Your Participation

At the Singularity Summit 2007, one of the speakers called for democratic, multinational development of AI. So I stepped up to the microphone and asked:

Suppose that a group of democratic republics form a consortium to develop AI, and there's a lot of politicking during the process - some interest groups have unusually large influence, others get shafted - in other words, the result looks just like the products of modern democracies. Alternatively, suppose a group of rebel nerds develops an AI in their basement, and instructs the AI to poll everyone in the world - dropping cellphones to anyone who doesn't have them - and do whatever the majority says. Which of these do you think is more "democratic", and would you feel safe with either?

I wanted to find out whether he believed in the pragmatic adequacy of the democratic political process, or if he believed in the moral rightness of voting. But the speaker replied:

The first scenario sounds like an editorial in Reason magazine, and the second sounds like a Hollywood movie plot.

Confused, I asked:

Then what kind of democratic process did you have in mind?

The speaker replied:

Something like the Human Genome Project - that was an internationally sponsored research project.

I asked:

How would different interest groups resolve their conflicts in a structure like the Human Genome Project?

And the speaker said:

I don't know.

This exchange puts me in mind of a quote (which I failed to Google found by Jeff Grey and Miguel) from some dictator or other, who was asked if he had any intentions to move his pet state toward democracy:

We believe we are already within a democratic system. Some factors are still missing, like the expression of the people's will.

The substance of a democracy is the specific mechanism that resolves policy conflicts. If all groups had the same preferred policies, there would be no need for democracy - we would automatically cooperate. The resolution process can be a direct majority vote, or an elected legislature, or even a voter-sensitive behavior of an AI, but it has to be something. What does it mean to call for a "democratic" solution if you don't have a conflict-resolution mechanism in mind?

I think it means that you have said the word "democracy", so the audience is supposed to cheer. It's not so much a propositional statement, as the equivalent of the "Applause" light that tells a studio audience when to clap.

This case is remarkable only in that I mistook the applause light for a policy suggestion, with subsequent embarrassment for all. Most applause lights are much more blatant, and can be detected by a simple reversal test. For example, suppose someone says:

We need to balance the risks and opportunities of AI.

If you reverse this statement, you get:

We shouldn't balance the risks and opportunities of AI.

Since the reversal sounds abnormal, the unreversed statement is probably normal, implying it does not convey new information. There are plenty of legitimate reasons for uttering a sentence that would be uninformative in isolation. "We need to balance the risks and opportunities of AI" can introduce a discussion topic; it can emphasize the importance of a specific proposal for balancing; it can criticize an unbalanced proposal. Linking to a normal assertion can convey new information to a bounded rationalist - the link itself may not be obvious. But if no specifics follow, the sentence is probably an applause light.

I am tempted to give a talk sometime that consists of nothing but applause lights, and see how long it takes for the audience to start laughing:

I am here to propose to you today that we need to balance the risks and opportunities of advanced Artificial Intelligence. We should avoid the risks and, insofar as it is possible, realize the opportunities. We should not needlessly confront entirely unnecessary dangers. To achieve these goals, we must plan wisely and rationally. We should not act in fear and panic, or give in to technophobia; but neither should we act in blind enthusiasm. We should respect the interests of all parties with a stake in the Singularity. We must try to ensure that the benefits of advanced technologies accrue to as many individuals as possible, rather than being restricted to a few. We must try to avoid, as much as possible, violent conflicts using these technologies; and we must prevent massive destructive capability from falling into the hands of individuals. We should think through these issues before, not after, it is too late to do anything about them...

Sequence: Mysterious Answers to Mysterious Questions

"Science" as Curiosity-Stopper

Chaotic Inversion

Referenced by: Rationality and the English Language • A Case Study of Motivated Continuation • An Alien God • Superhero Bias • Evaporative Cooling of Group Beliefs • When None Dare Urge Restraint • The Fallacy of Gray • Whither Moral Progress? • Fake Norms, or "Truth" vs. Truth • Interpersonal Morality • Raised in Technophilia • Beyond the Reach of God • Crisis of Faith • Prolegomena to a Theory of Fun • That Crisis thing seems pretty useful

Original with comments: Applause Lights

Rationality and the English Language

Eliezer Yudkowsky, 12 September 2007 10:55PM

Yesterday, someone said that my writing reminded them of George Orwell's Politics and the English Language. I was honored. Especially since I'd already thought of today's topic.

If you really want an artist's perspective on rationality, then read Orwell; he is mandatory reading for rationalists as well as authors. Orwell was not a scientist, but a writer; his tools were not numbers, but words; his adversary was not Nature, but human evil. If you wish to imprison people for years without trial, you must think of some other way to say it than "I'm going to imprison Mr. Jennings for years without trial." You must muddy the listener's thinking, prevent clear images from outraging conscience. You say, "Unreliable elements were subjected to an alternative justice process."

Orwell was the outraged opponent of totalitarianism and the muddy thinking in which evil cloaks itself - which is how Orwell's writings on language ended up as classic rationalist documents on a level with Feynman, Sagan, or Dawkins.

"Writers are told to avoid usage of the passive voice." A rationalist whose background comes exclusively from science, may fail to see the flaw in the previous sentence; but anyone who's done a little writing should see it right away. I wrote the sentence in the passive voice, without telling you who tells authors to avoid passive voice. Passive voice removes the actor, leaving only the acted-upon. "Unreliable elements were subjected to an alternative justice process" - subjected by who? What does an "alternative justice process" do? With enough static noun phrases, you can keep anything unpleasant from actually happening.

Journal articles are often written in passive voice. (Pardon me, some scientists write their journal articles in passive voice. It's not as if the articles are being written by no one, with no one to blame.) It sounds more authoritative to say "The subjects were administered Progenitorivox" than "I gave each college student a bottle of 20 Progenitorivox, and told them to take one every night until they were gone." If you remove the scientist from the description, that leaves only the all-important data. But in reality the scientist is there, and the subjects are college students, and the Progenitorivox wasn't "administered" but handed over with instructions. Passive voice obscures reality.

Judging from the comments I get on Overcoming Bias, someone will protest that using the passive voice in a journal article is hardly a sin - after all, if you think about it, you can realize the scientist is there. It doesn't seem like a logical flaw. And this is why rationalists need to read Orwell, not just Feynman or even Jaynes.

Nonfiction conveys knowledge, fiction conveys experience. Medical science can extrapolate what would happen to a human unprotected in a vacuum. Fiction can make you live through it.

Some rationalists will try to analyze a misleading phrase, try to see if there might possibly be anything meaningful to it, try to construct a logical interpretation. They will be charitable, give the author the benefit of the doubt. Authors, on the other hand, are trained not to give themselves the benefit of the doubt. Whatever the audience thinks you said is what you said, whether you meant to say it or not; you can't argue with the audience no matter how clever your justifications.

A writer knows that readers will not stop for a minute to think. A fictional experience is a continuous stream of first impressions. A writer-rationalist pays attention to the experience words create. If you are evaluating the public rationality of a statement, and you analyze the words deliberatively, rephrasing propositions, trying out different meanings, searching for nuggets of truthiness, then you're losing track of the first impression - what the audience sees, or rather feels.

A novelist would notice the screaming wrongness of "The subjects were administered Progenitorivox." What life is here for a reader to live? This sentence creates a distant feeling of authoritativeness, and that's all - the only experience is the feeling of being told something reliable. A novelist would see nouns too abstract to show what actually happened - the postdoc with the bottle in his hand, trying to look stern; the student listening with a nervous grin.

My point is not to say that journal articles should be written like novels, but that a rationalist should become consciously aware of the experiences which words create. A rationalist must understand the mind and how to operate it. That includes the stream of consciousness, the part of yourself that unfolds in language. A rationalist must become consciously aware of the actual, experiential impact of phrases, beyond their mere propositional semantics.

Or to say it more bluntly: Meaning does not excuse impact!

I don't care what rational interpretation you can construct for an applause light like "AI should be developed through democratic processes". That cannot excuse its irrational impact of signaling the audience to applaud, not to mention its cloudy question-begging vagueness.

Here is Orwell, railing against the impact of cliches, their effect on the experience of thinking:

When one watches some tired hack on the platform mechanically repeating the familiar phrases -- bestial, atrocities, iron heel, bloodstained tyranny, free peoples of the world, stand shoulder to shoulder -- one often has a curious feeling that one is not watching a live human being but some kind of dummy... A speaker who uses that kind of phraseology has gone some distance toward turning himself into a machine. The appropriate noises are coming out of his larynx, but his brain is not involved, as it would be if he were choosing his words for himself...

What is above all needed is to let the meaning choose the word, and not the other way around. In prose, the worst thing one can do with words is surrender to them. When you think of a concrete object, you think wordlessly, and then, if you want to describe the thing you have been visualising you probably hunt about until you find the exact words that seem to fit it. When you think of something abstract you are more inclined to use words from the start, and unless you make a conscious effort to prevent it, the existing dialect will come rushing in and do the job for you, at the expense of blurring or even changing your meaning. Probably it is better to put off using words as long as possible and get one's meaning as clear as one can through pictures and sensations.

Peirce might have written that last paragraph. More than one path can lead to the Way.

Sequence: Politics is the Mind-Killer

False Laughter

Human Evil and Muddled Thinking

Referenced by: Human Evil and Muddled Thinking • The Logical Fallacy of Generalization from Fictional Evidence • Politics and Awful Art • Replace the Symbol with the Substance

Original with comments: Rationality and the English Language

Human Evil and Muddled Thinking

Eliezer Yudkowsky, 13 September 2007 11:43PM

Followup to: Rationality and the English Language

George Orwell saw the descent of the civilized world into totalitarianism, the conversion or corruption of one country after another; the boot stamping on a human face, forever, and remember that it is forever. You were born too late to remember a time when the rise of totalitarianism seemed unstoppable, when one country after another fell to secret police and the thunderous knock at midnight, while the professors of free universities hailed the Soviet Union's purges as progress. It feels as alien to you as fiction; it is hard for you to take seriously. Because, in your branch of time, the Berlin Wall fell. And if Orwell's name is not carved into one of those stones, it should be.

Orwell saw the destiny of the human species, and he put forth a convulsive effort to wrench it off its path. Orwell's weapon was clear writing. Orwell knew that muddled language is muddled thinking; he knew that human evil and muddled thinking intertwine like conjugate strands of DNA:

In our time, political speech and writing are largely the defence of the indefensible. Things like the continuance of British rule in India, the Russian purges and deportations, the dropping of the atom bombs on Japan, can indeed be defended, but only by arguments which are too brutal for most people to face, and which do not square with the professed aims of the political parties. Thus political language has to consist largely of euphemism, question-begging and sheer cloudy vagueness. Defenceless villages are bombarded from the air, the inhabitants driven out into the countryside, the cattle machine-gunned, the huts set on fire with incendiary bullets: this is called pacification...

Orwell was clear on the goal of his clarity:

If you simplify your English, you are freed from the worst follies of orthodoxy. You cannot speak any of the necessary dialects, and when you make a stupid remark its stupidity will be obvious, even to yourself.

To make our stupidity obvious, even to ourselves - this is the heart of Overcoming Bias.

Evil sneaks, hidden, through the unlit shadows of the mind. We look back with the clarity of history, and weep to remember the planned famines of Stalin and Mao, which killed tens of millions. We call this evil, because it was done by deliberate human intent to inflict pain and death upon innocent human beings. We call this evil, because of the revulsion that we feel against it, looking back with the clarity of history. For perpetrators of evil to avoid its natural opposition, the revulsion must remain latent. Clarity must be avoided at any cost. Even as humans of clear sight tend to oppose the evil that they see; so too does human evil, wherever it exists, set out to muddle thinking.

1984 sets this forth starkly: Orwell's ultimate villains are cutters and airbrushers of photographs (based on historical cutting and airbrushing in the Soviet Union). At the peak of all darkness in the Ministry of Love, O'Brien tortures Winston to admit that two plus two equals five:

'Do you remember,' he went on, 'writing in your diary, "Freedom is the freedom to say that two plus two make four"?'

'Yes,' said Winston.

O'Brien held up his left hand, its back towards Winston, with the thumb hidden and the four fingers extended.

'How many fingers am I holding up, Winston?'

'Four.'

'And if the party says that it is not four but five -- then how many?'

'Four.'

The word ended in a gasp of pain. The needle of the dial had shot up to fifty-five. The sweat had sprung out all over Winston's body. The air tore into his lungs and issued again in deep groans which even by clenching his teeth he could not stop. O'Brien watched him, the four fingers still extended. He drew back the lever. This time the pain was only slightly eased.

I am continually aghast at apparently intelligent folks - such as Robin's colleague Tyler Cowen - who don't think that overcoming bias is important. This is your mind we're talking about. Your human intelligence. It separates you from an ape. It built this world. You don't think how the mind works is important? You don't think the mind's systematic malfunctions are important? Do you think the Inquisition would have tortured witches, if all were ideal Bayesians?

Tyler Cowen apparently feels that overcoming bias is just as biased as bias: "I view Robin's blog as exemplifying bias, and indeed showing that bias can be very useful." I hope this is only the result of thinking too abstractly while trying to sound clever. Does Tyler seriously think that scope insensitivity to the value of human life is on the same level with trying to create plans that will really save as many lives as possible?

Orwell was forced to fight a similar attitude - that to admit to any distinction is youthful naïveté:

Stuart Chase and others have come near to claiming that all abstract words are meaningless, and have used this as a pretext for advocating a kind of political quietism. Since you don't know what Fascism is, how can you struggle against Fascism?

Maybe overcoming bias doesn't look quite exciting enough, if it's framed as a struggle against mere accidental mistakes. Maybe it's harder to get excited if there isn't some clear evil to oppose. So let us be absolutely clear that where there is human evil in the world, where there is cruelty and torture and deliberate murder, there are biases enshrouding it. Where people of clear sight oppose these biases, the concealed evil fights back. The truth does have enemies. If Overcoming Bias were a newsletter in the old Soviet Union, every poster and commenter of this blog would have been shipped off to labor camps.

In all human history, every great leap forward has been driven by a new clarity of thought. Except for a few natural catastrophes, every great woe has been driven by a stupidity. Our last enemy is ourselves; and this is a war, and we are soldiers.

Sequence: Politics is the Mind-Killer

Rationality and the English Language

Referenced by: Doublethink (Choosing to be Biased) • Stop Voting For Nincompoops

Original with comments: Human Evil and Muddled Thinking

Doublethink (Choosing to be Biased)

Eliezer Yudkowsky, 14 September 2007 08:05PM

An oblong slip of newspaper had appeared between O'Brien's fingers. For perhaps five seconds it was within the angle of Winston's vision. It was a photograph, and there was no question of its identity. It was the photograph. It was another copy of the photograph of Jones, Aaronson, and Rutherford at the party function in New York, which he had chanced upon eleven years ago and promptly destroyed. For only an instant it was before his eyes, then it was out of sight again. But he had seen it, unquestionably he had seen it! He made a desperate, agonizing effort to wrench the top half of his body free. It was impossible to move so much as a centimetre in any direction. For the moment he had even forgotten the dial. All he wanted was to hold the photograph in his fingers again, or at least to see it.

'It exists!' he cried.

'No,' said O'Brien.

He stepped across the room.

There was a memory hole in the opposite wall. O'Brien lifted the grating. Unseen, the frail slip of paper was whirling away on the current of warm air; it was vanishing in a flash of flame. O'Brien turned away from the wall.

'Ashes,' he said. 'Not even identifiable ashes. Dust. It does not exist. It never existed.'

'But it did exist! It does exist! It exists in memory. I remember it. You remember it.'

'I do not remember it,' said O'Brien.

Winston's heart sank. That was doublethink. He had a feeling of deadly helplessness. If he could have been certain that O'Brien was lying, it would not have seemed to matter. But it was perfectly possible that O'Brien had really forgotten the photograph. And if so, then already he would have forgotten his denial of remembering it, and forgotten the act of forgetting. How could one be sure that it was simple trickery? Perhaps that lunatic dislocation in the mind could really happen: that was the thought that defeated him.

-- George Orwell, 1984

What if self-deception helps us be happy? What if just running out and overcoming bias will make us - gasp! - unhappy? Surely, true wisdom would be second-order rationality, choosing when to be rational. That way you can decide which cognitive biases should govern you, to maximize your happiness.

Leaving the morality aside, I doubt such a lunatic dislocation in the mind could really happen.

Second-order rationality implies that at some point, you will think to yourself, "And now, I will irrationally believe that I will win the lottery, in order to make myself happy." But we do not have such direct control over our beliefs. You cannot make yourself believe the sky is green by an act of will. You might be able to believe you believed it - though I have just made that more difficult for you by pointing out the difference. (You're welcome!) You might even believe you were happy and self-deceived; but you would not in fact be happy and self-deceived.

For second-order rationality to be genuinely rational, you would first need a good model of reality, to extrapolate the consequences of rationality and irrationality. If you then chose to be first-order irrational, you would need to forget this accurate view. And then forget the act of forgetting. I don't mean to commit the logical fallacy of generalizing from fictional evidence, but I think Orwell did a good job of extrapolating where this path leads.

You can't know the consequences of being biased, until you have already debiased yourself. And then it is too late for self-deception.

The other alternative is to choose blindly to remain biased, without any clear idea of the consequences. This is not second-order rationality. It is willful stupidity.

Be irrationally optimistic about your driving skills, and you will be happily unconcerned where others sweat and fear. You won't have to put up with the inconvenience of a seatbelt. You will be happily unconcerned for a day, a week, a year. Then CRASH, and spend the rest of your life wishing you could scratch the itch in your phantom limb. Or paralyzed from the neck down. Or dead. It's not inevitable, but it's possible; how probable is it? You can't make that tradeoff rationally unless you know your real driving skills, so you can figure out how much danger you're placing yourself in. You can't make that tradeoff rationally unless you know about biases like neglect of probability.

No matter how many days go by in blissful ignorance, it only takes a single mistake to undo a human life, to outweigh every penny you picked up from the railroad tracks of stupidity.

One of chief pieces of advice I give to aspiring rationalists is "Don't try to be clever." And, "Listen to those quiet, nagging doubts." If you don't know, you don't know what you don't know, you don't know how much you don't know, and you don't know how much you needed to know.

There is no second-order rationality. There is only a blind leap into what may or may not be a flaming lava pit. Once you know, it will be too late for blindness.

But people neglect this, because they do not know what they do not know. Unknown unknowns are not available. They do not focus on the blank area on the map, but treat it as if it corresponded to a blank territory. When they consider leaping blindly, they check their memory for dangers, and find no flaming lava pits in the blank map. Why not leap?

Been there. Tried that. Got burned. Don't try to be clever.

I once said to a friend that I suspected the happiness of stupidity was greatly overrated. And she shook her head seriously, and said, "No, it's not; it's really not."

Maybe there are stupid happy people out there. Maybe they are happier than you are. And life isn't fair, and you won't become happier by being jealous of what you can't have. I suspect the vast majority of Overcoming Bias readers could not achieve the "happiness of stupidity" if they tried. That way is closed to you. You can never achieve that degree of ignorance, you cannot forget what you know, you cannot unsee what you see.

The happiness of stupidity is closed to you. You will never have it short of actual brain damage, and maybe not even then. You should wonder, I think, whether the happiness of stupidity is optimal - if it is the most happiness that a human can aspire to - but it matters not. That way is closed to you, if it was ever open.

All that is left to you now, is to aspire to such happiness as a rationalist can achieve. I think it may prove greater, in the end. There are bounded paths and open-ended paths; plateaus on which to laze, and mountains to climb; and if climbing takes more effort, still the mountain rises higher in the end.

Also there is more to life than happiness; and other happinesses than your own may be at stake in your decisions.

But that is moot. By the time you realize you have a choice, there is no choice. You cannot unsee what you see. The other way is closed.

Sequence: Against Doublethink

Singlethink

No, Really, I've Deceived Myself

Referenced by: Singlethink • Resist the Happy Death Spiral • Something to Protect • Joy in the Merely Real • Principles of Disagreement • Shut up and do the impossible! • Belief in Self-Deception • Don't Believe You'll Self-Deceive • Incremental Progress and the Valley

Original with comments: Doublethink (Choosing to be Biased)

Why I'm Blooking

Eliezer Yudkowsky, 15 September 2007 05:49PM

Yesterday being my 100th Overcoming Bias post, it seems an opportune time to answer a commenter's question: Why am I posting?

For a long time I've suffered from writer's molasses. Like writer's block, only instead of not writing, I write very slooowly. At least when it comes to writing Documents - papers, book chapters, website material. If I haven't published a hundred papers, it's not for lack of a hundred ideas, but because writing one paper - at my current pace - takes four months full time. I sometimes wonder if I could become a respectable academic if I wrote at a respectable pace.

Oddly enough, I can write most emails around as fast as I type. Such disorders are hard to self-diagnose, but I suspect that part of the problem is that on Documents I repeatedly reread and tweak material I've already written, instead of writing new material. James Hogan (an SF author) once told me that he was more productive on a typewriter than a word processor, because the typewriter prevented him from tweaking until the second draft.

A blook is a collection of blog posts that have been edited into a book. Logically, then, publishing a book as a series of blog posts ought to be known as "blooking".

It would be more precise to say that I'm generating raw material to be edited into a book, and collecting some feedback along the way. I make no promises for this project. (I hate promising anything unless I have already done it.) The first part of the plan, generating the raw material as blog posts, has proceeded at a respectable pace so far. Will I be able to edit the posts into chapters, so long as all the raw material is there? Will I be able to generate all the raw material, or will the project, ahem, "blog down"?

In August I decided that I was going to write one blog post per day for Overcoming Bias. This challenge began to hone my writing speed somewhat - for example, I would look at the clock and try not to take longer than an hour... or three hours... but nonetheless I began to feel the need to shove the post out the door instead of perfecting it further. This is necessary and proper.

Near the end of August, I faced a new challenge - I also had to prepare two talks for the Singularity Summit 2007 (Sep 8-9). Those were actual Documents. I knew, from previous experience, that I couldn't possibly prepare the two talks and also keep up the pace of blogging on Overcoming Bias. Blogging was using up all my writing energy already - I have only a limited supply of words per day. If I overreach one day's budget I can't write at all the next day. So (I knew) I would have to temporarily stop blogging and resume after the Summit.

And then I said to myself, Hey, if I never try to do anything "impossible", I'll never grow.

I decided I would keep up the pace on Overcoming Bias while simultaneously writing my two Summit talks. Tsuyoku naritai!

I lost sleep, and skipped exercise. But I did it. I'll remember that the next time I'm thinking of trying something impossible.

Referenced by: Newcomb's Problem and Regret of Rationality • Penguicon & Blook • Why Quantum?

Original with comments: Why I'm Blooking

Planning Fallacy

Eliezer Yudkowsky, 17 September 2007 07:06AM

The Denver International Airport opened 16 months late, at a cost overrun of $2 billion (I've also seen $3.1 billion asserted). The Eurofighter Typhoon, a joint defense project of several European countries, was delivered 54 months late at a cost of £19 billion instead of £7 billion. The Sydney Opera House may be the most legendary construction overrun of all time, originally estimated to be completed in 1963 for $7 million, and finally completed in 1973 for $102 million.

Are these isolated disasters brought to our attention by selective availability? Are they symptoms of bureaucracy or government incentive failures? Yes, very probably. But there's also a corresponding cognitive bias, replicated in experiments with individual planners.

Buehler et. al. (1995) asked their students for estimates of when they (the students) thought they would complete their personal academic projects. Specifically, the researchers asked for estimated times by which the students thought it was 50%, 75%, and 99% probable their personal projects would be done. Would you care to guess how many students finished on or before their estimated 50%, 75%, and 99% probability levels?

- 13% of subjects finished their project by the time they had assigned a 50% probability level;

- 19% finished by the time assigned a 75% probability level;

- and only 45% (less than half!) finished by the time of their 99% probability level.

As Buehler et. al. (2002) wrote, "The results for the 99% probability level are especially striking: Even when asked to make a highly conservative forecast, a prediction that they felt virtually certain that they would fulfill, students' confidence in their time estimates far exceeded their accomplishments."

More generally, this phenomenon is known as the "planning fallacy". The planning fallacy is that people think they can plan, ha ha.

A clue to the underlying problem with the planning algorithm was uncovered by Newby-Clark et. al. (2000), who found that:

- Asking subjects for their predictions based on realistic "best guess" scenarios; or

- Asking subjects for their hoped-for "best case" scenarios...

...produced indistinguishable results.

When people are asked for a "realistic" scenario, they envision everything going exactly as planned, with no unexpected delays or unforeseen catastrophes - the same vision as their "best case".

Reality, it turns out, usually delivers results somewhat worse than the "worst case".

Unlike most cognitive biases, we know a good debiasing heuristic for the planning fallacy. It won't work for messes on the scale of the Denver International Airport, but it'll work for a lot of personal planning, and even some small-scale organizational stuff. Just use an "outside view" instead of an "inside view".

People tend to generate their predictions by thinking about the particular, unique features of the task at hand, and constructing a scenario for how they intend to complete the task - which is just what we usually think of as planning. When you want to get something done, you have to plan out where, when, how; figure out how much time and how much resource is required; visualize the steps from beginning to successful conclusion. All this is the "inside view", and it doesn't take into account unexpected delays and unforeseen catastrophes. As we saw before, asking people to visualize the "worst case" still isn't enough to counteract their optimism - they don't visualize enough Murphyness.

The outside view is when you deliberately avoid thinking about the special, unique features of this project, and just ask how long it took to finish broadly similar projects in the past. This is counterintuitive, since the inside view has so much more detail - there's a temptation to think that a carefully tailored prediction, taking into account all available data, will give better results.

But experiment has shown that the more detailed subjects' visualization, the more optimistic (and less accurate) they become. Buehler et. al. (2002) asked an experimental group of subjects to describe highly specific plans for their Christmas shopping - where, when, and how. On average, this group expected to finish shopping more than a week before Christmas. Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days. Both groups finished an average of 3 days before Christmas.

Likewise, Buehler et. al. (2002), reporting on a cross-cultural study, found that Japanese students expected to finish their essays 10 days before deadline. They actually finished 1 day before deadline. Asked when they had previously completed similar tasks, they responded, "1 day before deadline." This is the power of the outside view over the inside view.

A similar finding is that experienced outsiders, who know less of the details, but who have relevant memory to draw upon, are often much less optimistic and much more accurate than the actual planners and implementers.

So there is a fairly reliable way to fix the planning fallacy, if you're doing something broadly similar to a reference class of previous projects. Just ask how long similar projects have taken in the past, without considering any of the special properties of this project. Better yet, ask an experienced outsider how long similar projects have taken.

You'll get back an answer that sounds hideously long, and clearly reflects no understanding of the special reasons why this particular task will take less time. This answer is true. Deal with it.

Buehler, R., Griffin, D. and Ross, M. 1994. Exploring the "planning fallacy": Why people underestimate their task completion times. Journal of Personality and Social Psychology, 67: 366-381.

Buehler, R., Griffin, D. and Ross, M. 1995. It's about time: Optimistic predictions in work and love. Pp. 1-32 in European Review of Social Psychology, Volume 6, eds. W. Stroebe and M. Hewstone. Chichester: John Wiley & Sons.

Buehler, R., Griffin, D. and Ross, M. 2002. Inside the planning fallacy: The causes and consequences of optimistic time predictions. Pp. 250-270 in Gilovich, T., Griffin, D. and Kahneman, D. (eds.) Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge, U.K.: Cambridge University Press.

Newby-Clark, I. R., Ross, M., Buehler, R., Koehler, D. J. and Griffin, D. 2000. People focus on optimistic and disregard pessimistic scenarios while predicting their task completion times. Journal of Experimental Psychology: Applied, 6: 171-182.

Referenced by: Kahneman's Planning Anecdote • Fake Fake Utility Functions • On Expressing Your Concerns • Penguicon & Blook • The Outside View's Domain • Contaminated by Optimism • Magical Categories • Dreams of Friendliness • The Weak Inside View • Disjunctions, Antipredictions, Etc. • Getting Nearer • OB Status Update • "Outside View!" as Conversation-Halter

Original with comments: Planning Fallacy

Kahneman's Planning Anecdote

Eliezer Yudkowsky, 17 September 2007 04:39PM

Followup to: Planning Fallacy

From "Timid Choices and Bold Forecasts: Cognitive Perspective on Risk Taking" by Nobel Laureate Daniel Kahneman and Dan Lovallo, in a discussion on "Inside and Outside Views":

In 1976 one of us (Daniel Kahneman) was involved in a project designed to develop a curriculum for the study of judgment and decision making under uncertainty for high schools in Israel. When the team had been in operation for about a year, with some significant achievements already to its credit, the discussion at one of the team meetings turned to the question of how long the project would take. To make the debate more useful, I asked everyone to indicate on a slip of paper their best estimate of the number of months that would be needed to bring the project to a well-defined stage of completion: a complete draft ready for submission to the Ministry of education. The estimates, including my own, ranged from 18 to 30 months.

At this point I had the idea of turning to one of our members, a distinguished expert in curriculum development, asking him a question phrased about as follows:

"We are surely not the only team to have tried to develop a curriculum where none existed before. Please try to recall as many such cases as you can. Think of them as they were in a stage comparable to ours at present. How long did it take them, from that point, to complete their projects?"

After a long silence, something much like the following answer was given, with obvious signs of discomfort: "First, I should say that not all teams that I can think of in a comparable stage ever did complete their task. About 40% of them eventually gave up. Of the remaining, I cannot think of any that was completed in less than seven years, nor of any that took more than ten."

In response to a further question, he answered: "No, I cannot think of any relevant factor that distinguishes us favorably from the teams I have been thinking about. Indeed, my impression is that we are slightly below average in terms of our resources and potential."

Facing the facts can be intolerably demoralizing. The participants in the meeting had professional expertise in the logic of forecasting, and none even ventured to question the relevance of the forecast implied by our expert's statistics: an even chance of failure, and a completion time of seven to ten years in case of success. Neither of these outcomes was an acceptable basis for continuing the project, but no one was willing to draw the embarrassing conclusion that it should be scrapped.

So, the forecast was quietly dropped from active debate, along with any pretense of long-term planning, and the project went on along its predictably unforeseeable path to eventual completion some eight years later.

Referenced by: The Outside View's Domain

Original with comments: Kahneman's Planning Anecdote

Conjunction Fallacy

Eliezer Yudkowsky, 19 September 2007 01:54AM

The following experiment has been slightly modified for ease of blogging. You are given the following written description, which is assumed true:

Bill is 34 years old. He is intelligent, but unimaginative, compulsive, and generally lifeless. In school, he was strong in mathematics but weak in social studies and humanities.

No complaints about the description, please, this experiment was done in 1974. Anyway, we are interested in the probability of the following propositions, which may or may not be true, and are not mutually exclusive or exhaustive:

A: Bill is an accountant.
B: Bill is a physician who plays poker for a hobby.
C: Bill plays jazz for a hobby.
D: Bill is an architect.
E: Bill is an accountant who plays jazz for a hobby.
F: Bill climbs mountains for a hobby.

Take a moment before continuing to rank these six propositions by probability, starting with the most probable propositions and ending with the least probable propositions. Again, the starting description of Bill is assumed true, but the six propositions may be true or untrue (they are not additional evidence) and they are not assumed mutually exclusive or exhaustive.

In a very similar experiment conducted by Tversky and Kahneman (1982), 92% of 94 undergraduates at the University of British Columbia gave an ordering with A > E > C. That is, the vast majority of subjects indicated that Bill was more likely to be an accountant than an accountant who played jazz, and more likely to be an accountant who played jazz than a jazz player. The ranking E > C was also displayed by 83% of 32 grad students in the decision science program of Stanford Business School, all of whom had taken advanced courses in probability and statistics.

There is a certain logical problem with saying that Bill is more likely to be an account who plays jazz, than he is to play jazz. The conjunction rule of probability theory states that, for all X and Y, P(X&Y) <= P(Y). That is, the probability that X and Y are simultaneously true, is always less than or equal to the probability that Y is true. Violating this rule is called a conjunction fallacy.

Imagine a group of 100,000 people, all of whom fit Bill's description (except for the name, perhaps). If you take the subset of all these persons who play jazz, and the subset of all these persons who play jazz and are accountants, the second subset will always be smaller because it is strictly contained within the first subset.

Could the conjunction fallacy rest on students interpreting the experimental instructions in an unexpected way - misunderstanding, perhaps, what is meant by "probable"? Here's another experiment, Tversky and Kahneman (1983), played by 125 undergraduates at UBC and Stanford for real money:

Consider a regular six-sided die with four green faces and two red faces. The die will be rolled 20 times and the sequences of greens (G) and reds (R) will be recorded. You are asked to select one sequence, from a set of three, and you will win $25 if the sequence you chose appears on successive rolls of the die. Please check the sequence of greens and reds on which you prefer to bet.

1. RGRRR
2. GRGRRR
3. GRRRRR

65% of the subjects chose sequence 2, which is most representative of the die, since the die is mostly green and sequence 2 contains the greatest proportion of green rolls. However, sequence 1 dominates sequence 2, because sequence 1 is strictly included in 2. 2 is 1 preceded by a G; that is, 2 is the conjunction of an initial G with 1. This clears up possible misunderstandings of "probability", since the goal was simply to get the $25.

Another experiment from Tversky and Kahneman (1983) was conducted at the Second International Congress on Forecasting in July of 1982. The experimental subjects were 115 professional analysts, employed by industry, universities, or research institutes. Two different experimental groups were respectively asked to rate the probability of two different statements, each group seeing only one statement:

1. "A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983."

2. "A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983."

Estimates of probability were low for both statements, but significantly lower for the first group than the second (p < .01 by Mann-Whitney). Since each experimental group only saw one statement, there is no possibility that the first group interpreted (1) to mean "suspension but no invasion".

The moral? Adding more detail or extra assumptions can make an event seem more plausible, even though the event necessarily becomes less probable.

Do you have a favorite futurist? How many details do they tack onto their amazing, futuristic predictions?

Tversky, A. and Kahneman, D. 1982. Judgments of and by representativeness. Pp 84-98 in Kahneman, D., Slovic, P., and Tversky, A., eds. Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press.

Tversky, A. and Kahneman, D. 1983. Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90: 293-315.

Referenced by: Conjunction Controversy (Or, How They Nail It Down) • Burdensome Details • Resist the Happy Death Spiral • My Strange Beliefs • Perpetual Motion Beliefs • Decoherence is Simple • When (Not) To Use Probabilities • What Do We Mean By "Rationality"?

Original with comments: Conjunction Fallacy

Conjunction Controversy (Or, How They Nail It Down)

Eliezer Yudkowsky, 20 September 2007 02:41AM

Followup to: Conjunction Fallacy

When a single experiment seems to show that subjects are guilty of some horrifying sinful bias - such as thinking that the proposition "Bill is an accountant who plays jazz" has a higher probability than "Bill is an accountant" - people may try to dismiss (not defy) the experimental data. Most commonly, by questioning whether the subjects interpreted the experimental instructions in some unexpected fashion - perhaps they misunderstood what you meant by "more probable".

Experiments are not beyond questioning; on the other hand, there should always exist some mountain of evidence which suffices to convince you. It's not impossible for researchers to make mistakes. It's also not impossible for experimental subjects to be really genuinely and truly biased. It happens. On both sides, it happens. We're all only human here.

If you think to extend a hand of charity toward experimental subjects, casting them in a better light, you should also consider thinking charitably of scientists. They're not stupid, you know. If you can see an alternative interpretation, they can see it too. This is especially important to keep in mind when you read about a bias and one or two illustrative experiments in a blog post. Yes, if the few experiments you saw were all the evidence, then indeed you might wonder. But you might also wonder if you're seeing all the evidence that supports the standard interpretation. Especially if the experiments have dates on them like "1982" and are prefaced with adjectives like "famous" or "classic".

So! This is a long post. It is a long post because nailing down a theory requires more experiments than the one or two vivid illustrations needed to merely explain. I am going to cite maybe one in twenty of the experiments that I've read about, which is maybe a hundredth of what's out there. For more information, see Tversky and Kahneman (1983) or Kahneman and Frederick (2002), both available online, from which this post is primarily drawn.

Here is (probably) the single most questioned experiment in the literature of heuristics and biases, which I reproduce here exactly as it appears in Tversky and Kahneman (1982):

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Please rank the following statements by their probability, using 1 for the most probable and 8 for the least probable:

(5.2) Linda is a teacher in elementary school.
(3.3) Linda works in a bookstore and takes Yoga classes.
(2.1) Linda is active in the feminist movement. (F)
(3.1) Linda is a psychiatric social worker.
(5.4) Linda is a member of the League of Women Voters.
(6.2) Linda is a bank teller. (T)
(6.4) Linda is an insurance salesperson.
(4.1) Linda is a bank teller and is active in the feminist movement. (T & F)

(The numbers at the start of each line are the mean ranks of each proposition, lower being more probable.)

How do you know that subjects did not interpret "Linda is a bank teller" to mean "Linda is a bank teller and is not active in the feminist movement"? For one thing, dear readers, I offer the observation that most bank tellers, even the ones who participated in anti-nuclear demonstrations in college, are probably not active in the feminist movement. So, even so, Teller should rank above Teller & Feminist. You should be skeptical of your own objections, too; else it is disconfirmation bias. But the researchers did not stop with this observation; instead, in Tversky and Kahneman (1983), they created a between-subjects experiment in which either the conjunction or the two conjuncts were deleted. Thus, in the between-subjects version of the experiment, each subject saw either (T&F), or (T), but not both. With a total of five propositions ranked, the mean rank of (T&F) was 3.3 and the mean rank of (T) was 4.4, N=86. Thus, the fallacy is not due solely to interpreting "Linda is a bank teller" to mean "Linda is a bank teller and not active in the feminist movement."

Similarly, the experiment discussed yesterday used a between-subjects design (where each subject only saw one statement) to elicit higher probabilities for "A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983" versus "A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983".

Another way of knowing whether subjects have misinterpreted an experiment is to ask the subjects directly. Also in Tversky and Kahneman (1983), a total of 103 medical internists (including 37 internists taking a postgraduate course at Harvard, and 66 internists with admitting privileges at New England Medical Center) were given problems like the following:

A 55-year-old woman had pulmonary embolism documented angiographically 10 days after a cholecstectomy. Please rank order the following in terms of the probability that they will be among the conditions experienced by the patient (use 1 for the most likely and 6 for the least likely). Naturally, the patient could experience more than one of these conditions.

- Dyspnea and hemiparesis

- Calf pain

- Pleuritic chest pain

- Syncope and tachycardia

- Hemiparesis

- Hemoptysis

As Tversky and Kahneman note, "The symptoms listed for each problem included one, denoted B, that was judged by our consulting physicians to be nonrepresentative of the patient's condition, and the conjunction of B with another highly representative symptom denoted A. In the above example of pulmonary embolism (blood clots in the lung), dyspnea (shortness of breath) is a typical symptom, whereas hemiparesis (partial paralysis) is very atypical."

In indirect tests, the mean ranks of A&B and B respectively were 2.8 and 4.3; in direct tests, they were 2.7 and 4.6. In direct tests, subjects ranked A&B above B between 73% to 100% of the time, with an average of 91%.

The experiment was designed to eliminate, in four ways, the possibility that subjects were interpreting B to mean "only B (and not A)". First, carefully wording the instructions: "...the probability that they will be among the conditions experienced by the patient", plus an explicit reminder, "the patient could experience more than one of these conditions". Second, by including indirect tests as a comparison. Third, the researchers afterward administered a questionnaire:

In assessing the probability that the patient described has a particular symptom X, did you assume that (check one):
X is the only symptom experienced by the patient?
X is among the symptoms experienced by the patient?

60 of 62 physicians, asked this question, checked the second answer.

Fourth and finally, as Tversky and Kahneman write, "An additional group of 24 physicians, mostly residents at Stanford Hospital, participated in a group discussion in which they were confronted with their conjunction fallacies in the same questionnaire. The respondents did not defend their answers, although some references were made to 'the nature of clinical experience.' Most participants appeared surprised and dismayed to have made an elementary error of reasoning."

A further experiment is also discussed in Tversky and Kahneman (1983) in which 93 subjects rated the probability that Bjorn Borg, a strong tennis player, would in the Wimbledon finals "win the match", "lose the first set", "lose the first set but win the match", and "win the first set but lose the match". The conjunction fallacy was expressed: "lose the first set but win the match" was ranked more probable than"lose the first set". Subjects were also asked to verify whether various strings of wins and losses would count as an extensional example of each case, and indeed, subjects were interpreting the cases as conjuncts which were satisfied iff both constituents were satisfied, and not interpreting them as material implications, conditional statements, or disjunctions; also, constituent B was not interpreted to exclude constituent A. The genius of this experiment was that researchers could directly test what subjects thought was the meaning of each proposition, ruling out a very large class of misunderstandings.

Does the conjunction fallacy arise because subjects misinterpret what is meant by "probability"? This can be excluded by offering students bets with payoffs. In addition to the colored dice discussed yesterday, subjects have been asked which possibility they would prefer to bet $10 on in the classic Linda experiment. This did reduce the incidence of the conjunction fallacy, but only to 56% (N=60), which is still more than half the students.

But the ultimate proof of the conjunction fallacy is also the most elegant. In the conventional interpretation of the Linda experiment, subjects substitute judgment of representativeness for judgment of probability: Their feelings of similarity between each of the propositions and Linda's description, determines how plausible it feels that each of the propositions is true of Linda. If this central theory is true, then the way in which the conjunction fallacy follows is obvious - Linda more closely resembles a feminist than a feminist bank teller, and more closely resembles a feminist bank teller than a bank teller. Well, that is our theory about what goes on in the experimental subjects minds, but how could we possibly know? We can't look inside their neural circuits - not yet! So how would you construct an experiment to directly test the standard model of the Linda experiment?

Very easily. You just take another group of experimental subjects, and ask them how much each of the propositions "resembles" Linda. This was done - see Kahneman and Frederick (2002) - and the correlation between representativeness and probability was nearly perfect. 0.99, in fact. Here's the (rather redundant) graph:

This has been replicated for numerous other experiments. For example, in the medical experiment described above, an independent group of 32 physicians from Stanford University was asked to rank each list of symptoms "by the degree to which they are representative of the clinical condition of the patient". The correlation between probability rank and representativeness rank exceeded 95% on each of the five tested medical problems.

Now, a correlation near 1 does not prove that subjects are substituting judgment of representativeness for judgment of probability. But if you want to claim that subjects are doing something else, I would like to hear the explanation for why the correlation comes out so close to 1. It will really take quite a complicated story to explain, not just why the subjects have an elaborate misunderstanding that produces an innocent and blameless conjunction fallacy, but also how it comes out to a completely coincidental correlation of nearly 1 with subjects' feeling of similarity. Across multiple experimental designs.

And we all know what happens to the probability of complicated stories: They go down when you add details to them.

Really, you know, sometimes people just make mistakes. And I'm not talking about the researchers here.

The conjunction fallacy is probably the single most questioned bias ever introduced, which means that it now ranks among the best replicated. The conventional interpretation has been nearly absolutely nailed down. Questioning, in science, calls forth answers.

I emphasize this, because it seems that when I talk about biases (especially to audiences not previously familiar with the field), a lot of people want to be charitable to experimental subjects. But it is not only experimental subjects who deserve charity. Scientists can also be unstupid. Someone else has already thought of your alternative interpretation. Someone else has already devised an experiment to test it. Maybe more than one. Maybe more than twenty.

A blank map is not a blank territory; if you don't know whether someone has tested it, that doesn't mean no one has tested it. This is not a hunter-gatherer tribe of two hundred people, where if you do not know a thing, then probably no one in your tribe knows. There are six billion people in the world, and no one can say with certitude that science does not know a thing; there is too much science. Absence of such evidence is only extremely weak evidence of absence. So do not mistake your ignorance of whether an alternative interpretation has been tested, for the positive knowledge that no one has tested it. Be charitable to scientists too. Do not say, "I bet what really happened was X", but ask, "Which experiments discriminated between the standard interpretation versus X?"

If it seems that I am driving this point home with a sledgehammer, well, yes, I guess I am. It does become a little frustrating, sometimes - to know about this overwhelming mountain of evidence from thousands of experiments, but other people have no clue that it exists. After all, if there are other experiments supporting the result, why haven't they heard of them? It's a small tribe, after all; surely they would have heard. By the same token, I have to make a conscious effort to remember that other people don't know about the evidence, and they aren't deliberately ignoring it in order to annoy me. Which is why it gets a little frustrating sometimes! We just aren't built for worlds of 6 billion people.

I'm not saying, of course, that people should stop asking questions. If you stop asking questions, you'll never find out about the mountains of experimental evidence. Faith is not understanding, only belief in a password. It is futile to believe in something, however fervently, when you don't really know what you're supposed to believe in. So I'm not saying that you should take it all on faith. I'm not saying to shut up. I'm not trying to make you feel guilty for asking questions.

I'm just saying, you should suspect the existence of other evidence, when a brief account of accepted science raises further questions in your mind. Not believe in that unseen evidence, just suspect its existence. The more so if it is a classic experiment with a standard interpretation. Ask a little more gently. Put less confidence in your brilliant new alternative hypothesis. Extend some charity to the researchers, too.

And above all, talk like a pirate. Arr!

Kahneman, D. and Frederick, S. 2002. Representativeness revisited: Attribute substitution in intuitive judgment. Pp 49-81 in Gilovich, T., Griffin, D. and Kahneman, D., eds. Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press, Cambridge.

Tversky, A. and Kahneman, D. 1982. Judgments of and by representativeness. Pp 84-98 in Kahneman, D., Slovic, P., and Tversky, A., eds. Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press.

Tversky, A. and Kahneman, D. 1983. Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90: 293-315.

Referenced by: Burdensome Details

Original with comments: Conjunction Controversy (Or, How They Nail It Down)

Burdensome Details

Eliezer Yudkowsky, 20 September 2007 11:46PM

Followup to: Conjunction Fallacy

"Merely corroborative detail, intended to give artistic verisimilitude to an otherwise bald and unconvincing narrative..."
-- Pooh-Bah, in Gilbert and Sullivan's The Mikado

The conjunction fallacy is when humans rate the probability P(A&B) higher than the probability P(B), even though it is a theorem that P(A&B) <= P(B). For example, in one experiment in 1981, 68% of the subjects ranked it more likely that "Reagan will provide federal support for unwed mothers and cut federal support to local governments" than that "Reagan will provide federal support for unwed mothers."

A long series of cleverly designed experiments, which weeded out alternative hypotheses and nailed down the standard interpretation, confirmed that conjunction fallacy occurs because we "substitute judgment of representativeness for judgment of probability". By adding extra details, you can make an outcome seem more characteristic of the process that generates it. You can make it sound more plausible that Reagan will support unwed mothers, by adding the claim that Reagan will also cut support to local governments. The implausibility of one claim is compensated by the plausibility of the other; they "average out".

Which is to say: Adding detail can make a scenario SOUND MORE PLAUSIBLE, even though the event necessarily BECOMES LESS PROBABLE.

If so, then, hypothetically speaking, we might find futurists spinning unconscionably plausible and detailed future histories, or find people swallowing huge packages of unsupported claims bundled with a few strong-sounding assertions at the center.

If you are presented with the conjunction fallacy in a naked, direct comparison, then you may succeed on that particular problem by consciously correcting yourself. But this is only slapping a band-aid on the problem, not fixing it in general.

In the 1982 experiment where professional forecasters assigned systematically higher probabilities to "Russia invades Poland, followed by suspension of diplomatic relations between USA and USSR" versus "Suspension of diplomatic relations between USA and USSR", each experimental group was only presented with one proposition. What strategy could these forecasters have followed, as a group, that would have eliminated the conjunction fallacy, when no individual knew directly about the comparison? When no individual even knew that the experiment was about the conjunction fallacy? How could they have done better on their probability judgments?

Patching one gotcha as a special case doesn't fix the general problem. The gotcha is the symptom, not the disease. That would be as silly as, oh, say, prohibiting box cutters on airplanes.

What could the forecasters have done to avoid the conjunction fallacy, without seeing the direct comparison, or even knowing that anyone was going to test them on the conjunction fallacy? It seems to me, that they would need to notice the word "and". They would need to be wary of it - not just wary, but leap back from it. Even without knowing that researchers were afterward going to test them on the conjunction fallacy particularly. They would need to notice the conjunction of two entire details, and be shocked by the audacity of anyone asking them to endorse such an insanely complicated prediction. And they would need to penalize the probability substantially - a factor of four, at least, according to the experimental details.

It might also have helped the forecasters to think about possible reasons why the US and Soviet Union would suspend diplomatic relations. The scenario is not "The US and Soviet Union suddenly suspend diplomatic relations for no reason", but "The US and Soviet Union suspend diplomatic relations for any reason."

And the subjects who rated "Reagan will provide federal support for unwed mothers and cut federal support to local governments"? Again, they would need to be shocked by the word "and". Moreover, they would need to add absurdities - where the absurdity is the log probability, so you can add it - rather than averaging them. They would need to think, "Reagan might or might not cut support to local governments (1 bit), but it seems very unlikely that he will support unwed mothers (4 bits). Total absurdity: 5 bits." Or maybe, "Reagan won't support unwed mothers. One strike and it's out. The other proposition just makes it even worse."

Similarly, consider the six-sided die with four green faces and one red face. The subjects had to bet on the sequence (1) "RGRRR", (2) "GRGRRR", or "GRRRRR" appearing anywhere in 20 rolls of the dice. 65% of the subjects chose "GRGRRR", which is strictly dominated by "RGRRR", since any sequence containing "GRGRRR" also pays off for "RGRRR". How could the subjects have done better? By noticing the inclusion? Perhaps; but that is only a band-aid, it does not fix the fundamental problem. By explicitly calculating the probabilities? That would certainly fix the fundamental problem, but you can't always calculate an exact probability.

The subjects lost heuristically by thinking: "Aha! Sequence 2 has the highest proportion of green to red! I should bet on Sequence 2!" To win heuristically, the subjects would need to think: "Aha! Sequence 1 is short! I should go with Sequence 1!"

They would need to feel a stronger emotional impact from Occam's Razor - feel every added detail as a burden, even a single extra roll of the dice.

Once upon a time, I was speaking to someone who had been mesmerized by an incautious futurist. (One who adds on lots of details that sound neat.) I was trying to explain why I was not likewise mesmerized by these amazing, incredible theories. So I explained about the conjunction fallacy, specifically the "suspending relations +/- invading Poland" experiment. And he said, "Okay, but what does this have to do with -" And I said, "It is more probable that universes replicate for any reason, than that they replicate via black holes because advanced civilizations manufacture black holes because universes evolve to make them do it." And he said, "Oh."

Until then, he had not felt these extra details as extra burdens. Instead they were corroborative detail, lending verisimilitude to the narrative. Someone presents you with a package of strange ideas, one of which is that universes replicate. Then they present support for the assertion that universes replicate. But this is not support for the package, though it is all told as one story.

You have to disentangle the details. You have to hold up every one independently, and ask, "How do we know this detail?" Someone sketches out a picture of humanity's descent into nanotechnological warfare, where China refuses to abide by an international control agreement, followed by an arms race... Wait a minute - how do you know it will be China? Is that a crystal ball in your pocket or are you just happy to be a futurist? Where are all these details coming from? Where did that specific detail come from?

For it is written:

If you can lighten your burden you must do so.
There is no straw that lacks the power to break your back.

Referenced by: Occam's Razor • No One Can Exempt You From Rationality's Laws • The Logical Fallacy of Generalization from Fictional Evidence • An Alien God • Resist the Happy Death Spiral • Uncritical Supercriticality • Absolute Authority • Perpetual Motion Beliefs • Zombies! Zombies? • The Conscious Sorites Paradox • Do Scientists Already Know This Stuff? • AIs and Gatekeepers Unite! • Crisis of Faith • Total Nano Domination • Disappointment in the Future • Investing for the Long Slump • An Especially Elegant Evpsych Experiment • That Crisis thing seems pretty useful • Dreams with Damaged Priors

Original with comments: Burdensome Details

What is Evidence?

Eliezer Yudkowsky, 22 September 2007 06:43AM

"The sentence 'snow is white' is true if and only if snow is white."
-- Alfred Tarski
"To say of what is, that it is, or of what is not, that it is not, is true."
-- Aristotle, Metaphysics IV

If these two quotes don't seem like a sufficient definition of "truth", read this. Today I'm going to talk about "evidence". (I also intend to discuss beliefs-of-fact, not emotions or morality, as distinguished here.)

Walking along the street, your shoelaces come untied. Shortly thereafter, for some odd reason, you start believing your shoelaces are untied. Light leaves the Sun and strikes your shoelaces and bounces off; some photons enter the pupils of your eyes and strike your retina; the energy of the photons triggers neural impulses; the neural impulses are transmitted to the visual-processing areas of the brain; and there the optical information is processed and reconstructed into a 3D model that is recognized as an untied shoelace. There is a sequence of events, a chain of cause and effect, within the world and your brain, by which you end up believing what you believe. The final outcome of the process is a state of mind which mirrors the state of your actual shoelaces.

What is evidence? It is an event entangled, by links of cause and effect, with whatever you want to know about. If the target of your inquiry is your shoelaces, for example, then the light entering your pupils is evidence entangled with your shoelaces. This should not be confused with the technical sense of "entanglement" used in physics - here I'm just talking about "entanglement" in the sense of two things that end up in correlated states because of the links of cause and effect between them.

Not every influence creates the kind of "entanglement" required for evidence. It's no help to have a machine that beeps when you enter winning lottery numbers, if the machine also beeps when you enter losing lottery numbers. The light reflected from your shoes would not be useful evidence about your shoelaces, if the photons ended up in the same physical state whether your shoelaces were tied or untied.

To say it abstractly: For an event to be evidence about a target of inquiry, it has to happen differently in a way that's entangled with the different possible states of the target. (To say it technically: There has to be Shannon mutual information between the evidential event and the target of inquiry, relative to your current state of uncertainty about both of them.)

Entanglement can be contagious when processed correctly, which is why you need eyes and a brain. If photons reflect off your shoelaces and hit a rock, the rock won't change much. The rock won't reflect the shoelaces in any helpful way; it won't be detectably different depending on whether your shoelaces were tied or untied. This is why rocks are not useful witnesses in court. A photographic film will contract shoelace-entanglement from the incoming photons, so that the photo can itself act as evidence. If your eyes and brain work correctly, you will become tangled up with your own shoelaces.

This is why rationalists put such a heavy premium on the paradoxical-seeming claim that a belief is only really worthwhile if you could, in principle, be persuaded to believe otherwise. If your retina ended up in the same state regardless of what light entered it, you would be blind. Some belief systems, in a rather obvious trick to reinforce themselves, say that certain beliefs are only really worthwhile if you believe them unconditionally - no matter what you see, no matter what you think. Your brain is supposed to end up in the same state regardless. Hence the phrase, "blind faith". If what you believe doesn't depend on what you see, you've been blinded as effectively as by poking out your eyeballs.

If your eyes and brain work correctly, your beliefs will end up entangled with the facts. Rational thought produces beliefs which are themselves evidence.

If your tongue speaks truly, your rational beliefs, which are themselves evidence, can act as evidence for someone else. Entanglement can be transmitted through chains of cause and effect - and if you speak, and another hears, that too is cause and effect. When you say "My shoelaces are untied" over a cellphone, you're sharing your entanglement with your shoelaces with a friend.

Therefore rational beliefs are contagious, among honest folk who believe each other to be honest. And it's why a claim that your beliefs are not contagious - that you believe for private reasons which are not transmissible - is so suspicious. If your beliefs are entangled with reality, they should be contagious among honest folk.

If your model of reality suggests that the outputs of your thought processes should not be contagious to others, then your model says that your beliefs are not themselves evidence, meaning they are not entangled with reality. You should apply a reflective correction, and stop believing.

Indeed, if you feel, on a gut level, what this all means, you will automatically stop believing. Because "my belief is not entangled with reality" means "my belief is not accurate". As soon as you stop believing "'snow is white' is true", you should (automatically!) stop believing "snow is white", or something is very wrong.

So go ahead and explain why the kind of thought processes you use systematically produce beliefs that mirror reality. Explain why you think you're rational. Why you think that, using thought processes like the ones you use, minds will end up believing "snow is white" if and only if snow is white. If you don't believe that the outputs of your thought processes are entangled with reality, why do you believe the outputs of your thought processes? It's the same thing, or it should be.

Sequence: Map and Territory

Why truth? And...

How Much Evidence Does It Take?

Referenced by: The Lens That Sees Its Flaws • How Much Evidence Does It Take? • How to Convince Me That 2 + 2 = 3 • The Bottom Line • A Rational Argument • No One Can Exempt You From Rationality's Laws • A Priori • Fake Optimization Criteria • Argument Screens Off Authority • Zombies! Zombies? • Zombie Responses • On Being Decoherent • Crisis of Faith • The Mechanics of Disagreement • Don't Believe You'll Self-Deceive • Raising the Sanity Waterline • Dreams with Damaged Priors • Frequentist Statistics are Frequently Subjective

Original with comments: What is Evidence?

The Lens That Sees Its Flaws

Eliezer Yudkowsky, 23 September 2007 12:10AM

Continuation of: What is Evidence?

Light leaves the Sun and strikes your shoelaces and bounces off; some photons enter the pupils of your eyes and strike your retina; the energy of the photons triggers neural impulses; the neural impulses are transmitted to the visual-processing areas of the brain; and there the optical information is processed and reconstructed into a 3D model that is recognized as an untied shoelace; and so you believe that your shoelaces are untied.

Here is the secret of deliberate rationality - this whole entanglement process is not magic, and you can understand it. You can understand how you see your shoelaces. You can think about which sort of thinking processes will create beliefs which mirror reality, and which thinking processes will not.

Mice can see, but they can't understand seeing. You can understand seeing, and because of that, you can do things which mice cannot do. Take a moment to marvel at this, for it is indeed marvelous.

Mice see, but they don't know they have visual cortexes, so they can't correct for optical illusions. A mouse lives in a mental world that includes cats, holes, cheese and mousetraps - but not mouse brains. Their camera does not take pictures of its own lens. But we, as humans, can look at a seemingly bizarre image, and realize that part of what we're seeing is the lens itself. You don't always have to believe your own eyes, but you have to realize that you have eyes - you must have distinct mental buckets for the map and the territory, for the senses and reality. Lest you think this a trivial ability, remember how rare it is in the animal kingdom.

The whole idea of Science is, simply, reflective reasoning about a more reliable process for making the contents of your mind mirror the contents of the world. It is the sort of thing mice would never invent. Pondering this business of "performing replicable experiments to falsify theories", we can see why it works. Science is not a separate magisterium, far away from real life and the understanding of ordinary mortals. Science is not something that only applies to the inside of laboratories. Science, itself, is an understandable process-in-the-world that correlates brains with reality.

Science makes sense, when you think about it. But mice can't think about thinking, which is why they don't have Science. One should not overlook the wonder of this - or the potential power it bestows on us as individuals, not just scientific societies.

Admittedly, understanding the engine of thought may be a little more complicated than understanding a steam engine - but it is not a fundamentally different task.

Once upon a time, I went to EFNet's #philosophy to ask "Do you believe a nuclear war will occur in the next 20 years? If no, why not?" One person who answered the question said he didn't expect a nuclear war for 100 years, because "All of the players involved in decisions regarding nuclear war are not interested right now." "But why extend that out for 100 years?", I asked. "Pure hope," was his reply.

Reflecting on this whole thought process, we can see why the thought of nuclear war makes the person unhappy, and we can see how his brain therefore rejects the belief. But, if you imagine a billion worlds - Everett branches, or Tegmark duplicates - this thought process will not systematically correlate optimists to branches in which no nuclear war occurs. (Some clever fellow is bound to say, "Ah, but since I have hope, I'll work a little harder at my job, pump up the global economy, and thus help to prevent countries from sliding into the angry and hopeless state where nuclear war is a possibility. So the two events are related after all." At this point, we have to drag in Bayes's Theorem and measure the charge of entanglement quantitatively. Your optimistic nature cannot have that large an effect on the world; it cannot, of itself, decrease the probability of nuclear war by 20%, or however much your optimistic nature shifted your beliefs. Shifting your beliefs by a large amount, due to an event that only carries a very tiny charge of entanglement, will still mess up your mapping.)

To ask which beliefs make you happy, is to turn inward, not outward - it tells you something about yourself, but it is not evidence entangled with the environment. I have nothing anything against happiness, but it should follow from your picture of the world, rather than tampering with the mental paintbrushes.

If you can see this - if you can see that hope is shifting your first-order thoughts by too large a degree - if you can understand your mind as a mapping-engine with flaws in it - then you can apply a reflective correction. The brain is a flawed lens through which to see reality. This is true of both mouse brains and human brains. But a human brain is a flawed lens that can understand its own flaws - its systematic errors, its biases - and apply second-order corrections to them. This, in practice, makes the flawed lens far more powerful. Not perfect, but far more powerful.

Sequence: Map and Territory

Occam's Razor

Referenced by: The Allais Paradox • Zut Allais! • The "Intuitions" Behind "Utilitarianism" • Words as Hidden Inferences • Zombies! Zombies? • Where Recursive Justification Hits Bottom • Belief in Self-Deception

Original with comments: The Lens That Sees Its Flaws

How Much Evidence Does It Take?

Eliezer Yudkowsky, 24 September 2007 04:06AM

Followup to: What is Evidence?

Previously, I defined evidence as "an event entangled, by links of cause and effect, with whatever you want to know about", and entangled as "happening differently for different possible states of the target". So how much entanglement - how much evidence - is required to support a belief?

Let's start with a question simple enough to be mathematical: how hard would you have to entangle yourself with the lottery in order to win? Suppose there are seventy balls, drawn without replacement, and six numbers to match for the win. Then there are 131,115,985 possible winning combinations, hence a randomly selected ticket would have a 1/131,115,985 probability of winning (0.0000007%). To win the lottery, you would need evidence selective enough to visibly favor one combination over 131,115,984 alternatives.

Suppose there are some tests you can perform which discriminate, probabilistically, between winning and losing lottery numbers. For example, you can punch a combination into a little black box that always beeps if the combination is the winner, and has only a 1/4 (25%) chance of beeping if the combination is wrong. In Bayesian terms, we would say the likelihood ratio is 4 to 1. This means that the box is 4 times as likely to beep when we punch in a correct combination, compared to how likely it is to beep for an incorrect combination.

There are still a whole lot of possible combinations. If you punch in 20 incorrect combinations, the box will beep on 5 of them by sheer chance (on average). If you punch in all 131,115,985 possible combinations, then while the box is certain to beep for the one winning combination, it will also beep for 32,778,996 losing combinations (on average).

So this box doesn't let you win the lottery, but it's better than nothing. If you used the box, your odds of winning would go from 1 in 131,115,985 to 1 in 32,778,997. You've made some progress toward finding your target, the truth, within the huge space of possibilities.

Suppose you can use another black box to test combinations twice, independently. Both boxes are certain to beep for the winning ticket. But the chance of a box beeping for a losing combination is 1/4 independently for each box; hence the chance of both boxes beeping for a losing combination is 1/16. We can say that the cumulative evidence, of two independent tests, has a likelihood ratio of 16:1. The number of losing lottery tickets that pass both tests will be (on average) 8,194,749.

Since there are 131,115,985 possible lottery tickets, you might guess that you need evidence whose strength is around 131,115,985 to 1 - an event, or series of events, which is 131,115,985 times more likely to happen for a winning combination than a losing combination. Actually, this amount of evidence would only be enough to give you an even chance of winning the lottery. Why? Because if you apply a filter of that power to 131 million losing tickets, there will be, on average, one losing ticket that passes the filter. The winning ticket will also pass the filter. So you'll be left with two tickets that passed the filter, only one of them a winner. 50% odds of winning, if you can only buy one ticket.

A better way of viewing the problem: In the beginning, there is 1 winning ticket and 131,115,984 losing tickets, so your odds of winning are 1:131,115,984. If you use a single box, the odds of it beeping are 1 for a winning ticket and 0.25 for a losing ticket. So we multiply 1:131,115,984 by 1:0.25 and get 1:32,778,996. Adding another box of evidence multiplies the odds by 1:0.25 again, so now the odds are 1 winning ticket to 8,194,749 losing tickets.

It is convenient to measure evidence in bits - not like bits on a hard drive, but mathematician's bits, which are conceptually different. Mathematician's bits are the logarithms, base 1/2, of probabilities. For example, if there are four possible outcomes A, B, C, and D, whose probabilities are 50%, 25%, 12.5%, and 12.5%, and I tell you the outcome was "D", then I have transmitted three bits of information to you, because I informed you of an outcome whose probability was 1/8.

It so happens that 131,115,984 is slightly less than 2 to the 27th power. So 14 boxes or 28 bits of evidence - an event 268,435,456:1 times more likely to happen if the ticket-hypothesis is true than if it is false - would shift the odds from 1:131,115,984 to 268,435,456:131,115,984, which reduces to 2:1. Odds of 2 to 1 mean two chances to win for each chance to lose, so the probability of winning with 28 bits of evidence is 2/3. Adding another box, another 2 bits of evidence, would take the odds to 8:1. Adding yet another two boxes would take the chance of winning to 128:1.

So if you want to license a strong belief that you will win the lottery - arbitrarily defined as less than a 1% probability of being wrong - 34 bits of evidence about the winning combination should do the trick.

In general, the rules for weighing "how much evidence it takes" follow a similar pattern: The larger the space of possibilities in which the hypothesis lies, or the more unlikely the hypothesis seems a priori compared to its neighbors, or the more confident you wish to be, the more evidence you need.

You cannot defy the rules; you cannot form accurate beliefs based on inadequate evidence. Let's say you've got 10 boxes lined up in a row, and you start punching combinations into the boxes. You cannot stop on the first combination that gets beeps from all 10 boxes, saying, "But the odds of that happening for a losing combination are a million to one! I'll just ignore those ivory-tower Bayesian rules and stop here." On average, 131 losing tickets will pass such a test for every winner. Considering the space of possibilities and the prior improbability, you jumped to a too-strong conclusion based on insufficient evidence. That's not a pointless bureaucratic regulation, it's math.

Of course, you can still believe based on inadequate evidence, if that is your whim; but you will not be able to believe accurately. It is like trying to drive your car without any fuel, because you don't believe in the silly-dilly fuddy-duddy concept that it ought to take fuel to go places. It would be so much more fun, and so much less expensive, if we just decided to repeal the law that cars need fuel. Isn't it just obviously better for everyone? Well, you can try, if that is your whim. You can even shut your eyes and pretend the car is moving. But to really arrive at accurate beliefs requires evidence-fuel, and the further you want to go, the more fuel you need.

Sequence: Map and Territory

What is Evidence?

How to Convince Me That 2 + 2 = 3

Referenced by: Einstein's Arrogance • Occam's Razor • What Evidence Filtered Evidence? • No One Can Exempt You From Rationality's Laws • Resist the Happy Death Spiral • Uncritical Supercriticality • Stop Voting For Nincompoops • GAZP vs. GLUT

Original with comments: How Much Evidence Does It Take?

Einstein's Arrogance

Eliezer Yudkowsky, 25 September 2007 01:29AM

Prerequisite: How Much Evidence Does It Take?

In 1919, Sir Arthur Eddington led expeditions to Brazil and to the island of Principe, aiming to observe solar eclipses and thereby test an experimental prediction of Einstein's novel theory of General Relativity. A journalist asked Einstein what he would do if Eddington's observations failed to match his theory. Einstein famously replied: "Then I would feel sorry for the good Lord. The theory is correct."

It seems like a rather foolhardy statement, defying the trope of Traditional Rationality that experiment above all is sovereign. Einstein seems possessed of an arrogance so great that he would refuse to bend his neck and submit to Nature's answer, as scientists must do. Who can know that the theory is correct, in advance of experimental test?

Of course, Einstein did turn out to be right. I try to avoid criticizing people when they are right. If they genuinely deserve criticism, I will not need to wait long for an occasion where they are wrong.

And Einstein may not have been quite so foolhardy as he sounded...

To assign more than 50% probability to the correct candidate from a pool of 100,000,000 possible hypotheses, you need at least 27 bits of evidence (or thereabouts). You cannot expect to find the correct candidate without tests that are this strong, because lesser tests will yield more than one candidate that passes all the tests. If you try to apply a test that only has a million-to-one chance of a false positive (~20 bits), you'll end up with a hundred candidates. Just finding the right answer, within a large space of possibilities, requires a large amount of evidence.

Traditional Rationality emphasizes justification: "If you want to convince me of X, you've got to present me with Y amount of evidence." I myself often slip into this phrasing, whenever I say something like, "To justify believing in this proposition, at more than 99% probability, requires 34 bits of evidence." Or, "in order to assign more than 50% probability to your hypothesis, you need 27 bits of evidence." The Traditional phrasing implies that you start out with a hunch, or some private line of reasoning that leads you to a suggested hypothesis, and then you have to gather "evidence" to confirm it - to convince the scientific community, or justify saying that you believe in your hunch.

But from a Bayesian perspective, you need an amount of evidence roughly equivalent to the complexity of the hypothesis just to locate the hypothesis in theory-space. It's not a question of justifying anything to anyone. If there's a hundred million alternatives, you need at least 27 bits of evidence just to focus your attention uniquely on the correct answer.

This is true even if you call your guess a "hunch" or "intuition". Hunchings and intuitings are real processes in a real brain. If your brain doesn't have at least 10 bits of genuinely entangled valid Bayesian evidence to chew on, your brain cannot single out a correct 10-bit hypothesis for your attention - consciously, subconsciously, whatever. Subconscious processes can't find one out of a million targets using only 19 bits of entanglement any more than conscious processes can. Hunches can be mysterious to the huncher, but they can't violate the laws of physics.

You see where this is going: At the time of first formulating the hypothesis - the very first time the equations popped into his head - Einstein must have had, already in his possession, sufficient observational evidence to single out the complex equations of General Relativity for his unique attention. Or he couldn't have gotten them right.

Now, how likely is it that Einstein would have exactly enough observational evidence to raise General Relativity to the level of his attention, but only justify assigning it a 55% probability? Suppose General Relativity is a 29.3-bit hypothesis. How likely is it that Einstein would stumble across exactly 29.5 bits of evidence in the course of his physics reading?

Not likely! If Einstein had enough observational evidence to single out the correct equations of General Relativity in the first place, then he probably had enough evidence to be damn sure that General Relativity was true.

In fact, since the human brain is not a perfectly efficient processor of information, Einstein probably had overwhelmingly more evidence than would, in principle, be required for a perfect Bayesian to assign massive confidence to General Relativity.

"Then I would feel sorry for the good Lord; the theory is correct," doesn't sound nearly as appalling when you look at it from that perspective. And remember that General Relativity was correct, from all the vast space of possibilities.

Referenced by: Occam's Razor • The Logical Fallacy of Generalization from Fictional Evidence • An Alien God • The Tragedy of Group Selectionism • No Evolutions for Corporations or Nanodevices • Artificial Addition • Superexponential Conceptspace, and Simple Words • GAZP vs. GLUT • Many Worlds, One Best Guess • Faster Than Science • Einstein's Speed • The Genetic Fallacy • Contaminated by Optimism • Building Something Smarter • Dreams with Damaged Priors

Original with comments: Einstein's Arrogance

Occam's Razor

Eliezer Yudkowsky, 26 September 2007 06:36AM

Followup to: Burdensome Details, How Much Evidence?

The more complex an explanation is, the more evidence you need just to find it in belief-space. (In Traditional Rationality this is often phrased misleadingly, as "The more complex a proposition is, the more evidence is required to argue for it.") How can we measure the complexity of an explanation? How can we determine how much evidence is required?

Occam's Razor is often phrased as "The simplest explanation that fits the facts." Robert Heinlein replied that the simplest explanation is "The lady down the street is a witch; she did it."

One observes that the length of an English sentence is not a good way to measure "complexity". And "fitting" the facts by merely failing to prohibit them is insufficient.

Why, exactly, is the length of an English sentence a poor measure of complexity? Because when you speak a sentence aloud, you are using labels for concepts that the listener shares - the receiver has already stored the complexity in them. Suppose we abbreviated Heinlein's whole sentence as "Tldtsiawsdi!" so that the entire explanation can be conveyed in one word; better yet, we'll give it a short arbitrary label like "Fnord!" Does this reduce the complexity? No, because you have to tell the listener in advance that "Tldtsiawsdi!" stands for "The lady down the street is a witch; she did it." "Witch", itself, is a label for some extraordinary assertions - just because we all know what it means doesn't mean the concept is simple.

An enormous bolt of electricity comes out of the sky and hits something, and the Norse tribesfolk say, "Maybe a really powerful agent was angry and threw a lightning bolt." The human brain is the most complex artifact in the known universe. If anger seems simple, it's because we don't see all the neural circuitry that's implementing the emotion. (Imagine trying to explain why Saturday Night Live is funny, to an alien species with no sense of humor. But don't feel superior; you yourself have no sense of fnord.) The complexity of anger, and indeed the complexity of intelligence, was glossed over by the humans who hypothesized Thor the thunder-agent.

To a human, Maxwell's Equations take much longer to explain than Thor. Humans don't have a built-in vocabulary for calculus the way we have a built-in vocabulary for anger. You've got to explain your language, and the language behind the language, and the very concept of mathematics, before you can start on electricity.

And yet it seems that there should be some sense in which Maxwell's Equations are simpler than a human brain, or Thor the thunder-agent.

There is: It's enormously easier (as it turns out) to write a computer program that simulates Maxwell's Equations, compared to a computer program that simulates an intelligent emotional mind like Thor.

The formalism of Solomonoff Induction measures the "complexity of a description" by the length of the shortest computer program which produces that description as an output. To talk about the "shortest computer program" that does something, you need to specify a space of computer programs, which requires a language and interpreter. Solomonoff Induction uses Turing machines, or rather, bitstrings that specify Turing machines. What if you don't like Turing machines? Then there's only a constant complexity penalty to design your own Universal Turing Machine that interprets whatever code you give it in whatever programming language you like. Different inductive formalisms are penalized by a worst-case constant factor relative to each other, corresponding to the size of a universal interpreter for that formalism.

In the better (IMHO) versions of Solomonoff Induction, the computer program does not produce a deterministic prediction, but assigns probabilities to strings. For example, we could write a program to explain a fair coin by writing a program that assigns equal probabilities to all 2^N strings of length N. This is Solomonoff Induction's approach to fitting the observed data. The higher the probability a program assigns to the observed data, the better that program fits the data. And probabilities must sum to 1, so for a program to better "fit" one possibility, it must steal probability mass from some other possibility which will then "fit" much more poorly. There is no superfair coin that assigns 100% probability to heads and 100% probability to tails.

How do we trade off the fit to the data, against the complexity of the program? If you ignore complexity penalties, and think only about fit, then you will always prefer programs that claim to deterministically predict the data, assign it 100% probability. If the coin shows "HTTHHT", then the program which claims that the coin was fixed to show "HTTHHT" fits the observed data 64 times better than the program which claims the coin is fair. Conversely, if you ignore fit, and consider only complexity, then the "fair coin" hypothesis will always seem simpler than any other hypothesis. Even if the coin turns up "HTHHTHHHTHHHHTHHHHHT..." Indeed, the fair coin is simpler and it fits this data exactly as well as it fits any other string of 20 coinflips - no more, no less - but we see another hypothesis, seeming not too complicated, that fits the data much better.

If you let a program store one more binary bit of information, it will be able to cut down a space of possibilities by half, and hence assign twice as much probability to all the points in the remaining space. This suggests that one bit of program complexity should cost at least a "factor of two gain" in the fit. If you try to design a computer program that explicitly stores an outcome like "HTTHHT", the six bits that you lose in complexity must destroy all plausibility gained by a 64-fold improvement in fit. Otherwise, you will sooner or later decide that all fair coins are fixed.

Unless your program is being smart, and compressing the data, it should do no good just to move one bit from the data into the program description.

The way Solomonoff induction works to predict sequences is that you sum up over all allowed computer programs - if any program is allowed, Solomonoff induction becomes uncomputable - with each program having a prior probability of (1/2) to the power of its code length in bits, and each program is further weighted by its fit to all data observed so far. This gives you a weighted mixture of experts that can predict future bits.

The Minimum Message Length formalism is nearly equivalent to Solomonoff induction. You send a string describing a code, and then you send a string describing the data in that code. Whichever explanation leads to the shortest total message is the best. If you think of the set of allowable codes as a space of computer programs, and the code description language as a universal machine, then Minimum Message Length is nearly equivalent to Solomonoff induction. (Nearly, because it chooses the shortest program, rather than summing up over all programs.)

This lets us see clearly the problem with using "The lady down the street is a witch; she did it" to explain the pattern in the sequence "0101010101". If you're sending a message to a friend, trying to describe the sequence you observed, you would have to say: "The lady down the street is a witch; she made the sequence come out 0101010101." Your accusation of witchcraft wouldn't let you shorten the rest of the message; you would still have to describe, in full detail, the data which her witchery caused.

Witchcraft may fit our observations in the sense of qualitatively permitting them; but this is because witchcraft permits everything, like saying "Phlogiston!" So, even after you say "witch", you still have to describe all the observed data in full detail. You have not compressed the total length of the message describing your observations by transmitting the message about witchcraft; you have simply added a useless prologue, increasing the total length.

The real sneakiness was concealed in the word "it" of "A witch did it". A witch did what?

Of course, thanks to hindsight bias and anchoring and fake explanations and fake causality and positive bias and motivated cognition, it may seem all too obvious that if a woman is a witch, of course she would make the coin come up 0101010101. But of this I have already spoken.

Sequence: Map and Territory

How to Convince Me That 2 + 2 = 3

The Lens That Sees Its Flaws

Referenced by: Einstein's Arrogance • How to Convince Me That 2 + 2 = 3 • A Priori • The Logical Fallacy of Generalization from Fictional Evidence • Pascal's Mugging: Tiny Probabilities of Vast Utilities • The Hidden Complexity of Wishes • Resist the Happy Death Spiral • Fake Utility Functions • Entropy, and Short Codes • Zombies! Zombies? • Belief in the Implied Invisible • Where Philosophy Meets Science • Which Basis Is More Fundamental? • Where Recursive Justification Hits Bottom • My Kind of Reflection • Magical Categories • Points of Departure • Excluding the Supernatural • Crisis of Faith • Measuring Optimization Power • Complexity and Intelligence • Interpersonal Entanglement • Raising the Sanity Waterline

Original with comments: Occam's Razor

9/26 is Petrov Day

Eliezer Yudkowsky, 26 September 2007 04:14PM

(Posted last year, but bumped to the top for today.)

Today is September 26th, Petrov Day, celebrated to honor the deed of Stanislav Yevgrafovich Petrov on September 26th, 1983. Wherever you are, whatever you're doing, take a minute to not destroy the world.

The story begins on September 1st, 1983, when Soviet jet interceptors shot down a Korean Air Lines civilian airliner after the aircraft crossed into Soviet airspace and then, for reasons still unknown, failed to respond to radio hails. 269 passengers and crew died, including US Congressman Lawrence McDonald. Ronald Reagan called it "barbarism", "inhuman brutality", "a crime against humanity that must never be forgotten". Note that this was already a very, very poor time for US/USSR relations. Andropov, the ailing Soviet leader, was half-convinced the US was planning a first strike. The KGB sent a flash message to its operatives warning them to prepare for possible nuclear war.

On September 26th, 1983, Lieutenant Colonel Stanislav Yevgrafovich Petrov was the officer on duty when the warning system reported a US missile launch. Petrov kept calm, suspecting a computer error.

Then the system reported another US missile launch.

And another, and another, and another.

What had actually happened, investigators later determined, was sunlight on high-altitude clouds aligning with the satellite view on a US missile base.

In the command post there were beeping signals, flashing lights, and officers screaming at people to remain calm. According to several accounts I've read, there was a large flashing screen from the automated computer system saying simply "START" (presumably in Russian). Afterward, when investigators asked Petrov why he hadn't written everything down in the logbook, Petrov replied,"Because I had a phone in one hand and the intercom in the other, and I don't have a third hand."

The policy of the Soviet Union called for launch on warning. The Soviet Union's land radar could not detect missiles over the horizon, and waiting for positive identification would limit the response time to minutes. Petrov's report would be relayed to his military superiors, who would decide whether to start a nuclear war.

Petrov decided that, all else being equal, he would prefer not to destroy the world. He sent messages declaring the launch detection a false alarm, based solely on his personal belief that the US did not seem likely to start an attack using only five missiles.

Petrov was first congratulated, then extensively interrogated, then reprimanded for failing to follow procedure. He resigned in poor health from the military several months later. According to Wikipedia, he is spending his retirement in relative poverty in the town of Fryazino, on a pension of $200/month. In 2004, the Association of World Citizens gave Petrov a trophy and $1000. There is also a movie scheduled for release in 2008, entitled The Red Button and the Man Who Saved the World.

Maybe someday, the names of people who decide not to start nuclear wars will be as well known as the name of Britney Spears. Looking forward to such a time, when humankind has grown a little wiser, let us celebrate, in this moment, Petrov Day.

Referenced by: Hiroshima Day

Original with comments: 9/26 is Petrov Day

How to Convince Me That 2 + 2 = 3

Eliezer Yudkowsky, 27 September 2007 11:00PM

In "What is Evidence?", I wrote:

This is why rationalists put such a heavy premium on the paradoxical-seeming claim that a belief is only really worthwhile if you could, in principle, be persuaded to believe otherwise. If your retina ended up in the same state regardless of what light entered it, you would be blind... Hence the phrase, "blind faith". If what you believe doesn't depend on what you see, you've been blinded as effectively as by poking out your eyeballs.

Cihan Baran replied:

I can not conceive of a situation that would make 2+2 = 4 false. Perhaps for that reason, my belief in 2+2=4 is unconditional.

I admit, I cannot conceive of a "situation" that would make 2 + 2 = 4 false. (There are redefinitions, but those are not "situations", and then you're no longer talking about 2, 4, =, or +.) But that doesn't make my belief unconditional. I find it quite easy to imagine a situation which would convince me that 2 + 2 = 3.

Suppose I got up one morning, and took out two earplugs, and set them down next to two other earplugs on my nighttable, and noticed that there were now three earplugs, without any earplugs having appeared or disappeared - in contrast to my stored memory that 2 + 2 was supposed to equal 4. Moreover, when I visualized the process in my own mind, it seemed that making XX and XX come out to XXXX required an extra X to appear from nowhere, and was, moreover, inconsistent with other arithmetic I visualized, since subtracting XX from XXX left XX, but subtracting XX from XXXX left XXX. This would conflict with my stored memory that 3 - 2 = 1, but memory would be absurd in the face of physical and mental confirmation that XXX - XX = XX.

I would also check a pocket calculator, Google, and perhaps my copy of 1984 where Winston writes that "Freedom is the freedom to say two plus two equals three." All of these would naturally show that the rest of the world agreed with my current visualization, and disagreed with my memory, that 2 + 2 = 3.

How could I possibly have ever been so deluded as to believe that 2 + 2 = 4? Two explanations would come to mind: First, a neurological fault (possibly caused by a sneeze) had made all the additive sums in my stored memory go up by one. Second, someone was messing with me, by hypnosis or by my being a computer simulation. In the second case, I would think it more likely that they had messed with my arithmetic recall than that 2 + 2 actually equalled 4. Neither of these plausible-sounding explanations would prevent me from noticing that I was very, very, very confused.

What would convince me that 2 + 2 = 3, in other words, is exactly the same kind of evidence that currently convinces me that 2 + 2 = 4: The evidential crossfire of physical observation, mental visualization, and social agreement.

There was a time when I had no idea that 2 + 2 = 4. I did not arrive at this new belief by random processes - then there would have been no particular reason for my brain to end up storing "2 + 2 = 4" instead of "2 + 2 = 7". The fact that my brain stores an answer surprisingly similar to what happens when I lay down two earplugs alongside two earplugs, calls forth an explanation of what entanglement produces this strange mirroring of mind and reality.

There's really only two possibilities, for a belief of fact - either the belief got there via a mind-reality entangling process, or not. If not, the belief can't be correct except by coincidence. For beliefs with the slightest shred of internal complexity (requiring a computer program of more than 10 bits to simulate), the space of possibilities is large enough that coincidence vanishes.

Unconditional facts are not the same as unconditional beliefs. If entangled evidence convinces me that a fact is unconditional, this doesn't mean I always believed in the fact without need of entangled evidence.

I believe that 2 + 2 = 4, and I find it quite easy to conceive of a situation which would convince me that 2 + 2 = 3. Namely, the same sort of situation that currently convinces me that 2 + 2 = 4. Thus I do not fear that I am a victim of blind faith.

If there are any Christians in the audience who know Bayes's Theorem (no numerophobes, please) might I inquire of you what situation would convince you of the truth of Islam? Presumably it would be the same sort of situation causally responsible for producing your current belief in Christianity: We would push you screaming out of the uterus of a Muslim woman, and have you raised by Muslim parents who continually told you that it is good to believe unconditionally in Islam. Or is there more to it than that? If so, what situation would convince you of Islam, or at least, non-Christianity?

Sequence: Map and Territory

How Much Evidence Does It Take?

Occam's Razor

Sequence: Overly Convenient Excuses

Absolute Authority

Infinite Certainty

Referenced by: A Priori • Infinite Certainty • The Parable of Hemlock • What Would You Do Without Morality? • Complexity and Intelligence

Original with comments: How to Convince Me That 2 + 2 = 3

The Bottom Line

Eliezer Yudkowsky, 28 September 2007 05:47PM

There are two sealed boxes up for auction, box A and box B. One and only one of these boxes contains a valuable diamond. There are all manner of signs and portents indicating whether a box contains a diamond; but I have no sign which I know to be perfectly reliable. There is a blue stamp on one box, for example, and I know that boxes which contain diamonds are more likely than empty boxes to show a blue stamp. Or one box has a shiny surface, and I have a suspicion - I am not sure - that no diamond-containing box is ever shiny.

Now suppose there is a clever arguer, holding a sheet of paper, and he says to the owners of box A and box B: "Bid for my services, and whoever wins my services, I shall argue that their box contains the diamond, so that the box will receive a higher price." So the box-owners bid, and box B's owner bids higher, winning the services of the clever arguer.

The clever arguer begins to organize his thoughts. First, he writes, "And therefore, box B contains the diamond!" at the bottom of his sheet of paper. Then, at the top of the paper, he writes, "Box B shows a blue stamp," and beneath it, "Box A is shiny", and then, "Box B is lighter than box A", and so on through many signs and portents; yet the clever arguer neglects all those signs which might argue in favor of box A. And then the clever arguer comes to me and recites from his sheet of paper: "Box B shows a blue stamp, and box A is shiny," and so on, until he reaches: "And therefore, box B contains the diamond."

But consider: At the moment when the clever arguer wrote down his conclusion, at the moment he put ink on his sheet of paper, the evidential entanglement of that physical ink with the physical boxes became fixed.

It may help to visualize a collection of worlds - Everett branches or Tegmark duplicates - within which there is some objective frequency at which box A or box B contains a diamond. There's likewise some objective frequency within the subset "worlds with a shiny box A" where box B contains the diamond; and some objective frequency in "worlds with shiny box A and blue-stamped box B" where box B contains the diamond.

The ink on paper is formed into odd shapes and curves, which look like this text: "And therefore, box B contains the diamond." If you happened to be a literate English speaker, you might become confused, and think that this shaped ink somehow meant that box B contained the diamond. Subjects instructed to say the color of printed pictures and shown the picture "green" often say "green" instead of "red". It helps to be illiterate, so that you are not confused by the shape of the ink.

To us, the true import of a thing is its entanglement with other things. Consider again the collection of worlds, Everett branches or Tegmark duplicates. At the moment when all clever arguers in all worlds put ink to the bottom line of their paper - let us suppose this is a single moment - it fixed the correlation of the ink with the boxes. The clever arguer writes in non-erasable pen; the ink will not change. The boxes will not change. Within the subset of worlds where the ink says "And therefore, box B contains the diamond," there is already some fixed percentage of worlds where box A contains the diamond. This will not change regardless of what is written in on the blank lines above.

So the evidential entanglement of the ink is fixed, and I leave to you to decide what it might be. Perhaps box owners who believe a better case can be made for them are more liable to hire advertisers; perhaps box owners who fear their own deficiencies bid higher. If the box owners do not themselves understand the signs and portents, then the ink will be completely unentangled with the boxes' contents, though it may tell you something about the owners' finances and bidding habits.

Now suppose another person present is genuinely curious, and she first writes down all the distinguishing signs of both boxes on a sheet of paper, and then applies her knowledge and the laws of probability and writes down at the bottom: "Therefore, I estimate an 85% probability that box B contains the diamond." Of what is this handwriting evidence? Examining the chain of cause and effect leading to this physical ink on physical paper, I find that the chain of causality wends its way through all the signs and portents of the boxes, and is dependent on these signs; for in worlds with different portents, a different probability is written at the bottom.

So the handwriting of the curious inquirer is entangled with the signs and portents and the contents of the boxes, whereas the handwriting of the clever arguer is evidence only of which owner paid the higher bid. There is a great difference in the indications of ink, though one who foolishly read aloud the ink-shapes might think the English words sounded similar.

Your effectiveness as a rationalist is determined by whichever algorithm actually writes the bottom line of your thoughts. If your car makes metallic squealing noises when you brake, and you aren't willing to face up to the financial cost of getting your brakes replaced, you can decide to look for reasons why your car might not need fixing. But the actual percentage of you that survive in Everett branches or Tegmark worlds - which we will take to describe your effectiveness as a rationalist - is determined by the algorithm that decided which conclusion you would seek arguments for. In this case, the real algorithm is "Never repair anything expensive." If this is a good algorithm, fine; if this is a bad algorithm, oh well. The arguments you write afterward, above the bottom line, will not change anything either way.

Addendum: This is intended as a caution for your own thinking, not a Fully General Counterargument against conclusions you don't like. For it is indeed a clever argument to say "My opponent is a clever arguer", if you are paying yourself to retain whatever beliefs you had at the start. The world's cleverest arguer may point out that the sun is shining, and yet it is still probably daytime. See What Evidence Filtered Evidence? for more on this topic.

Sequence: Against Rationalization

One Argument Against An Army

What Evidence Filtered Evidence?

Referenced by: What Evidence Filtered Evidence? • Rationalization • A Rational Argument • We Change Our Minds Less Often Than We Think • A Priori • Priming and Contamination • Hold Off On Proposing Solutions • Fake Justification • An Alien God • The Tragedy of Group Selectionism • Fake Optimization Criteria • Decoherent Essences • LA-602 vs. RHIC Review • Math is Subjunctively Objective • A Genius for Destruction • Anthropomorphic Optimism • Crisis of Faith • Back Up and Ask Whether, Not Why • The Weak Inside View • Is That Your True Rejection? • Formative Youth • Raising the Sanity Waterline • That Crisis thing seems pretty useful • Practical Advice Backed By Deep Theories

Original with comments: The Bottom Line

What Evidence Filtered Evidence?

Eliezer Yudkowsky, 29 September 2007 11:10PM

Yesterday I discussed the dilemma of the clever arguer, hired to sell you a box that may or may not contain a diamond. The clever arguer points out to you that the box has a blue stamp, and it is a valid known fact that diamond-containing boxes are more likely than empty boxes to bear a blue stamp. What happens at this point, from a Bayesian perspective? Must you helplessly update your probabilities, as the clever arguer wishes?

If you can look at the box yourself, you can add up all the signs yourself. What if you can't look? What if the only evidence you have is the word of the clever arguer, who is legally constrained to make only true statements, but does not tell you everything he knows? Each statement that he makes is valid evidence - how could you not update your probabilities? Has it ceased to be true that, in such-and-such a proportion of Everett branches or Tegmark duplicates in which box B has a blue stamp, box B contains a diamond? According to Jaynes, a Bayesian must always condition on all known evidence, on pain of paradox. But then the clever arguer can make you believe anything he chooses, if there is a sufficient variety of signs to selectively report. That doesn't sound right.

Consider a simpler case, a biased coin, which may be biased to 2/3 heads 1/3 tails, or 1/3 heads 2/3 tails, both cases being equally likely a priori. Each H observed is 1 bit of evidence for an H-biased coin; each T observed is 1 bit of evidence for a T-biased coin. I flip the coin ten times, and then I tell you, "The 4th flip, 6th flip, and 9th flip came up heads." What is your posterior probability that the coin is H-biased?

And the answer is that it could be almost anything, depending on what chain of cause and effect lay behind my utterance of those words - my selection of which flips to report.

- I might be following the algorithm of reporting the result of the 4th, 6th, and 9th flips, regardless of the result of that and all other flips. If you know that I used this algorithm, the posterior odds are 8:1 in favor of an H-biased coin.

- I could be reporting on all flips, and only flips, that came up heads. In this case, you know that all 7 other flips came up tails, and the posterior odds are 1:16 against the coin being H-biased.

- I could have decided in advance to say the result of the 4th, 6th, and 9th flips only if the probability of the coin being H-biased exceeds 98%. And so on.

Or consider the Monty Hall problem:

On a game show, you are given the choice of three doors leading to three rooms. You know that in one room is $100,000, and the other two are empty. The host asks you to pick a door, and you pick door #1. Then the host opens door #2, revealing an empty room. Do you want to switch to door #3, or stick with door #1?

The answer depends on the host's algorithm. If the host always opens a door and always picks a door leading to an empty room, then you should switch to door #3. If the host always opens door #2 regardless of what is behind it, #1 and #3 both have 50% probabilities of containing the money. If the host only opens a door, at all, if you initially pick the door with the money, then you should definitely stick with #1.

You shouldn't just condition on #2 being empty, but this fact plus the fact of the host choosing to open door #2. Many people are confused by the standard Monty Hall problem because they update only on #2 being empty, in which case #1 and #3 have equal probabilities of containing the money. This is why Bayesians are commanded to condition on all of their knowledge, on pain of paradox.

When someone says, "The 4th coinflip came up heads", we are not conditioning on the 4th coinflip having come up heads - we are not taking the subset of all possible worlds where the 4th coinflip came up heads - rather we are conditioning on the subset of all possible worlds where a speaker following some particular algorithm said "The 4th coinflip came up heads." The spoken sentence is not the fact itself; don't be led astray by the mere meanings of words.

Most legal processes work on the theory that every case has exactly two opposed sides and that it is easier to find two biased humans than one unbiased one. Between the prosecution and the defense, someone has a motive to present any given piece of evidence, so the court will see all the evidence; that is the theory. If there are two clever arguers in the box dilemma, it is not quite as good as one curious inquirer, but it is almost as good. But that is with two boxes. Reality often has many-sided problems, and deep problems, and nonobvious answers, which are not readily found by Blues and Greens screaming at each other.

Beware lest you abuse the notion of evidence-filtering as a Fully General Counterargument to exclude all evidence you don't like: "That argument was filtered, therefore I can ignore it." If you're ticked off by a contrary argument, then you are familiar with the case, and care enough to take sides. You probably already know your own side's strongest arguments. You have no reason to infer, from a contrary argument, the existence of new favorable signs and portents which you have not yet seen. So you are left with the uncomfortable facts themselves; a blue stamp on box B is still evidence.

But if you are hearing an argument for the first time, and you are only hearing one side of the argument, then indeed you should beware! In a way, no one can really trust the theory of natural selection until after they have listened to creationists for five minutes; and then they know it's solid.

Sequence: Against Rationalization

The Bottom Line

Rationalization

Referenced by: The Bottom Line • Rationalization • A Rational Argument • Priming and Contamination • An Alien God • Fake Optimization Criteria • Conjuring An Evolution To Serve You • Resist the Happy Death Spiral

Original with comments: What Evidence Filtered Evidence?

Rationalization

Eliezer Yudkowsky, 30 September 2007 07:29PM

Followup to: The Bottom Line, What Evidence Filtered Evidence?

In "The Bottom Line", I presented the dilemma of two boxes only one of which contains a diamond, with various signs and portents as evidence. I dichotomized the curious inquirer and the clever arguer. The curious inquirer writes down all the signs and portents, and processes them, and finally writes down "Therefore, I estimate an 85% probability that box B contains the diamond." The clever arguer works for the highest bidder, and begins by writing, "Therefore, box B contains the diamond", and then selects favorable signs and portents to list on the lines above.

The first procedure is rationality. The second procedure is generally known as "rationalization".

"Rationalization." What a curious term. I would call it a wrong word. You cannot "rationalize" what is not already rational. It is as if "lying" were called "truthization".

On a purely computational level, there is a rather large difference between:

1. Starting from evidence, and then crunching probability flows, in order to output a probable conclusion. (Writing down all the signs and portents, and then flowing forward to a probability on the bottom line which depends on those signs and portents.)

2. Starting from a conclusion, and then crunching probability flows, in order to output evidence apparently favoring that conclusion. (Writing down the bottom line, and then flowing backward to select signs and portents for presentation on the lines above.)

What fool devised such confusingly similar words, "rationality" and "rationalization", to describe such extraordinarily different mental processes? I would prefer terms that made the algorithmic difference obvious, like "rationality" versus "giant sucking cognitive black hole".

Not every change is an improvement, but every improvement is necessarily a change. You cannot obtain more truth for a fixed proposition by arguing it; you can make more people believe it, but you cannot make it more true. To improve our beliefs, we must necessarily change our beliefs. Rationality is the operation that we use to obtain more truth-value for our beliefs by changing them. Rationalization operates to fix beliefs in place; it would be better named "anti-rationality", both for its pragmatic results and for its reversed algorithm.

"Rationality" is the forward flow that gathers evidence, weighs it, and outputs a conclusion. The curious inquirer used a forward-flow algorithm: first gathering the evidence, writing down a list of all visible signs and portents, which they then processed forward to obtain a previously unknown probability for the box containing the diamond. During the entire time that the rationality-process was running forward, the curious inquirer did not yet know their destination, which was why they were curious. In the Way of Bayes, the prior probability equals the expected posterior probability: If you know your destination, you are already there.

"Rationalization" is a backward flow from conclusion to selected evidence. First you write down the bottom line, which is known and fixed; the purpose of your processing is to find out which arguments you should write down on the lines above. This, not the bottom line, is the variable unknown to the running process.

I fear that Traditional Rationality does not properly sensitize its users to the difference between forward flow and backward flow. In Traditional Rationality, there is nothing wrong with the scientist who arrives at a pet hypothesis and then sets out to find an experiment that proves it. A Traditional Rationalist would look at this approvingly, and say, "This pride is the engine that drives Science forward." Well, it is the engine that drives Science forward. It is easier to find a prosecutor and defender biased in opposite directions, than to find a single unbiased human.

But just because everyone does something, doesn't make it okay. It would be better yet if the scientist, arriving at a pet hypothesis, set out to test that hypothesis for the sake of curiosity - creating experiments that would drive their own beliefs in an unknown direction.

If you genuinely don't know where you are going, you will probably feel quite curious about it. Curiosity is the first virtue, without which your questioning will be purposeless and your skills without direction.

Feel the flow of the Force, and make sure it isn't flowing backwards.

Sequence: Against Rationalization

What Evidence Filtered Evidence?

A Rational Argument

Referenced by: A Rational Argument • Avoiding Your Belief's Real Weak Points • No One Can Exempt You From Rationality's Laws • Priming and Contamination • Hold Off On Proposing Solutions • Pascal's Mugging: Tiny Probabilities of Vast Utilities • An Alien God • The Tragedy of Group Selectionism • Fake Optimization Criteria • Affective Death Spirals • Resist the Happy Death Spiral • Many Worlds, One Best Guess • Against Devil's Advocacy • Anthropomorphic Optimism • Contaminated by Optimism • Raised in Technophilia • Crisis of Faith • That Crisis thing seems pretty useful

Original with comments: Rationalization

Recommended Rationalist Reading

Eliezer Yudkowsky, 01 October 2007 06:36PM

From this month's Open Thread, Stirling Westrup asks:

There is much mention in this blog about Bayesian rationality, or the use of Bayes' methods in decision making. Now, I studied Bayes conditional probabilities in Statistics class in University many years ago, but my knowledge of the theory ends there. Can you recommend any good books on the subject?

In fact, do you folks have a recommended reading list (other than this blog, of course!) for those trying to identify and overcome their own biases?

I second the question. My own recommendations will be found in the comments.

Original with comments: Recommended Rationalist Reading

A Rational Argument

Eliezer Yudkowsky, 02 October 2007 06:35PM

Followup to: The Bottom Line, Rationalization

You are, by occupation, a campaign manager, and you've just been hired by Mortimer Q. Snodgrass, the Green candidate for Mayor of Hadleyburg. As a campaign manager reading a blog on rationality, one question lies foremost on your mind: "How can I construct an impeccable rational argument that Mortimer Q. Snodgrass is the best candidate for Mayor of Hadleyburg?"

Sorry. It can't be done.

"What?" you cry. "But what if I use only valid support to construct my structure of reason? What if every fact I cite is true to the best of my knowledge, and relevant evidence under Bayes's Rule?"

Sorry. It still can't be done. You defeated yourself the instant you specified your argument's conclusion in advance.

This year, the Hadleyburg Trumpet sent out a 16-item questionnaire to all mayoral candidates, with questions like "Can you paint with all the colors of the wind?" and "Did you inhale?" Alas, the Trumpet's offices are destroyed by a meteorite before publication. It's a pity, since your own candidate, Mortimer Q. Snodgrass, compares well to his opponents on 15 out of 16 questions. The only sticking point was Question 11, "Are you now, or have you ever been, a supervillain?"

So you are tempted to publish the questionnaire as part of your own campaign literature... with the 11th question omitted, of course.

Which crosses the line between rationality and rationalization. It is no longer possible for the voters to condition on the facts alone; they must condition on the additional fact of their presentation, and infer the existence of hidden evidence.

Indeed, you crossed the line at the point where you considered whether the questionnaire was favorable or unfavorable to your candidate, before deciding whether to publish it. "What!" you cry. "A campaign should publish facts unfavorable to their candidate?" But put yourself in the shoes of a voter, still trying to select a candidate - why would you censor useful information? You wouldn't, if you were genuinely curious. If you were flowing forward from the evidence to an unknown choice of candidate, rather than flowing backward from a fixed candidate to determine the arguments.

A "logical" argument is one that follows from its premises. Thus the following argument is illogical:

- All rectangles are quadrilaterals.

- All squares are quadrilaterals.

- Therefore, all squares are rectangles.

This syllogism is not rescued from illogic by the truth of its premises or even the truth of its conclusion. It is worth distinguishing logical deductions from illogical ones, and to refuse to excuse them even if their conclusions happen to be true. For one thing, the distinction may affect how we revise our beliefs in light of future evidence. For another, sloppiness is habit-forming.

Above all, the syllogism fails to state the real explanation. Maybe all squares are rectangles, but, if so, it's not because they are both quadrilaterals. You might call it a hypocritical syllogism - one with a disconnect between its stated reasons and real reasons.

If you really want to present an honest, rational argument for your candidate, in a political campaign, there is only one way to do it:

- Before anyone hires you, gather up all the evidence you can about the different candidates.

- Make a checklist which you, yourself, will use to decide which candidate seems best.

- Process the checklist.

- Go to the winning candidate.

- Offer to become their campaign manager.

- When they ask for campaign literature, print out your checklist.

Only in this way can you offer a rational chain of argument, one whose bottom line was written flowing forward from the lines above it. Whatever actually decides your bottom line, is the only thing you can honestly write on the lines above.

Sequence: Against Rationalization

Rationalization

Avoiding Your Belief's Real Weak Points

Original with comments: A Rational Argument

We Change Our Minds Less Often Than We Think

Eliezer Yudkowsky, 03 October 2007 06:14PM

"Over the past few years, we have discreetly approached colleagues faced with a choice between job offers, and asked them to estimate the probability that they will choose one job over another. The average confidence in the predicted choice was a modest 66%, but only 1 of the 24 respondents chose the option to which he or she initially assigned a lower probability, yielding an overall accuracy rate of 96%."
-- Dale Griffin and Amos Tversky, "The Weighing of Evidence and the Determinants of Confidence." (Cognitive Psychology, 24, pp. 411-435.)

When I first read the words above - on August 1st, 2003, at around 3 o'clock in the afternoon - it changed the way I thought. I realized that once I could guess what my answer would be - once I could assign a higher probability to deciding one way than other - then I had, in all probability, already decided. We change our minds less often than we think. And most of the time we become able to guess what our answer will be within half a second of hearing the question.

How swiftly that unnoticed moment passes, when we can't yet guess what our answer will be; the tiny window of opportunity for intelligence to act. In questions of choice, as in questions of fact.

The principle of the bottom line is that only the actual causes of your beliefs determine your effectiveness as a rationalist. Once your belief is fixed, no amount of argument will alter the truth-value; once your decision is fixed, no amount of argument will alter the consequences.

You might think that you could arrive at a belief, or a decision, by non-rational means, and then try to justify it, and if you found you couldn't justify it, reject it.

But we change our minds less often - much less often - than we think.

I'm sure that you can think of at least one occasion in your life when you've changed your mind. We all can. How about all the occasions in your life when you didn't change your mind? Are you they as available, in your heuristic estimate of your competence?

Between hindsight bias, fake causality, positive bias, anchoring/priming, et cetera et cetera, and above all the dreaded confirmation bias, once an idea gets into your head, it's probably going to stay there.

Sequence: Seeing with Fresh Eyes

How to Seem (and Be) Deep

Hold Off On Proposing Solutions

Referenced by: Priming and Contamination • Hold Off On Proposing Solutions • Fake Justification • The Tragedy of Group Selectionism • Many Worlds, One Best Guess • Causality and Moral Responsibility • LA-602 vs. RHIC Review • The Genetic Fallacy • Anthropomorphic Optimism • Contaminated by Optimism • Harder Choices Matter Less • Crisis of Faith • Is That Your True Rejection? • That Crisis thing seems pretty useful

Original with comments: We Change Our Minds Less Often Than We Think

Avoiding Your Belief's Real Weak Points

Eliezer Yudkowsky, 05 October 2007 01:59AM

A few years back, my great-grandmother died, in her nineties, after a long, slow, and cruel disintegration. I never knew her as a person, but in my distant childhood, she cooked for her family; I remember her gefilte fish, and her face, and that she was kind to me. At her funeral, my grand-uncle, who had taken care of her for years, spoke: He said, choking back tears, that God had called back his mother piece by piece: her memory, and her speech, and then finally her smile; and that when God finally took her smile, he knew it wouldn't be long before she died, because it meant that she was almost entirely gone.

I heard this and was puzzled, because it was an unthinkably horrible thing to happen to anyone, and therefore I would not have expected my grand-uncle to attribute it to God. Usually, a Jew would somehow just-not-think-about the logical implication that God had permitted a tragedy. According to Jewish theology, God continually sustains the universe and chooses every event in it; but ordinarily, drawing logical implications from this belief is reserved for happier occasions. By saying "God did it!" only when you've been blessed with a baby girl, and just-not-thinking "God did it!" for miscarriages and stillbirths and crib deaths, you can build up quite a lopsided picture of your God's benevolent personality.

Hence I was surprised to hear my grand-uncle attributing the slow disintegration of his mother to a deliberate, strategically planned act of God. It violated the rules of religious self-deception as I understood them.

If I had noticed my own confusion, I could have made a successful surprising prediction. Not long afterward, my grand-uncle left the Jewish religion. (The only member of my extended family besides myself to do so, as far as I know.)

Modern Orthodox Judaism is like no other religion I have ever heard of, and I don't know how to describe it to anyone who hasn't been forced to study Mishna and Gemara. There is a tradition of questioning, but the kind of questioning... It would not be at all surprising to hear a rabbi, in his weekly sermon, point out the conflict between the seven days of creation and the 13.7 billion years since the Big Bang - because he thought he had a really clever explanation for it, involving three other Biblical references, a Midrash, and a half-understood article in Scientific American. In Orthodox Judaism you're allowed to notice inconsistencies and contradictions, but only for purposes of explaining them away, and whoever comes up with the most complicated explanation gets a prize.

There is a tradition of inquiry. But you only attack targets for purposes of defending them. You only attack targets you know you can defend.

In Modern Orthodox Judaism I have not heard much emphasis of the virtues of blind faith. You're allowed to doubt. You're just not allowed to successfully doubt.

I expect that the vast majority of educated Orthodox Jews have questioned their faith at some point in their lives. But the questioning probably went something like this: "According to the skeptics, the Torah says that the universe was created in seven days, which is not scientifically accurate. But would the original tribespeople of Israel, gathered at Mount Sinai, have been able to understand the scientific truth, even if it had been presented to them? Did they even have a word for 'billion'? It's easier to see the seven-days story as a metaphor - first God created light, which represents the Big Bang..."

Is this the weakest point at which to attack one's own Judaism? Read a bit further on in the Torah, and you can find God killing the first-born male children of Egypt to convince an unelected Pharaoh to release slaves who logically could have been teleported out of the country. An Orthodox Jew is most certainly familiar with this episode, because they are supposed to read through the entire Torah in synagogue once per year, and this event has an associated major holiday. The name "Passover" ("Pesach") comes from God passing over the Jewish households while killing every male firstborn in Egypt.

Modern Orthodox Jews are, by and large, kind and civilized people; far more civilized than the several editors of the Old Testament. Even the old rabbis were more civilized. There's a ritual in the Seder where you take ten drops of wine from your cup, one drop for each of the Ten Plagues, to emphasize the suffering of the Egyptians. (Of course, you're supposed to be sympathetic to the suffering of the Egyptians, but not so sympathetic that you stand up and say, "This is not right! It is wrong to do such a thing!") It shows an interesting contrast - the rabbis were sufficiently kinder than the compilers of the Old Testament that they saw the harshness of the Plagues. But Science was weaker in these days, and so rabbis could ponder the more unpleasant aspects of Scripture without fearing that it would break their faith entirely.

You don't even ask whether the incident reflects poorly on God, so there's no need to quickly blurt out "The ways of God are mysterious!" or "We're not wise enough to question God's decisions!" or "Murdering babies is okay when God does it!" That part of the question is just-not-thought-about.

The reason that educated religious people stay religious, I suspect, is that when they doubt, they are subconsciously very careful to attack their own beliefs only at the strongest points - places where they know they can defend. Moreover, places where rehearsing the standard defense will feel strengthening.

It probably feels really good, for example, to rehearse one's prescripted defense for "Doesn't Science say that the universe is just meaningless atoms bopping around?", because it confirms the meaning of the universe and how it flows from God, etc.. Much more comfortable to think about than an illiterate Egyptian mother wailing over the crib of her slaughtered son. Anyone who spontaneously thinks about the latter, when questioning their faith in Judaism, is really questioning it, and is probably not going to stay Jewish much longer.

My point here is not just to beat up on Orthodox Judaism. I'm sure that there's some reply or other for the Slaying of the Firstborn, and probably a dozen of them. My point is that, when it comes to spontaneous self-questioning, one is much more likely to spontaneously self-attack strong points with comforting replies to rehearse, then to spontaneously self-attack the weakest, most vulnerable points. Similarly, one is likely to stop at the first reply and be comforted, rather than further criticizing the reply. A better title than "Avoiding Your Belief's Real Weak Points" would be "Not Spontaneously Thinking About Your Belief's Most Painful Weaknesses".

More than anything, the grip of religion is sustained by people just-not-thinking-about the real weak points of their religion. I don't think this is a matter of training, but a matter of instinct. People don't think about the real weak points of their beliefs for the same reason they don't touch an oven's red-hot burners; it's painful.

To do better: When you're doubting one of your most cherished beliefs, close your eyes, empty your mind, grit your teeth, and deliberately think about whatever hurts the most. Don't rehearse standard objections whose standard counters would make you feel better. Ask yourself what smart people who disagree would say to your first reply, and your second reply. Whenever you catch yourself flinching away from an objection you fleetingly thought of, drag it out into the forefront of your mind. Punch yourself in the solar plexus. Stick a knife in your heart, and wiggle to widen the hole. In the face of the pain, rehearse only this:

What is true is already so.
Owning up to it doesn't make it worse.
Not being open about it doesn't make it go away.
And because it's true, it is what is there to be interacted with.
Anything untrue isn't there to be lived.
People can stand what is true,
for they are already enduring it.
-- Eugene Gendlin

Sequence: Against Rationalization

A Rational Argument

Motivated Stopping and Motivated Continuation

Referenced by: The Meditation on Curiosity • Why Are Individual IQ Differences OK? • Resist the Happy Death Spiral • Fake Utility Functions • Leave a Line of Retreat • Against Devil's Advocacy • Rebelling Within Nature • Could Anything Be Right? • Crisis of Faith • That Crisis thing seems pretty useful

Original with comments: Avoiding Your Belief's Real Weak Points

The Meditation on Curiosity

Eliezer Yudkowsky, 06 October 2007 12:26AM

"The first virtue is curiosity."
-- The Twelve Virtues of Rationality

As rationalists, we are obligated to criticize ourselves and question our beliefs... are we not?

Consider what happens to you, on a psychological level, if you begin by saying: "It is my duty to criticize my own beliefs." Roger Zelazny once distinguished between "wanting to be an author" versus "wanting to write". Mark Twain said: "A classic is something that everyone wants to have read and no one one wants to read." Criticizing yourself from a sense of duty leaves you wanting to have investigated, so that you'll be able to say afterward that your faith is not blind. This is not the same as wanting to investigate.

This can lead to motivated stopping of your investigation. You consider an objection, then a counterargument to that objection, then you stop there. You repeat this with several objections, until you feel that you have done your duty to investigate, and then you stop there. You have achieved your underlying psychological objective: to get rid of the cognitive dissonance that would result from thinking of yourself as a rationalist, and yet knowing that you had not tried to criticize your belief. You might call it purchase of rationalist satisfaction - trying to create a "warm glow" of discharged duty.

Afterward, your stated probability level will be high enough to justify your keeping the plans and beliefs you started with, but not so high as to evoke incredulity from yourself or other rationalists.

When you're really curious, you'll gravitate to inquiries that seem most promising of producing shifts in belief, or inquiries that are least like the ones you've tried before. Afterward, your probability distribution likely should not look like it did when you started out - shifts should have occurred, whether up or down; and either direction is equally fine to you, if you're genuinely curious.

Contrast this to the subconscious motive of keeping your inquiry on familiar ground, so that you can get your investigation over with quickly, so that you can have investigated, and restore the familiar balance on which your familiar old plans and beliefs are based.

As for what I think true curiosity should look like, and the power that it holds, I refer you to A Fable of Science and Politics. Each of the characters is intended to illustrate different lessons. Ferris, the last character, embodies the power of innocent curiosity: which is lightness, and an eager reaching forth for evidence.

Ursula K. LeGuin wrote: "In innocence there is no strength against evil. But there is strength in it for good." Innocent curiosity may turn innocently awry; and so the training of a rationalist, and its accompanying sophistication, must be dared as a danger if we want to become stronger. Nonetheless we can try to keep the lightness and the eager reaching of innocence.

As it is written in the Twelve Virtues:

"If in your heart you believe you already know, or if in your heart you do not wish to know, then your questioning will be purposeless and your skills without direction. Curiosity seeks to annihilate itself; there is no curiosity that does not want an answer."

There just isn't any good substitute for genuine curiosity. "A burning itch to know is higher than a solemn vow to pursue truth." But you can't produce curiosity just by willing it, any more than you can will your foot to feel warm when it feels cold. Sometimes, all we have is our mere solemn vows.

So what can you do with duty? For a start, we can try to take an interest in our dutiful investigations - keep a close eye out for sparks of genuine intrigue, or even genuine ignorance and a desire to resolve it. This goes right along with keeping a special eye out for possibilities that are painful, that you are flinching away from - it's not all negative thinking.

It should also help to meditate on Conservation of Expected Evidence. For every new point of inquiry, for every piece of unseen evidence that you suddenly look at, the expected posterior probability should equal your prior probability. In the microprocess of inquiry, your belief should always be evenly poised to shift in either direction. Not every point may suffice to blow the issue wide open - to shift belief from 70% to 30% probability - but if your current belief is 70%, you should be as ready to drop it to 69% as raising it to 71%. You should not think that you know which direction it will go in (on average), because by the laws of probability theory, if you know your destination, you are already there. If you can investigate honestly, so that each new point really does have equal potential to shift belief upward or downward, this may help to keep you interested or even curious about the microprocess of inquiry.

If the argument you are considering is not new, then why is your attention going here? Is this where you would look if you were genuinely curious? Are you subconsciously criticizing your belief at its strong points, rather than its weak points? Are you rehearsing the evidence?

If you can manage not to rehearse already known support, and you can manage to drop down your belief by one tiny bite at a time from the new evidence, you may even be able to relinquish the belief entirely - to realize from which quarter the winds of evidence are blowing against you.

Another restorative for curiosity is what I have taken to calling the Litany of Tarski, which is really a meta-litany that specializes for each instance (this is only appropriate). For example, if I am tensely wondering whether a locked box contains a diamond, then, rather than thinking about all the wonderful consequences if the box does contain a diamond, I can repeat the Litany of Tarski:

If the box contains a diamond,
I desire to believe that the box contains a diamond;
If the box does not contain a diamond,
I desire to believe that the box does not contain a diamond;
Let me not become attached to beliefs I may not want.

Then you should meditate upon the possibility that there is no diamond, and the subsequent advantage that will come to you if you believe there is no diamond, and the subsequent disadvantage if you believe there is a diamond. See also the Litany of Gendlin.

If you can find within yourself the slightest shred of true uncertainty, then guard it like a forester nursing a campfire. If you can make it blaze up into a flame of curiosity, it will make you light and eager, and give purpose to your questioning and direction to your skills.

Sequence: Letting Go

You Can Face Reality

Something to Protect

Referenced by: No One Knows What Science Doesn't Know • Motivated Stopping and Motivated Continuation • Fake Justification • An Alien God • Resist the Happy Death Spiral • Fake Utility Functions • Every Cause Wants To Be A Cult • Two Cult Koans • Zen and the Art of Rationality • Cultish Countercultishness • Leave a Line of Retreat • Dissolving the Question • Zombies! Zombies? • Against Devil's Advocacy • LA-602 vs. RHIC Review • My Kind of Reflection • Crisis of Faith • Belief in Self-Deception • The Sacred Mundane • That Crisis thing seems pretty useful • You're Entitled to Arguments, But Not (That Particular) Proof

Original with comments: The Meditation on Curiosity

Singlethink

Eliezer Yudkowsky, 06 October 2007 07:24PM

I remember the exact moment when I began my journey as a rationalist.

It was not while reading Surely You're Joking, Mr. Feynman or any existing work upon rationality; for these I simply accepted as obvious. The journey begins when you see a great flaw in your existing art, and discover a drive to improve, to create new skills beyond the helpful but inadequate ones you found in books.

In the last moments of my first life, I was fifteen years old, and rehearsing a pleasantly self-righteous memory of a time when I was much younger. My memories this far back are vague; I have a mental image, but I don't remember how old I was exactly. I think I was six or seven, and that the original event happened during summer camp.

What happened originally was that a camp counselor, a teenage male, got us much younger boys to form a line, and proposed the following game: the boy at the end of the line would crawl through our legs, and we would spank him as he went past, and then it would be the turn of the next eight-year-old boy at the end of the line. (Maybe it's just that I've lost my youthful innocence, but I can't help but wonder...) I refused to play this game, and was told to go sit in the corner.

This memory - of refusing to spank and be spanked - came to symbolize to me that even at this very early age I had refused to take joy in hurting others. That I would not purchase a spank on another's butt, at the price of a spank on my own; would not pay in hurt for the opportunity to inflict hurt. I had refused to play a negative-sum game.

And then, at the age of fifteen, I suddenly realized that it wasn't true. I hadn't refused out of a principled stand against negative-sum games. I found out about the Prisoner's Dilemma pretty early in life, but not at the age of seven. I'd refused simply because I didn't want to get hurt, and standing in the corner was an acceptable price to pay for not getting hurt.

More importantly, I realized that I had always known this - that the real memory had always been lurking in a corner of my mind, my mental eye glancing at it for a fraction of a second and then looking away.

In my very first step along the Way, I caught the feeling - generalized over the subjective experience - and said, "So that's what it feels like to shove an unwanted truth into the corner of my mind! Now I'm going to notice every time I do that, and clean out all my corners!"

This discipline I named singlethink, after Orwell's doublethink. In doublethink, you forget, and then forget you have forgotten. In singlethink, you notice you are forgetting, and then you remember. You hold only a single non-contradictory thought in your mind at once.

"Singlethink" was the first new rationalist skill I created, which I had not read about in books. I doubt that it is original in the sense of academic priority, but this is thankfully not required.

Oh, and my fifteen-year-old self liked to name things.

The terrifying depths of the confirmation bias go on and on. Not forever, for the brain is of finite complexity, but long enough that it feels like forever. You keep on discovering (or reading about) new mechanisms by which your brain shoves things out of the way.

But my young self swept out quite a few corners with that first broom.

Sequence: Against Doublethink

Belief in Belief

Doublethink (Choosing to be Biased)

Referenced by: Crisis of Faith • That Crisis thing seems pretty useful

Original with comments: Singlethink

No One Can Exempt You From Rationality's Laws

Eliezer Yudkowsky, 07 October 2007 05:24PM

Traditional Rationality is phrased in terms of social rules, with violations interpretable as cheating - as defections from cooperative norms. If you want me to accept a belief from you, you are obligated to provide me with a certain amount of evidence. If you try to get out of it, we all know you're cheating on your obligation. A theory is obligated to make bold predictions for itself, not just steal predictions that other theories have labored to make. A theory is obligated to expose itself to falsification - if it tries to duck out, that's like trying to duck out of a fearsome initiation ritual; you must pay your dues.

Traditional Rationality is phrased similarly to the customs that govern human societies, which makes it easy to pass on by word of mouth. Humans detect social cheating with much greater reliability than isomorphic violations of abstract logical rules. But viewing rationality as a social obligation gives rise to some strange ideas.

For example, one finds religious people defending their beliefs by saying, "Well, you can't justify your belief in science!" In other words, "How dare you criticize me for having unjustified beliefs, you hypocrite! You're doing it too!"

To Bayesians, the brain is an engine of accuracy: it processes and concentrates entangled evidence into a map that reflects the territory. The principles of rationality are laws in the same sense as the second law of thermodynamics: obtaining a reliable belief requires a calculable amount of entangled evidence, just as reliably cooling the contents of a refrigerator requires a calculable minimum of free energy.

In principle, the laws of physics are time-reversible, so there's an infinitesimally tiny probability - indistinguishable from zero to all but mathematicians - that a refrigerator will spontaneously cool itself down while generating electricity. There's a slightly larger infinitesimal chance that you could accurately draw a detailed street map of New York without ever visiting, sitting in your living room with your blinds closed and no Internet connection. But I wouldn't hold your breath.

Before you try mapping an unseen territory, pour some water into a cup at room temperature and wait until it spontaneously freezes before proceeding. That way you can be sure the general trick - ignoring infinitesimally tiny probabilities of success - is working properly. You might not realize directly that your map is wrong, especially if you never visit New York; but you can see that water doesn't freeze itself.

If the rules of rationality are social customs, then it may seem to excuse behavior X if you point out that others are doing the same thing. It wouldn't be fair to demand evidence from you, if we can't provide it ourselves. We will realize that none of us are better than the rest, and we will relent and mercifully excuse you from your social obligation to provide evidence for your belief. And we'll all live happily ever afterward in liberty, fraternity, and equality.

If the rules of rationality are mathematical laws, then trying to justify evidence-free belief by pointing to someone else doing the same thing, will be around as effective as listing 30 reasons why you shouldn't fall off a cliff. Even if we all vote that it's unfair for your refrigerator to need electricity, it still won't run (with probability 1). Even if we all vote that you shouldn't have to visit New York, the map will still be wrong. Lady Nature is famously indifferent to such pleading, and so is Lady Math.

So - to shift back to the social language of Traditional Rationality - don't think you can get away with claiming that it's okay to have arbitrary beliefs about XYZ, because other people have arbitrary beliefs too. If two parties to a contract both behave equally poorly, a human judge may decide to impose penalties on neither. But if two engineers design their engines equally poorly, neither engine will work. One design error cannot excuse another. Even if I'm doing XYZ wrong, it doesn't help you, or exempt you from the rules; it just means we're both screwed.

As a matter of human law in liberal democracies, everyone is entitled to their own beliefs. As a matter of Nature's law, you are not entitled to accuracy. We don't arrest people for believing weird things, at least not in the wiser countries. But no one can revoke the law that you need evidence to generate accurate beliefs. Not even a vote of the whole human species can obtain mercy in the court of Nature.

Physicists don't decide the laws of physics, they just guess what they are. Rationalists don't decide the laws of rationality, we just guess what they are. You cannot "rationalize" anything that is not rational to begin with. If by dint of extraordinary persuasiveness you convince all the physicists in the world that you are exempt from the law of gravity, and you walk off a cliff, you'll fall. Even saying "We don't decide" is too anthropomorphic. There is no higher authority that could exempt you. There is only cause and effect.

Remember this, when you plead to be excused just this once. We can't excuse you. It isn't up to us.

Sequence: Letting Go

Something to Protect

Leave a Line of Retreat

Referenced by: A Priori • Hold Off On Proposing Solutions

Original with comments: No One Can Exempt You From Rationality's Laws

A Priori

Eliezer Yudkowsky, 08 October 2007 09:02PM

See also: Comments on "How to Convince Me That 2 + 2 = 3"

Traditional Rationality is phrased as social rules, with violations interpretable as cheating: if you break the rules and no one else is doing so, you're the first to defect - making you a bad, bad person. To Bayesians, the brain is an engine of accuracy: if you violate the laws of rationality, the engine doesn't run, and this is equally true whether anyone else breaks the rules or not.

Consider the problem of Occam's Razor, as confronted by Traditional philosophers. If two hypotheses fit the same observations equally well, why believe the simpler one is more likely to be true?

You could argue that Occam's Razor has worked in the past, and is therefore likely to continue to work in the future. But this, itself, appeals to a prediction from Occam's Razor. "Occam's Razor works up to October 8th, 2007 and then stops working thereafter" is more complex, but it fits the observed evidence equally well.

You could argue that Occam's Razor is a reasonable distribution on prior probabilities. But what is a "reasonable" distribution? Why not label "reasonable" a very complicated prior distribution, which makes Occam's Razor work in all observed tests so far, but generates exceptions in future cases?

Indeed, it seems there is no way to justify Occam's Razor except by appealing to Occam's Razor, making this argument unlikely to convince any judge who does not already accept Occam's Razor. (What's special about the words I italicized?)

If you are a philosopher whose daily work is to write papers, criticize other people's papers, and respond to others' criticisms of your own papers, then you may look at Occam's Razor and shrug. Here is an end to justifying, arguing and convincing. You decide to call a truce on writing papers; if your fellow philosophers do not demand justification for your un-arguable beliefs, you will not demand justification for theirs. And as the symbol of your treaty, your white flag, you use the phrase "a priori truth".

But to a Bayesian, in this era of cognitive science and evolutionary biology and Artificial Intelligence, saying "a priori" doesn't explain why the brain-engine runs. If the brain has an amazing "a priori truth factory" that works to produce accurate beliefs, it makes you wonder why a thirsty hunter-gatherer can't use the "a priori truth factory" to locate drinkable water. It makes you wonder why eyes evolved in the first place, if there are ways to produce accurate beliefs without looking at things.

James R. Newman said: "The fact that one apple added to one apple invariably gives two apples helps in the teaching of arithmetic, but has no bearing on the truth of the proposition that 1 + 1 = 2." The Internet Encyclopedia of Philosophy defines "a priori" propositions as those knowable independently of experience. Wikipedia quotes Hume: Relations of ideas are "discoverable by the mere operation of thought, without dependence on what is anywhere existent in the universe." You can see that 1 + 1 = 2 just by thinking about it, without looking at apples.

But in this era of neurology, one ought to be aware that thoughts are existent in the universe; they are identical to the operation of brains. Material brains, real in the universe, composed of quarks in a single unified mathematical physics whose laws draw no border between the inside and outside of your skull.

When you add 1 + 1 and get 2 by thinking, these thoughts are themselves embodied in flashes of neural patterns. In principle, we could observe, experientially, the exact same material events as they occurred within someone else's brain. It would require some advances in computational neurobiology and brain-computer interfacing, but in principle, it could be done. You could see someone else's engine operating materially, through material chains of cause and effect, to compute by "pure thought" that 1 + 1 = 2. How is observing this pattern in someone else's brain any different, as a way of knowing, from observing your own brain doing the same thing? When "pure thought" tells you that 1 + 1 = 2, "independently of any experience or observation", you are, in effect, observing your own brain as evidence.

If this seems counterintuitive, try to see minds/brains as engines - an engine that collides the neural pattern for 1 and the neural pattern for 1 and gets the neural pattern for 2. If this engine works at all, then it should have the same output if it observes (with eyes and retina) a similar brain-engine carrying out a similar collision, and copies into itself the resulting pattern. In other words, for every form of a priori knowledge obtained by "pure thought", you are learning exactly the same thing you would learn if you saw an outside brain-engine carrying out the same pure flashes of neural activation. The engines are equivalent, the bottom-line outputs are equivalent, the belief-entanglements are the same.

There is nothing you can know "a priori", which you could not know with equal validity by observing the chemical release of neurotransmitters within some outside brain. What do you think you are, dear reader?

This is why you can predict the result of adding 1 apple and 1 apple by imagining it first in your mind, or punch "3 x 4" into a calculator to predict the result of imagining 4 rows with 3 apples per row. You and the apple exist within a boundary-less unified physical process, and one part may echo another.

Are the sort of neural flashes that philosophers label "a priori beliefs", arbitrary? Many AI algorithms function better with "regularization" that biases the solution space toward simpler solutions. But the regularized algorithms are themselves more complex; they contain an extra line of code (or 1000 extra lines) compared to unregularized algorithms. The human brain is biased toward simplicity, and we think more efficiently thereby. If you press the Ignore button at this point, you're left with a complex brain that exists for no reason and works for no reason. So don't try to tell me that "a priori" beliefs are arbitrary, because they sure aren't generated by rolling random numbers. (What does the adjective "arbitrary" mean, anyway?)

You can't excuse calling a proposition "a priori" by pointing out that other philosophers are having trouble justifying their propositions. If a philosopher fails to explain something, this fact cannot supply electricity to a refrigerator, nor act as a magical factory for accurate beliefs. There's no truce, no white flag, until you understand why the engine works.

If you clear your mind of justification, of argument, then it seems obvious why Occam's Razor works in practice: we live in a simple world, a low-entropy universe in which there are short explanations to be found. "But," you cry, "why is the universe itself orderly?" This I do not know, but it is what I see as the next mystery to be explained. This is not the same question as "How do I argue Occam's Razor to a hypothetical debater who has not already accepted it?"

Perhaps you cannot argue anything to a hypothetical debater who has not accepted Occam's Razor, just as you cannot argue anything to a rock. A mind needs a certain amount of dynamic structure to be an argument-acceptor. If a mind doesn't implement Modus Ponens, it can accept "A" and "A->B" all day long without ever producing "B". How do you justify Modus Ponens to a mind that hasn't accepted it? How do you argue a rock into becoming a mind?

Brains evolved from non-brainy matter by natural selection; they were not justified into existence by arguing with an ideal philosophy student of perfect emptiness. This does not make our judgments meaningless. A brain-engine can work correctly, producing accurate beliefs, even if it was merely built - by human hands or cumulative stochastic selection pressures - rather than argued into existence. But to be satisfied by this answer, one must see rationality in terms of engines, rather than arguments.

Referenced by: The Allais Paradox • The "Intuitions" Behind "Utilitarianism" • Newcomb's Problem and Regret of Rationality • Mind Projection Fallacy • No Universally Compelling Arguments • Moral Complexities • Where Recursive Justification Hits Bottom • Probability is Subjectively Objective • Abstracted Idealized Dynamics

Original with comments: A Priori

Priming and Contamination

Eliezer Yudkowsky, 10 October 2007 02:23AM

Suppose you ask subjects to press one button if a string of letters forms a word, and another button if the string does not form a word. (E.g., "banack" vs. "banner".) Then you show them the string "water". Later, they will more quickly identify the string "drink" as a word. This is known as "cognitive priming"; this particular form would be "semantic priming" or "conceptual priming".

The fascinating thing about priming is that it occurs at such a low level - priming speeds up identifying letters as forming a word, which one would expect to take place before you deliberate on the word's meaning.

Priming also reveals the massive parallelism of spreading activation: if seeing "water" activates the word "drink", it probably also activates "river", or "cup", or "splash"... and this activation spreads, from the semantic linkage of concepts, all the way back to recognizing strings of letters.

Priming is subconscious and unstoppable, an artifact of the human neural architecture. Trying to stop yourself from priming is like trying to stop the spreading activation of your own neural circuits. Try to say aloud the color - not the meaning, but the color - of the following letter-string: "GREEN"

In Mussweiler and Strack (2000), subjects were asked the anchoring question: "Is the annual mean temperature in Germany higher or lower than 5 Celsius / 20 Celsius?" Afterward, on a word-identification task, subjects presented with the 5 Celsius anchor were faster on identifying words like "cold" and "snow", while subjects with the high anchor were faster to identify "hot" and "sun". This shows a non-adjustment mechanism for anchoring: priming compatible thoughts and memories.

The more general result is that completely uninformative, known false, or totally irrelevant "information" can influence estimates and decisions. In the field of heuristics and biases, this more general phenomenon is known as contamination. (Chapman and Johnson 2002.)

Early research in heuristics and biases discovered anchoring effects, such as subjects giving lower (higher) estimates of the percentage of UN countries found within Africa, depending on whether they were first asked if the percentage was more or less than 10 (65). This effect was originally attributed to subjects adjusting from the anchor as a starting point, stopping as soon as they reached a plausible value, and under-adjusting because they were stopping at one end of a confidence interval. (Tversky and Kahneman 1974.)

Tversky and Kahneman's early hypothesis still appears to be the correct explanation in some circumstances, notably when subjects generate the initial estimate themselves (Epley and Gilovich 2001). But modern research seems to show that most anchoring is actually due to contamination, not sliding adjustment. (Hat tip for Unnamed for reminding me of this - I'd read the Epley/Gilovich paper years ago, as a chapter in Heuristics and Biases, but forgotten it.)

Your grocery store probably has annoying signs saying "Limit 12 per customer" or "5 for $10". Are these signs effective at getting customers to buy in larger quantities? You probably think you're not influenced. But someone must be, because these signs have been shown to work, which is why stores keep putting them up. (Wansink et. al. 1998.)

Yet the most fearsome aspect of contamination is that it serves as yet another of the thousand faces of confirmation bias. Once an idea gets into your head, it primes information compatible with it - and thereby ensures its continued existence. Never mind the selection pressures for winning political arguments; confirmation bias is built directly into our hardware, associational networks priming compatible thoughts and memories. An unfortunate side effect of our existence as neural creatures.

A single fleeting image can be enough to prime associated words for recognition. Don't think it takes anything more to set confirmation bias in motion. All it takes is that one quick flash, and the bottom line is already decided, for we change our minds less often than we think...

Chapman, G.B. and Johnson, E.J. 2002. Incorporating the irrelevant: Anchors in judgments of belief and value. In Gilovich et. al. (2003).

Epley, N., & Gilovich, T. (2001). Putting adjustment back in the anchoring and adjustment heuristic: Differential processing of self-generated and experimenter-provided anchors. Psychological Science, 12, 391–396.

Mussweiler, T. and Strack, F. Comparing is believing: a selective accessibility model of judgmental anchoring. European Review of Social Psychology, 10, 135-167.

Tversky, A. and Kahneman, D. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185: 251-284.

Wansink, B., Kent, R.J. and Hoch, S.J. 1998. An Anchoring and Adjustment Model of Purchase Quantity Decisions. Journal of Marketing Research, 35(February): 71-81.

Sequence: Seeing with Fresh Eyes

Anchoring and Adjustment

Do We Believe Everything We're Told?

Referenced by: Do We Believe Everything We're Told? • The Logical Fallacy of Generalization from Fictional Evidence • Self-Anchoring • Contaminated by Optimism

Original with comments: Priming and Contamination

Do We Believe Everything We're Told?

Eliezer Yudkowsky, 10 October 2007 11:52PM

Some early experiments on anchoring and adjustment tested whether distracting the subjects - rendering subjects cognitively "busy" by asking them to keep a lookout for "5" in strings of numbers, or some such - would decrease adjustment, and hence increase the influence of anchors. Most of the experiments seemed to bear out the idea that cognitive busyness increased anchoring, and more generally contamination.

Looking over the accumulating experimental results - more and more findings of contamination, exacerbated by cognitive busyness - Daniel Gilbert saw a truly crazy pattern emerging: Do we believe everything we're told?

One might naturally think that on being told a proposition, we would first comprehend what the proposition meant, then consider the proposition, and finally accept or reject it. This obvious-seeming model of cognitive process flow dates back to Descartes. But Descartes's rival, Spinoza, disagreed; Spinoza suggested that we first passively accept a proposition in the course of comprehending it, and only afterward actively disbelieve propositions which are rejected by consideration.

Over the last few centuries, philosophers pretty much went along with Descartes, since his view seemed more, y'know, logical and intuitive. But Gilbert saw a way of testing Descartes's and Spinoza's hypotheses experimentally.

If Descartes is right, then distracting subjects should interfere with both accepting true statements and rejecting false statements. If Spinoza is right, then distracting subjects should cause them to remember false statements as being true, but should not cause them to remember true statements as being false.

Gilbert, Krull, and Malone (1990) bears out this result, showing that, among subjects presented with novel statements labeled TRUE or FALSE, distraction had no effect on identifying true propositions (55% success for uninterrupted presentations, vs. 58% when interrupted); but did affect identifying false propositions (55% success when uninterrupted, vs. 35% when interrupted).

A much more dramatic illustration was produced in followup experiments by Gilbert, Tafarodi and Malone (1993). Subjects read aloud crime reports crawling across a video monitor, in which the color of the text indicated whether a particular statement was true or false. Some reports contained false statements that exacerbated the severity of the crime, other reports contained false statements that extenuated (excused) the crime. Some subjects also had to pay attention to strings of digits, looking for a "5", while reading the crime reports - this being the distraction task to create cognitive busyness. Finally, subjects had to recommend the length of prison terms for each criminal, from 0 to 20 years.

Subjects in the cognitively busy condition recommended an average of 11.15 years in prison for criminals in the "exacerbating" condition, that is, criminals whose reports contained labeled false statements exacerbating the severity of the crime. Busy subjects recommended an average of 5.83 years in prison for criminals whose reports contained labeled false statements excusing the crime. This nearly twofold difference was, as you might suspect, statistically significant.

Non-busy participants read exactly the same reports, with the same labels, and the same strings of numbers occasionally crawling past, except that they did not have to search for the number "5". Thus, they could devote more attention to "unbelieving" statements labeled false. These non-busy participants recommended 7.03 years versus 6.03 years for criminals whose reports falsely exacerbated or falsely excused.

Gilbert, Tafarodi and Malone's paper was entitled "You Can't Not Believe Everything You Read".

This suggests - to say the very least - that we should be more careful when we expose ourselves to unreliable information, especially if we're doing something else at the time. Be careful when you glance at that newspaper in the supermarket.

PS: According to an unverified rumor I just made up, people will be less skeptical of this blog post because of the distracting color changes.

Gilbert, D. 2002. Inferential correction. In Heuristics and biases: The psychology of intuitive judgment. You recognize this citation by now, right?

Gilbert, D., Krull, D. and Malone, P. 1990. Unbelieving the unbelievable: Some problems in the rejection of false information. Journal of Personality and Social PSychology, 59(4), 601-613.

Gilbert, D., Tafarodi, R. and Malone, P. 1993. You can't not believe everything you read. Journal of Personality and Social Psychology, 65(2), 221-233.

Sequence: Seeing with Fresh Eyes

Priming and Contamination

Cached Thoughts

Referenced by: The Logical Fallacy of Generalization from Fictional Evidence • Beware of Stephen J. Gould • The Tragedy of Group Selectionism • Do Fandoms Need Awfulness?

Original with comments: Do We Believe Everything We're Told?

Cached Thoughts

Eliezer Yudkowsky, 11 October 2007 11:46PM

One of the single greatest puzzles about the human brain is how the damn thing works at all when most neurons fire 10-20 times per second, or 200Hz tops. In neurology, the "hundred-step rule" is that any postulated operation has to complete in at most 100 sequential steps - you can be as parallel as you like, but you can't postulate more than 100 (preferably less) neural spikes one after the other.

Can you imagine having to program using 100Hz CPUs, no matter how many of them you had? You'd also need a hundred billion processors just to get anything done in realtime.

If you did need to write realtime programs for a hundred billion 100Hz processors, one trick you'd use as heavily as possible is caching. That's when you store the results of previous operations and look them up next time, instead of recomputing them from scratch. And it's a very neural idiom - recognition, association, completing the pattern.

It's a good guess that the actual majority of human cognition consists of cache lookups.

This thought does tend to go through my mind at certain times.

There was a wonderfully illustrative story which I thought I had bookmarked, but couldn't re-find: it was the story of a man whose know-it-all neighbor had once claimed in passing that the best way to remove a chimney from your house was to knock out the fireplace, wait for the bricks to drop down one level, knock out those bricks, and repeat until the chimney was gone. Years later, when the man wanted to remove his own chimney, this cached thought was lurking, waiting to pounce...

As the man noted afterward - you can guess it didn't go well - his neighbor was not particularly knowledgeable in these matters, not a trusted source. If he'd questioned the idea, he probably would have realized it was a poor one. Some cache hits we'd be better off recomputing. But the brain completes the pattern automatically - and if you don't consciously realize the pattern needs correction, you'll be left with a completed pattern.

I suspect that if the thought had occurred to the man himself - if he'd personally had this bright idea for how to remove a chimney - he would have examined the idea more critically. But if someone else has already thought an idea through, you can save on computing power by caching their conclusion - right?

In modern civilization particularly, no one can think fast enough to think their own thoughts. If I'd been abandoned in the woods as an infant, raised by wolves or silent robots, I would scarcely be recognizable as human. No one can think fast enough to recapitulate the wisdom of a hunter-gatherer tribe in one lifetime, starting from scratch. As for the wisdom of a literate civilization, forget it.

But the flip side of this is that I continually see people who aspire to critical thinking, repeating back cached thoughts which were not invented by critical thinkers.

A good example is the skeptic who concedes, "Well, you can't prove or disprove a religion by factual evidence." As I have pointed out elsewhere, this is simply false as probability theory. And it is also simply false relative to the real psychology of religion - a few centuries ago, saying this would have gotten you burned at the stake. A mother whose daughter has cancer prays, "God, please heal my daughter", not, "Dear God, I know that religions are not allowed to have any falsifiable consequences, which means that you can't possibly heal my daughter, so... well, basically, I'm praying to make myself feel better, instead of doing something that could actually help my daughter."

But people read "You can't prove or disprove a religion by factual evidence," and then, the next time they see a piece of evidence disproving a religion, their brain completes the pattern. Even some atheists repeat this absurdity without hesitation. If they'd thought of the idea themselves, rather than hearing it from someone else, they would have been more skeptical.

Death: complete the pattern: "Death gives meaning to life."

It's frustrating, talking to good and decent folk - people who would never in a thousand years spontaneously think of wiping out the human species - raising the topic of existential risk, and hearing them say, "Well, maybe the human species doesn't deserve to survive." They would never in a thousand years shoot their own child, who is a part of the human species, but the brain completes the pattern.

What patterns are being completed, inside your mind, that you never chose to be there?

Rationality: complete the pattern: "Love isn't rational."

If this idea had suddenly occurred to you personally, as an entirely new thought, how would you examine it critically? I know what I would say, but what would you? It can be hard to see with fresh eyes. Try to keep your mind from completing the pattern in the standard, unsurprising, already-known way. It may be that there is no better answer than the standard one, but you can't think about the answer until you can stop your brain from filling in the answer automatically.

Now that you've read this blog post, the next time you hear someone unhesitatingly repeating a meme you think is silly or false, you'll think, "Cached thoughts." My belief is now there in your mind, waiting to complete the pattern. But is it true? Don't let your mind complete the pattern! Think!

Sequence: Seeing with Fresh Eyes

Do We Believe Everything We're Told?

The "Outside the Box" Box

Referenced by: The "Outside the Box" Box • Original Seeing • How to Seem (and Be) Deep • The Logical Fallacy of Generalization from Fictional Evidence • The Simple Math of Everything • Resist the Happy Death Spiral • Uncritical Supercriticality • False Laughter • Replace the Symbol with the Substance • Fake Reductionism • Is Humanism A Religion-Substitute? • Heat vs. Motion • Where Physics Meets Experience • Einstein's Superpowers • A Premature Word on AI • Timeless Identity • What Would You Do Without Morality? • The Genetic Fallacy • Fundamental Doubts • Raised in Technophilia • My Naturalistic Awakening • Crisis of Faith • Dark Side Epistemology • Logical or Connectionist AI? • Hard Takeoff • Not Taking Over the World • Prolegomena to a Theory of Fun • Building Weirdtopia • (Moral) Truth in Fiction? • Raising the Sanity Waterline • Church vs. Taskforce • That Crisis thing seems pretty useful • The End (of Sequences)

Original with comments: Cached Thoughts

The "Outside the Box" Box

Eliezer Yudkowsky, 12 October 2007 10:50PM

Whenever someone exhorts you to "think outside the box", they usually, for your convenience, point out exactly where "outside the box" is located. Isn't it funny how nonconformists all dress the same...

In Artificial Intelligence, everyone outside the field has a cached result for brilliant new revolutionary AI idea - neural networks, which work just like the human brain! New AI Idea: complete the pattern: "Logical AIs, despite all the big promises, have failed to provide real intelligence for decades - what we need are neural networks!"

This cached thought has been around for three decades. Still no general intelligence. But, somehow, everyone outside the field knows that neural networks are the Dominant-Paradigm-Overthrowing New Idea, ever since backpropagation was invented in the 1970s. Talk about your aging hippies.

Nonconformist images, by their nature, permit no departure from the norm. If you don't wear black, how will people know you're a tortured artist? How will people recognize uniqueness if you don't fit the standard pattern for what uniqueness is supposed to look like? How will anyone recognize you've got a revolutionary AI concept, if it's not about neural networks?

Another example of the same trope is "subversive" literature, all of which sounds the same, backed up by a tiny defiant league of rebels who control the entire English Department. As Anonymous asks on Scott Aaronson's blog:

"Has any of the subversive literature you've read caused you to modify any of your political views?"

Or as Lizard observes:

"Revolution has already been televised. Revolution has been *merchandised*. Revolution is a commodity, a packaged lifestyle, available at your local mall. $19.95 gets you the black mask, the spray can, the "Crush the Fascists" protest sign, and access to your blog where you can write about the police brutality you suffered when you chained yourself to a fire hydrant. Capitalism has learned how to sell anti-capitalism."

Many in Silicon Valley have observed that the vast majority of venture capitalists at any given time are all chasing the same Revolutionary Innovation, and it's the Revolutionary Innovation that IPO'd six months ago. This is an especially crushing observation in venture capital, because there's a direct economic motive to not follow the herd - either someone else is also developing the product, or someone else is bidding too much for the startup. Steve Jurvetson once told me that at Draper Fisher Jurvetson, only two partners need to agree in order to fund any startup up to $1.5 million. And if all the partners agree that something sounds like a good idea, they won't do it. If only grant committees were this sane.

The problem with originality is that you actually have to think in order to attain it, instead of letting your brain complete the pattern. There is no conveniently labeled "Outside the Box" to which you can immediately run off. There's an almost Zen-like quality to it - like the way you can't teach satori in words because satori is the experience of words failing you. The more you try to follow the Zen Master's instructions in words, the further you are from attaining an empty mind.

There is a reason, I think, why people do not attain novelty by striving for it. Properties like truth or good design are independent of novelty: 2 + 2 = 4, yes, really, even though this is what everyone else thinks too. People who strive to discover truth or to invent good designs, may in the course of time attain creativity. Not every change is an improvement, but every improvement is a change.

Every improvement is a change, but not every change is an improvement. The one who says, "I want to build an original mousetrap!", and not, "I want to build an optimal mousetrap!", nearly always wishes to be perceived as original. "Originality" in this sense is inherently social, because it can only be determined by comparison to other people. So their brain simply completes the standard pattern for what is perceived as "original", and their friends nod in agreement and say it is subversive.

Business books always tell you, for your convenience, where your cheese has been moved to. Otherwise the readers would be left around saying, "Where is this 'Outside the Box' I'm supposed to go?"

Actually thinking, like satori, is a wordless act of mind.

The eminent philosophers of Monty Python said it best of all:

Sequence: Seeing with Fresh Eyes

Cached Thoughts

Original Seeing

Referenced by: How to Seem (and Be) Deep • Artificial Addition • Truly Part Of You • Resist the Happy Death Spiral • Lonely Dissent • Stop Voting For Nincompoops • The Opposite Sex • Competent Elites • Logical or Connectionist AI?

Original with comments: The "Outside the Box" Box

Original Seeing

Eliezer Yudkowsky, 14 October 2007 04:38AM

Followup to: Cached Thoughts, The Virtue of Narrowness

Since Robert Pirsig put this very well, I'll just copy down what he said. I don't know if this story is based on reality or not, but either way, it's true.

He'd been having trouble with students who had nothing to say. At first he thought it was laziness but later it became apparent that it wasn't. They just couldn't think of anything to say.
One of them, a girl with strong-lensed glasses, wanted to write a five-hundred word essay about the United States. He was used to the sinking feeling that comes from statements like this, and suggested without disparagement that she narrow it down to just Bozeman.
When the paper came due she didn't have it and was quite upset. She had tried and tried but she just couldn't think of anything to say.
It just stumped him. Now he couldn't think of anything to say. A silence occurred, and then a peculiar answer: "Narrow it down to the main street of Bozeman." It was a stroke of insight.
She nodded dutifully and went out. But just before her next class she came back in real distress, tears this time, distress that had obviously been there for a long time. She still couldn't think of anything to say, and couldn't understand why, if she couldn't think of anything about all of Bozeman, she should be able to think of something about just one street.

He was furious. "You're not looking!" he said. A memory came back of his own dismissal from the University for having too much to say. For every fact there is an infinity of hypotheses. The more you look the more you see. She really wasn't looking and yet somehow didn't understand this.
He told her angrily, "Narrow it down to the front of one building on the main street of Bozeman. The Opera House. Start with the upper left-hand brick."
Her eyes, behind the thick-lensed glasses, opened wide.
She came in the next class with a puzzled look and handed him a five-thousand-word essay on the front of the Opera House on the main street of Bozeman, Montana. "I sat in the hamburger stand across the street," she said, "and started writing about the first brick, and the second brick, and then by the third brick it all started to come and I couldn't stop. They thought I was crazy, and they kept kidding me, but here it all is. I don't understand it."
Neither did he, but on long walks through the streets of town he thought about it and concluded she was evidently stopped with the same kind of blockage that had paralyzed him on his first day of teaching. She was blocked because she was trying to repeat, in her writing, things she had already heard, just as on the first day he had tried to repeat things he had already decided to say. She couldn't think of anything to write about Bozeman because she couldn't recall anything she had heard worth repeating. She was strangely unaware that she could look and see freshly for herself, as she wrote, without primary regard for what had been said before. The narrowing down to one brick destroyed the blockage because it was so obvious she had to do some original and direct seeing.

-- Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance

Sequence: Seeing with Fresh Eyes

The "Outside the Box" Box

The Logical Fallacy of Generalization from Fictional Evidence

Referenced by: How to Seem (and Be) Deep • The Logical Fallacy of Generalization from Fictional Evidence • Double Illusion of Transparency • Replace the Symbol with the Substance • Crisis of Faith • That Crisis thing seems pretty useful • My Way

Original with comments: Original Seeing

How to Seem (and Be) Deep

Eliezer Yudkowsky, 14 October 2007 06:13PM

I recently attended a discussion group whose topic, at that session, was Death. It brought out deep emotions. I think that of all the Silicon Valley lunches I've ever attended, this one was the most honest; people talked about the death of family, the death of friends, what they thought about their own deaths. People really listened to each other. I wish I knew how to reproduce those conditions reliably.

I was the only transhumanist present, and I was extremely careful not to be obnoxious about it. ("A fanatic is someone who can't change his mind and won't change the subject." I endeavor to at least be capable of changing the subject.) Unsurprisingly, people talked about the meaning that death gives to life, or how death is truly a blessing in disguise. But I did, very cautiously, explain that transhumanists are generally positive on life but thumbs down on death.

Afterward, several people came up to me and told me I was very "deep". Well, yes, I am, but this got me thinking about what makes people seem deep.

At one point in the discussion, a woman said that thinking about death led her to be nice to people because, who knows, she might not see them again. "When I have a nice thing to say about someone," she said, "now I say it to them right away, instead of waiting."

"That is a beautiful thought," I said, "and even if someday the threat of death is lifted from you, I hope you will keep on doing it -"

Afterward, this woman was one of the people who told me I was deep.

At another point in the discussion, a man spoke of some benefit X of death, I don't recall exactly what. And I said: "You know, given human nature, if people got hit on the head by a baseball bat every week, pretty soon they would invent reasons why getting hit on the head with a baseball bat was a good thing. But if you took someone who wasn't being hit on the head with a baseball bat, and you asked them if they wanted it, they would say no. I think that if you took someone who was immortal, and asked them if they wanted to die for benefit X, they would say no."

Afterward, this man told me I was deep.

Correlation is not causality. Maybe I was just speaking in a deep voice that day, and so sounded wise.

But my suspicion is that I came across as "deep" because I coherently violated the cached pattern for "deep wisdom" in a way that made immediate sense.

There's a stereotype of Deep Wisdom. Death: complete the pattern: "Death gives meaning to life." Everyone knows this standard Deeply Wise response. And so it takes on some of the characteristics of an applause light. If you say it, people may nod along, because the brain completes the pattern and they know they're supposed to nod. They may even say "What deep wisdom!", perhaps in the hope of being thought deep themselves. But they will not be surprised; they will not have heard anything outside the box; they will not have heard anything they could not have thought of for themselves. One might call it belief in wisdom - the thought is labeled "deeply wise", and it's the completed standard pattern for "deep wisdom", but it carries no experience of insight.

People who try to seem Deeply Wise often end up seeming hollow, echoing as it were, because they're trying to seem Deeply Wise instead of optimizing.

How much thinking did I need to do, in the course of seeming deep? Human brains only run at 100Hz and I responded in realtime, so most of the work must have been precomputed. The part I experienced as effortful was picking a response understandable in one inferential step and then phrasing it for maximum impact.

Philosophically, nearly all of my work was already done. Complete the pattern: Existing condition X is really justified because it has benefit Y: "Naturalistic fallacy?" / "Status quo bias?" / "Could we get Y without X?" / "If we had never even heard of X before, would we voluntarily take it on to get Y?" I think it's fair to say that I execute these thought-patterns at around the same level of automaticity as I breathe. After all, most of human thought has to be cache lookups if the brain is to work at all.

And I already held to the developed philosophy of transhumanism. Transhumanism also has cached thoughts about death. Death: complete the pattern: "Death is a pointless tragedy which people rationalize." This was a nonstandard cache, one with which my listeners were unfamiliar. I had several opportunities to use nonstandard cache, and because they were all part of the developed philosophy of transhumanism, they all visibly belonged to the same theme. This made me seem coherent, as well as original.

I suspect this is one reason Eastern philosophy seems deep to Westerners - it has nonstandard but coherent cache for Deep Wisdom. Symmetrically, in works of Japanese fiction, one sometimes finds Christians depicted as repositories of deep wisdom and/or mystical secrets. (And sometimes not.)

If I recall correctly an economist once remarked that popular audiences are so unfamiliar with standard economics that, when he was called upon to make a television appearance, he just needed to repeat back Econ 101 in order to sound like a brilliantly original thinker.

Also crucial was that my listeners could see immediately that my reply made sense. They might or might not have agreed with the thought, but it was not a complete non-sequitur unto them. I know transhumanists who are unable to seem deep because they are unable to appreciate what their listener does not already know. If you want to sound deep, you can never say anything that is more than a single step of inferential distance away from your listener's current mental state. That's just the way it is.

To seem deep, study nonstandard philosophies. Seek out discussions on topics that will give you a chance to appear deep. Do your philosophical thinking in advance, so you can concentrate on explaining well. Above all, practice staying within the one-inferential-step bound.

To be deep, think for yourself about "wise" or important or emotionally fraught topics. Thinking for yourself isn't the same as coming up with an unusual answer. It does mean seeing for yourself, rather than letting your brain complete the pattern. If you don't stop at the first answer, and cast out replies that seem vaguely unsatisfactory, in time your thoughts will form a coherent whole, flowing from the single source of yourself, rather than being fragmentary repetitions of other people's conclusions.

Sequence: Seeing with Fresh Eyes

The Logical Fallacy of Generalization from Fictional Evidence

We Change Our Minds Less Often Than We Think

Referenced by: Why Are Individual IQ Differences OK? • Resist the Happy Death Spiral • Every Cause Wants To Be A Cult • Something to Protect • Zombies! Zombies? • Timeless Identity • Against Devil's Advocacy • Interpersonal Morality • Raised in Technophilia • Trying to Try • On Doing the Impossible • Entangled Truths, Contagious Lies • Dark Side Epistemology • Serious Stories • Belief in Self-Deception • Raising the Sanity Waterline • My Way

Original with comments: How to Seem (and Be) Deep

The Logical Fallacy of Generalization from Fictional Evidence

Eliezer Yudkowsky, 16 October 2007 03:57AM

When I try to introduce the subject of advanced AI, what's the first thing I hear, more than half the time?

"Oh, you mean like the Terminator movies / the Matrix / Asimov's robots!"

And I reply, "Well, no, not exactly. I try to avoid the logical fallacy of generalizing from fictional evidence."

Some people get it right away, and laugh. Others defend their use of the example, disagreeing that it's a fallacy.

What's wrong with using movies or novels as starting points for the discussion? No one's claiming that it's true, after all. Where is the lie, where is the rationalist sin? Science fiction represents the author's attempt to visualize the future; why not take advantage of the thinking that's already been done on our behalf, instead of starting over?

Not every misstep in the precise dance of rationality consists of outright belief in a falsehood; there are subtler ways to go wrong.

First, let us dispose of the notion that science fiction represents a full-fledged rational attempt to forecast the future. Even the most diligent science fiction writers are, first and foremost, storytellers; the requirements of storytelling are not the same as the requirements of forecasting. As Nick Bostrom points out:

"When was the last time you saw a movie about humankind suddenly going extinct (without warning and without being replaced by some other civilization)? While this scenario may be much more probable than a scenario in which human heroes successfully repel an invasion of monsters or robot warriors, it wouldn’t be much fun to watch."

So there are specific distortions in fiction. But trying to correct for these specific distortions is not enough. A story is never a rational attempt at analysis, not even with the most diligent science fiction writers, because stories don't use probability distributions. I illustrate as follows:

Bob Merkelthud slid cautiously through the door of the alien spacecraft, glancing right and then left (or left and then right) to see whether any of the dreaded Space Monsters yet remained. At his side was the only weapon that had been found effective against the Space Monsters, a Space Sword forged of pure titanium with 30% probability, an ordinary iron crowbar with 20% probability, and a shimmering black discus found in the smoking ruins of Stonehenge with 45% probability, the remaining 5% being distributed over too many minor outcomes to list here.

Merklethud (though there's a significant chance that Susan Wifflefoofer was there instead) took two steps forward or one step back, when a vast roar split the silence of the black airlock! Or the quiet background hum of the white airlock! Although Amfer and Woofi (1997) argue that Merklethud is devoured at this point, Spacklebackle (2003) points out that -

Characters can be ignorant, but the author can't say the three magic words "I don't know." The protagonist must thread a single line through the future, full of the details that lend flesh to the story, from Wifflefoofer's appropriately futuristic attitudes toward feminism, down to the color of her earrings.

Then all these burdensome details and questionable assumptions are wrapped up and given a short label, creating the illusion that they are a single package.

On problems with large answer spaces, the greatest difficulty is not verifying the correct answer but simply locating it in answer space to begin with. If someone starts out by asking whether or not AIs are gonna put us into capsules like in "The Matrix", they're jumping to a 100-bit proposition, without a corresponding 98 bits of evidence to locate it in the answer space as a possibility worthy of explicit consideration. It would only take a handful more evidence after the first 98 bits to promote that possibility to near-certainty, which tells you something about where nearly all the work gets done.

The "preliminary" step of locating possibilities worthy of explicit consideration includes steps like: Weighing what you know and don't know, what you can and can't predict, making a deliberate effort to avoid absurdity bias and widen confidence intervals, pondering which questions are the important ones, trying to adjust for possible Black Swans and think of (formerly) unknown unknowns. Jumping to "The Matrix: Yes or No?" skips over all of this.

Any professional negotiator knows that to control the terms of a debate is very nearly to control the outcome of the debate. If you start out by thinking of The Matrix, it brings to mind marching robot armies defeating humans after a long struggle - not a superintelligence snapping nanotechnological fingers. It focuses on an "Us vs. Them" struggle, directing attention to questions like "Who will win?" and "Who should win?" and "Will AIs really be like that?" It creates a general atmosphere of entertainment, of "What is your amazing vision of the future?"

Lost to the echoing emptiness are: considerations of more than one possible mind design that an "Artificial Intelligence" could implement; the future's dependence on initial conditions; the power of smarter-than-human intelligence and the argument for its unpredictability; people taking the whole matter seriously and trying to do something about it.

If some insidious corrupter of debates decided that their preferred outcome would be best served by forcing discussants to start out by refuting Terminator, they would have done well in skewing the frame. Debating gun control, the NRA spokesperson does not wish to introduced as a "shooting freak", the anti-gun opponent does not wish to be introduced as a "victim disarmament advocate". Why should you allow the same order of frame-skewing by Hollywood scriptwriters, even accidentally?

Journalists don't tell me, "The future will be like 2001". But they ask, "Will the future be like 2001, or will it be like A.I.?" This is just as huge a framing issue as asking "Should we cut benefits for disabled veterans, or raise taxes on the rich?"

In the ancestral environment, there were no moving pictures; what you saw with your own eyes was true. A momentary glimpse of a single word can prime us and make compatible thoughts more available, with demonstrated strong influence on probability estimates. How much havoc do you think a two-hour movie can wreak on your judgment? It will be hard enough to undo the damage by deliberate concentration - why invite the vampire into your house? In Chess or Go, every wasted move is a loss; in rationality, any non-evidential influence is (on average) entropic.

Do movie-viewers succeed in unbelieving what they see? So far as I can tell, few movie viewers act as if they have directly observed Earth's future. People who watched the Terminator movies didn't hide in fallout shelters on August 29, 1997. But those who commit the fallacy seem to act as if they had seen the movie events occurring on some other planet; not Earth, but somewhere similar to Earth.

You say, "Suppose we build a very smart AI," and they say, "But didn't that lead to nuclear war in The Terminator?" As far as I can tell, it's identical reasoning, down to the tone of voice, of someone who might say: "But didn't that lead to nuclear war on Alpha Centauri?" or "Didn't that lead to the fall of the Italian city-state of Piccolo in the fourteenth century?" The movie is not believed, but it is available. It is treated, not as a prophecy, but as an illustrative historical case. Will history repeat itself? Who knows?

In a recent Singularity discussion, someone mentioned that Vinge didn't seem to think that brain-computer interfaces would increase intelligence much, and cited Marooned in Realtime and Tunç Blumenthal, who was the most advanced traveller but didn't seem all that powerful. I replied indignantly, "But Tunç  lost most of his hardware! He was crippled!" And then I did a mental double-take and thought to myself: What the hell am I saying.

Does the issue not have to be argued in its own right, regardless of how Vinge depicted his characters? Tunç Blumenthal is not "crippled", he's unreal. I could say "Vinge chose to depict Tunç as crippled, for reasons that may or may not have had anything to do with his personal best forecast," and that would give his authorial choice an appropriate weight of evidence. I cannot say "Tunç was crippled." There is no was of Tunç Blumenthal.

I deliberately left in a mistake I made, in my first draft of the top of this post: "Others defend their use of the example, disagreeing that it's a fallacy." But the Matrix is not an example!

A neighboring flaw is the logical fallacy of arguing from imaginary evidence: "Well, if you did go to the end of the rainbow, you would find a pot of gold - which just proves my point!" (Updating on evidence predicted, but not observed, is the mathematical mirror image of hindsight bias.)

The brain has many mechanisms for generalizing from observation, not just the availability heuristic. You see three zebras, you form the category "zebra", and this category embodies an automatic perceptual inference. Horse-shaped creatures with white and black stripes are classified as "Zebras", therefore they are fast and good to eat; they are expected to be similar to other zebras observed.

So people see (moving pictures of) three Borg, their brain automatically creates the category "Borg", and they infer automatically that humans with brain-computer interfaces are of class "Borg" and will be similar to other Borg observed: cold, uncompassionate, dressing in black leather, walking with heavy mechanical steps. Journalists don't believe that the future will contain Borg - they don't believe Star Trek is a prophecy. But when someone talks about brain-computer interfaces, they think, "Will the future contain Borg?" Not, "How do I know computer-assisted telepathy makes people less nice?" Not, "I've never seen a Borg and never has anyone else." Not, "I'm forming a racial stereotype based on literally zero evidence."

As George Orwell said of cliches:

"What is above all needed is to let the meaning choose the word, and not the other way around... When you think of something abstract you are more inclined to use words from the start, and unless you make a conscious effort to prevent it, the existing dialect will come rushing in and do the job for you, at the expense of blurring or even changing your meaning."

Yet in my estimation, the most damaging aspect of using other authors' imaginations is that it stops people from using their own. As Robert Pirsig said:

"She was blocked because she was trying to repeat, in her writing, things she had already heard, just as on the first day he had tried to repeat things he had already decided to say. She couldn't think of anything to write about Bozeman because she couldn't recall anything she had heard worth repeating. She was strangely unaware that she could look and see freshly for herself, as she wrote, without primary regard for what had been said before."

Remembered fictions rush in and do your thinking for you; they substitute for seeing - the deadliest convenience of all.

Viewpoints taken here are further supported in: Anchoring, Contamination, Availability, Cached Thoughts, Do We Believe Everything We're Told?, Einstein's Arrogance, Burdensome details

Sequence: Seeing with Fresh Eyes

Original Seeing

How to Seem (and Be) Deep

Referenced by: Thou Art Godshatter • Artificial Addition • Fake Utility Functions • Something to Protect • My Childhood Role Model • Touching the Old • Qualitative Strategies of Friendliness • Trying to Try • Devil's Offers • Amputation of Destiny • Eutopia is Scary • Getting Nearer • (Moral) Truth in Fiction? • Bayesians vs. Barbarians

Original with comments: The Logical Fallacy of Generalization from Fictional Evidence

Hold Off On Proposing Solutions

Eliezer Yudkowsky, 17 October 2007 03:16AM

From pp. 55-56 of Robyn Dawes's Rational Choice in an Uncertain World. Bolding added.

Norman R. F. Maier noted that when a group faces a problem, the natural tendency of its members is to propose possible solutions as they begin to discuss the problem. Consequently, the group interaction focuses on the merits and problems of the proposed solutions, people become emotionally attached to the ones they have suggested, and superior solutions are not suggested. Maier enacted an edict to enhance group problem solving: "Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any." It is easy to show that this edict works in contexts where there are objectively defined good solutions to problems.

Maier devised the following "role playing" experiment to demonstrate his point. Three employees of differing ability work on an assembly line. They rotate among three jobs that require different levels of ability, because the most able - who is also the most dominant - is strongly motivated to avoid boredom. In contrast, the least able worker, aware that he does not perform the more difficult jobs as well as the other two, has agreed to rotation because of the dominance of his able co-worker. An "efficiency expert" notes that if the most able employee were given the most difficult task and the least able the least difficult, productivity could be improved by 20%, and the expert recommends that the employees stop rotating. The three employees and the a fourth person designated to play the role of foreman are asked to discuss the expert's recommendation. Some role-playing groups are given Maier's edict not to discuss solutions until having discussed the problem thoroughly, while others are not. Those who are not given the edict immediately begin to argue about the importance of productivity versus worker autonomy and the avoidance of boredom. Groups presented with the edict have a much higher probability of arriving at the solution that the two more able workers rotate, while the least able one sticks to the least demanding job - a solution that yields a 19% increase in productivity.

I have often used this edict with groups I have led - particularly when they face a very tough problem, which is when group members are most apt to propose solutions immediately. While I have no objective criterion on which to judge the quality of the problem solving of the groups, Maier's edict appears to foster better solutions to problems.

This is so true it's not even funny. And it gets worse and worse the tougher the problem becomes. Take Artificial Intelligence, for example. A surprising number of people I meet seem to know exactly how to build an Artificial General Intelligence, without, say, knowing how to build an optical character recognizer or a collaborative filtering system (much easier problems). And as for building an AI with a positive impact on the world - a Friendly AI, loosely speaking - why, that problem is so incredibly difficult that an actual majority resolve the whole issue within 15 seconds. Give me a break.

(Added: This problem is by no means unique to AI. Physicists encounter plenty of nonphysicists with their own theories of physics, economists get to hear lots of amazing new theories of economics. If you're an evolutionary biologist, anyone you meet can instantly solve any open problem in your field, usually by postulating group selection. Et cetera.)

Maier's advice echoes the principle of the bottom line, that the effectiveness of our decisions is determined only by whatever evidence and processing we did in first arriving at our decisions - after you write the bottom line, it is too late to write more reasons above. If you make your decision very early on, it will, in fact, be based on very little thought, no matter how many amazing arguments you come up with afterward.

And consider furthermore that We Change Our Minds Less Often Than We Think: 24 people assigned an average 66% probability to the future choice thought more probable, but only 1 in 24 actually chose the option thought less probable. Once you can guess what your answer will be, you have probably already decided. If you can guess your answer half a second after hearing the question, then you have half a second in which to be intelligent. It's not a lot of time.

Traditional Rationality emphasizes falsification - the ability to relinquish an initial opinion when confronted by clear evidence against it. But once an idea gets into your head, it will probably require way too much evidence to get it out again. Worse, we don't always have the luxury of overwhelming evidence.

I suspect that a more powerful (and more difficult) method is to hold off on thinking of an answer. To suspend, draw out, that tiny moment when we can't yet guess what our answer will be; thus giving our intelligence a longer time in which to act.

Even half a minute would be an improvement over half a second.

Sequence: Seeing with Fresh Eyes

We Change Our Minds Less Often Than We Think

Asch's Conformity Experiment

Referenced by: Motivated Stopping and Motivated Continuation • A Case Study of Motivated Continuation • Fake Utility Functions • LA-602 vs. RHIC Review • A Genius for Destruction • My Best and Worst Mistake • My Bayesian Enlightenment • Crisis of Faith • Back Up and Ask Whether, Not Why • That Crisis thing seems pretty useful • Working Mantras

Original with comments: Hold Off On Proposing Solutions

"Can't Say No" Spending

Eliezer Yudkowsky, 18 October 2007 02:08AM

The remarkable observation that medical spending has zero net marginal effect is shocking, but not completely unprecedented.

According to Spiegel in "Too Much of a Good Thing: Choking on Aid Money in Africa", the Washington Center for Global Development calculated that it would require $3,521 of marginal development aid invested, per person, in order to increase per capita yearly income by $3.65 (one penny per day).

The Kenyan economist James Shikwati is even more pessimistic in "For God's Sake, Please Stop the Aid!": The net effect of Western aid to Africa is actively destructive (even when it isn't stolen to prop up corrupt regimes), a chaotic flux of money and goods that destroys local industry.

What does aid to Africa have in common with healthcare spending? Besides, of course, that it's heartbreaking to just say no -

Referenced by: Motivated Stopping and Motivated Continuation • Higher Purpose

Original with comments: "Can't Say No" Spending

Congratulations to Paris Hilton

Eliezer Yudkowsky, 19 October 2007 12:31AM

...on signing up for cryopreservation with the Cryonics Institute.

(No, it's not a joke.)

Anyone not signed up for cryonics has now lost the right to make fun of Paris Hilton,
because no matter what else she does wrong, and what else you do right,
all of it together can't outweigh the life consequences of that one little decision.

Congratulations, Paris. I look forward to meeting you someday.

Addendum: On Nov 28 '07, Paris Hilton denied being signed up for cryonics. Oh well.

Original with comments: Congratulations to Paris Hilton

Pascal's Mugging: Tiny Probabilities of Vast Utilities

Eliezer Yudkowsky, 19 October 2007 11:37PM

The most common formalizations of Occam's Razor, Solomonoff induction and Minimum Description Length, measure the program size of a computation used in a hypothesis, but don't measure the running time or space requirements of the computation. What if this makes a mind vulnerable to finite forms of Pascal's Wager? A compactly specified wager can grow in size much faster than it grows in complexity. The utility of a Turing machine can grow much faster than its prior probability shrinks.

Consider Knuth's up-arrow notation:

- 3^3 = 3*3*3 = 27

- 3^^3 = (3^(3^3)) = 3^27 = 3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3 = 7625597484987

- 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = 3^(3^(3^(... 7625597484987 times ...)))

In other words: 3^^^3 describes an exponential tower of threes 7625597484987 layers tall. Since this number can be computed by a simple Turing machine, it contains very little information and requires a very short message to describe. This, even though writing out 3^^^3 in base 10 would require enormously more writing material than there are atoms in the known universe (a paltry 10^80).

Now suppose someone comes to me and says, "Give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people."

Call this Pascal's Mugging.

"Magic powers from outside the Matrix" are easier said than done - we have to suppose that our world is a computing simulation run from within an environment that can afford simulation of arbitrarily large finite Turing machines, and that the would-be wizard has been spliced into our own Turing tape and is in continuing communication with an outside operator, etc.

Thus the Kolmogorov complexity of "magic powers from outside the Matrix" is larger than the mere English words would indicate. Therefore the Solomonoff-inducted probability, two to the negative Kolmogorov complexity, is exponentially tinier than one might naively think.

But, small as this probability is, it isn't anywhere near as small as 3^^^^3 is large. If you take a decimal point, followed by a number of zeros equal to the length of the Bible, followed by a 1, and multiply this unimaginably tiny fraction by 3^^^^3, the result is pretty much 3^^^^3.

Most people, I think, envision an "infinite" God that is nowhere near as large as 3^^^^3. "Infinity" is reassuringly featureless and blank. "Eternal life in Heaven" is nowhere near as intimidating as the thought of spending 3^^^^3 years on one of those fluffy clouds. The notion that the diversity of life on Earth springs from God's infinite creativity, sounds more plausible than the notion that life on Earth was created by a superintelligence 3^^^^3 bits large. Similarly for envisioning an "infinite" God interested in whether women wear men's clothing, versus a superintelligence of 3^^^^3 bits, etc.

The original version of Pascal's Wager is easily dealt with by the gigantic multiplicity of possible gods, an Allah for every Christ and a Zeus for every Allah, including the "Professor God" who places only atheists in Heaven. And since all the expected utilities here are allegedly "infinite", it's easy enough to argue that they cancel out. Infinities, being featureless and blank, are all the same size.

But suppose I built an AI which worked by some bounded analogue of Solomonoff induction - an AI sufficiently Bayesian to insist on calculating complexities and assessing probabilities, rather than just waving them off as "large" or "small".

If the probabilities of various scenarios considered did not exactly cancel out, the AI's action in the case of Pascal's Mugging would be overwhelmingly dominated by whatever tiny differentials existed in the various tiny probabilities under which 3^^^^3 units of expected utility were actually at stake.

You or I would probably wave off the whole matter with a laugh, planning according to the dominant mainline probability: Pascal's Mugger is just a philosopher out for a fast buck.

But a silicon chip does not look over the code fed to it, assess it for reasonableness, and correct it if not. An AI is not given its code like a human servant given instructions. An AI is its code. What if a philosopher tries Pascal's Mugging on the AI for a joke, and the tiny probabilities of 3^^^^3 lives being at stake, override everything else in the AI's calculations? What is the mere Earth at stake, compared to a tiny probability of 3^^^^3 lives?

How do I know to be worried by this line of reasoning? How do I know to rationalize reasons a Bayesian shouldn't work that way? A mind that worked strictly by Solomonoff induction would not know to rationalize reasons that Pascal's Mugging mattered less than Earth's existence. It would simply go by whatever answer Solomonoff induction obtained.

It would seem, then, that I've implicitly declared my existence as a mind that does not work by the logic of Solomonoff, at least not the way I've described it. What am I comparing Solomonoff's answer to, to determine whether Solomonoff induction got it "right" or "wrong"?

Why do I think it's unreasonable to focus my entire attention on the magic-bearing possible worlds, faced with a Pascal's Mugging? Do I have an instinct to resist exploitation by arguments "anyone could make"? Am I unsatisfied by any visualization in which the dominant mainline probability leads to a loss? Do I drop sufficiently small probabilities from consideration entirely? Would an AI that lacks these instincts be exploitable by Pascal's Mugging?

Is it me who's wrong? Should I worry more about the possibility of some Unseen Magical Prankster of very tiny probability taking this post literally, than about the fate of the human species in the "mainline" probabilities?

It doesn't feel to me like 3^^^^3 lives are really at stake, even at very tiny probability. I'd sooner question my grasp of "rationality" than give five dollars to a Pascal's Mugger because I thought it was "rational".

Should we penalize computations with large space and time requirements? This is a hack that solves the problem, but is it true? Are computationally costly explanations less likely? Should I think the universe is probably a coarse-grained simulation of my mind rather than real quantum physics, because a coarse-grained human mind is fifty(?) orders of magnitude cheaper than real quantum physics? Should I think the galaxies are tiny lights on a painted backdrop, because that Turing machine would require less space to compute?

Given that, in general, a Turing machine can increase in utility vastly faster than it increases in complexity, how should an Occam-abiding mind avoid being dominated by tiny probabilities of vast utilities?

If I could formalize whichever internal criterion was telling me I didn't want this to happen, I might have an answer.

I talked over a variant of this problem with Nick Hay, Peter de Blanc, and Marcello Herreshoff in summer of 2006. I don't feel I have a satisfactory resolution as yet, so I'm throwing it open to any analytic philosophers who might happen to read Overcoming Bias.

Referenced by: Belief in the Implied Invisible • Complexity and Intelligence • Not Taking Over the World • Can Humanism Match Religion's Output?

Original with comments: Pascal's Mugging: Tiny Probabilities of Vast Utilities

Illusion of Transparency: Why No One Understands You

Eliezer Yudkowsky, 20 October 2007 11:49PM

In hindsight bias, people who know the outcome of a situation believe the outcome should have been easy to predict in advance. Knowing the outcome, we reinterpret the situation in light of that outcome. Even when warned, we can't de-interpret to empathize with someone who doesn't know what we know.

Closely related is the illusion of transparency: We always know what we mean by our words, and so we expect others to know it too. Reading our own writing, the intended interpretation falls easily into place, guided by our knowledge of what we really meant. It's hard to empathize with someone who must interpret blindly, guided only by the words.

June recommends a restaurant to Mark; Mark dines there and discovers (a) unimpressive food and mediocre service (b) delicious food and impeccable service. Then Mark leaves the following message on June's answering machine: "June, I just finished dinner at the restaurant you recommended, and I must say, it was marvelous, just marvelous." Keysar (1994) presented a group of subjects with scenario (a), and 59% thought that Mark's message was sarcastic and that Jane would perceive the sarcasm. Among other subjects, told scenario (b), only 3% thought that Jane would perceive Mark's message as sarcastic. Keysar and Barr (2002) seem to indicate that an actual voice message was played back to the subjects.

Keysar (1998) showed that if subjects were told that the restaurant was horrible but that Mark wanted to conceal his response, they believed June would not perceive sarcasm in the (same) message:

"They were just as likely to predict that she would perceive sarcasm when he attempted to conceal his negative experience as when he had a positive experience and was truly sincere. So participants took Mark's communicative intention as transparent. It was as if they assumed that June would perceive whatever intention Mark wanted her to perceive."

(The above wording is from Keysar and Barr 2002.)

"The goose hangs high" is an archaic English idiom which has passed out of use in modern language. Keysar and Bly (1995) told one group of subjects that "the goose hangs high" meant that the future looks good; another group of subjects learned that "the goose hangs high" meant the future looks gloomy. Subjects were then asked which of these two meanings an uninformed listener would be more likely to attribute to the idiom. Each group thought that listeners would perceive the meaning presented as "standard".

(Other idioms tested included "come the uncle over someone", "to go by the board", and "to lay out in lavender". Ah, English, such a lovely language.)

Keysar and Henly (2002) tested the calibration of speakers: Would speakers underestimate, overestimate, or correctly estimate how often listeners understood them? Speakers were given ambiguous sentences ("The man is chasing a woman on a bicycle") and disambiguating pictures (a man running after a cycling woman), then asked the speakers to utter the words in front of addressees, then asked speakers to estimate how many addressees understood the intended meaning. Speakers thought that they were understood in 72% of cases and were actually understood in 61% of cases. When addressees did not understand, speakers thought they did in 46% of cases; when addressees did understand, speakers thought they did not in only 12% of cases.

Additional subjects who overheard the explanation showed no such bias, expecting listeners to understand in only 56% of cases.

As Keysar and Barr (2002) note, two days before Germany's attack on Poland, Chamberlain sent a letter intended to make it clear that Britain would fight if any invasion occurred. The letter, phrased in polite dipliomatese, was heard by Hitler as conciliatory - and the tanks rolled.

Be not too quick to blame those who misunderstand your perfectly clear sentences, spoken or written. Chances are, your words are more ambiguous than you think.

Keysar, B. (1994). The illusory transparency of intention: Linguistic perspective taking in text. Cognitive Psychology, 26, 165-208.

Keysar, B. (1998). Language users as problem solvers: Just what ambiguity problem do they solve? In S. R. Fussell and R. J. Kreuz (Eds.), Social and Cognitive Psychological Approaches to Interpersonal Communication, 175-200. Hillsdale, NJ: Erlbaum.

Keysar, B., & Barr, D. J. (2002). Self anchoring in conversation: Why language users do not do what they "should". (2002). In T. Gilovich, D. W. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment. (pp. 150-166). Cambridge University Press.

Keysar, B., & Bly, B. (1995.) Intuitions of the transparency of idioms: Can one keep a secret by spilling the beans? Journal of Memory and Language, 34, pp. 89-109.

Keysar, B., & Henly, A. S. (2002). Speakers' overestimation of their effectiveness. Psychological Science, 13, 207-212.

Referenced by: Expecting Short Inferential Distances • Explainers Shoot High. Aim Low! • Double Illusion of Transparency • Fake Utility Functions • Moral Error and Moral Disagreement • Abstracted Idealized Dynamics • Whence Your Abstractions? • Practical Advice Backed By Deep Theories

Original with comments: Illusion of Transparency: Why No One Understands You

Self-Anchoring

Eliezer Yudkowsky, 22 October 2007 06:11AM

Sometime between the age of 3 and 4, a human child becomes able, for the first time, to model other minds as having different beliefs. The child sees a box, sees candy in the box, and sees that Sally sees the box. Sally leaves, and then the experimenter, in front of the child, replaces the candy with pencils and closes the box so that the inside is not visible. Sally returns, and the child is asked what Sally thinks is in the box. Children younger than 3 say "pencils", children older than 4 say "candy".

Our ability to visualize other minds is imperfect. Neural circuitry is not as flexible as a program fed to a general-purpose computer. An AI, with fast read-write access to its own memory, might be able to create a distinct, simulated visual cortex to imagine what a human "sees". We humans only have one visual cortex, and if we want to imagine what someone else is seeing, we've got to simulate it using our own visual cortex - put our own brains into the other mind's shoes. And because you can't reconfigure memory to simulate a new brain from stratch, pieces of you leak into your visualization of the Other.

The diagram above is from Keysar, Barr, Balin, & Brauner (2000). The experimental subject, the "addressee", sat in front of an array of objects, viewed as seen on the left. On the other side, across from the addressee, sat the "director", with the view as seen on the right. The addressee had an unblocked view, which also allowed the addressee to see which objects were not visible to the director.

The experiment used the eye-tracking method: the direction of a subject's gaze can be measured using computer vision. Tanenhaus et. al. (1995) had previously demonstrated that when people understand a spoken reference, their gaze fixates on the identified object almost immediately.

The key test was when the director said "Put the small candle next to the truck." As the addressee can clearly observe, the director only knows about two candles, the largest and medium ones; the smallest candle is occluded.

And, lo and behold, subjects' eyes fixated on the occluded smallest candle an average of 1,487 milliseconds before they correctly identified the medium-sized candle as the one the director must have meant.

This seems to suggest that subjects first computed the meaning according to their brains' settings, their knowledge, and then afterward adjusted for the other mind's different knowledge.

Numerous experiments suggest that where there is adjustment, there is usually under-adjustment, which leads to anchoring. In this case, "self-anchoring".

Barr (2003) argues that the processes are actually more akin to contamination and under-correction; we can't stop ourselves from leaking over, and then we can't correct for the leakage. Different process, same outcome:

We can put our feet in other minds' shoes, but we keep our own socks on.

Barr, D. J. (2003). Listeners are mentally contaminated. Poster presented at the 44th annual meeting of the Psychonomic Society, Vancouver.

Keysar, B., Barr, D. J., Balin, J. A., & Brauner, J. S. (2000). Taking perspective in conversation: The role of mutual knowledge in comprehension. Psychological Sciences, 11, 32-38.

Perner, J., Leekam, S. R., & Wimmer, H. (1987). Three-year-olds' difficulty with false belief: The case for a conceptual deficit. British Journal of Developmental Psychology, 5(2), 125–137.

Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K.M. & Sedivy, J.C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science 268: 1632-1634.

Referenced by: Expecting Short Inferential Distances • Double Illusion of Transparency

Original with comments: Self-Anchoring

Expecting Short Inferential Distances

Eliezer Yudkowsky, 22 October 2007 11:42PM

Homo sapiens' environment of evolutionary adaptedness (aka EEA or "ancestral environment") consisted of hunter-gatherer bands of at most 200 people, with no writing. All inherited knowledge was passed down by speech and memory.

In a world like that, all background knowledge is universal knowledge. All information not strictly private is public, period.

In the ancestral environment, you were unlikely to end up more than one inferential step away from anyone else. When you discover a new oasis, you don't have to explain to your fellow tribe members what an oasis is, or why it's a good idea to drink water, or how to walk. Only you know where the oasis lies; this is private knowledge. But everyone has the background to understand your description of the oasis, the concepts needed to think about water; this is universal knowledge. When you explain things in an ancestral environment, you almost never have to explain your concepts. At most you have to explain one new concept, not two or more simultaneously.

In the ancestral environment there were no abstract disciplines with vast bodies of carefully gathered evidence generalized into elegant theories transmitted by written books whose conclusions are a hundred inferential steps removed from universally shared background premises.

In the ancestral environment, anyone who says something with no obvious support, is a liar or an idiot. You're not likely to think, "Hey, maybe this guy has well-supported background knowledge that no one in my band has even heard of," because it was a reliable invariant of the ancestral environment that this didn't happen.

Conversely, if you say something blatantly obvious and the other person doesn't see it, they're the idiot, or they're being deliberately obstinate to annoy you.

And to top it off, if someone says something with no obvious support and expects you to believe it - acting all indignant when you don't - then they must be crazy.

Combined with the illusion of transparency and self-anchoring, I think this explains a lot about the legendary difficulty most scientists have in communicating with a lay audience - or even communicating with scientists from other disciplines. When I observe failures of explanation, I usually see the explainer taking one step back, when they need to take two or more steps back. Or listeners, assuming that things should be visible in one step, when they take two or more steps to explain. Both sides act as if they expect very short inferential distances from universal knowledge to any new knowledge.

A biologist, speaking to a physicist, can justify evolution by saying it is "the simplest explanation". But not everyone on Earth has been inculcated with that legendary history of science, from Newton to Einstein, which invests the phrase "simplest explanation" with its awesome import: a Word of Power, spoken at the birth of theories and carved on their tombstones. To someone else, "But it's the simplest explanation!" may sound like an interesting but hardly knockdown argument; it doesn't feel like all that powerful a tool for comprehending office politics or fixing a broken car. Obviously the biologist is infatuated with his own ideas, too arrogant to be open to alternative explanations which sound just as plausible. (If it sounds plausible to me, it should sound plausible to any sane member of my band.)

And from the biologist's perspective, he can understand how evolution might sound a little odd at first - but when someone rejects evolution even after the biologist explains that it's the simplest explanation, well, it's clear that nonscientists are just idiots and there's no point in talking to them.

A clear argument has to lay out an inferential pathway, starting from what the audience already knows or accepts. If you don't recurse far enough, you're just talking to yourself.

If at any point you make a statement without obvious justification in arguments you've previously supported, the audience just thinks you're a cult victim.

This also happens when you allow yourself to be seen visibly attaching greater weight to an argument than is justified in the eyes of the audience at that time. For example, talking as if you think "simpler explanation" is a knockdown argument for evolution (which it is), rather than a sorta-interesting idea (which it sounds like to someone who hasn't been raised to revere Occam's Razor).

Oh, and you'd better not drop any hints that you think you're working a dozen inferential steps away from what the audience knows, or that you think you have special background knowledge not available to them. The audience doesn't know anything about an evolutionary-psychological argument for a cognitive bias to underestimate inferential distances leading to traffic jams in communication. They'll just think you're condescending.

And if you think you can explain the concept of "systematically underestimated inferential distances" briefly, in just a few words, I've got some sad news for you...

Referenced by: Explainers Shoot High. Aim Low! • Double Illusion of Transparency • No One Knows What Science Doesn't Know • Fake Fake Utility Functions • Fake Utility Functions • Cultish Countercultishness • My Strange Beliefs • Absolute Authority • Zut Allais! • Searching for Bayes-Structure • Identity Isn't In Specific Atoms • My Childhood Role Model • A Premature Word on AI • Timeless Identity • Why Quantum? • Contaminated by Optimism • Moral Error and Moral Disagreement • Abstracted Idealized Dynamics • You Provably Can't Trust Yourself • Is That Your True Rejection? • Beware of Other-Optimizing • Practical Advice Backed By Deep Theories

Original with comments: Expecting Short Inferential Distances

Explainers Shoot High. Aim Low!

Eliezer Yudkowsky, 24 October 2007 01:13AM

Followup to: Illusion of Transparency: Why No One Understands You, Expecting Short Inferential Distances

A few years ago, an eminent scientist once told me how he'd written an explanation of his field aimed at a much lower technical level than usual. He had thought it would be useful to academics outside the field, or even reporters. This ended up being one of his most popular papers within his field, cited more often than anything else he'd written.

The lesson was not that his fellow scientists were stupid, but that we tend to enormously underestimate the effort required to properly explain things.

He told me this, because I'd just told him about my experience publishing "An Intuitive Explanation of Bayesian Reasoning". This is still one of my most popular, most blogged, and most appreciated works today. I regularly get fan mail from formerly confused undergraduates taking statistics classes, and journalists, and professors from outside fields. In short, I successfully hit the audience the eminent scientist had thought he was aiming for.

I'd thought I was aiming for elementary school.

Today, when I look back at the Intuitive Explanation, it seems pretty silly as an attempt on grade school:

- It's assumed that the reader knows what a "probability" is.

- No single idea requires more than a single example.

- No homework problems! I've gotten several complaints about this.

(Then again, I get a roughly equal number of complaints that the Intuitive Explanation is too long and drawn-out, as that it is too short. The current version does seem to be "just right" for a fair number of people.)

Explainers shoot way, way higher than they think they're aiming, thanks to the illusion of transparency and self-anchoring. We miss the mark by several major grades of expertise. Aiming for outside academics gets you an article that will be popular among specialists in your field. Aiming at grade school (admittedly, naively so) will hit undergraduates. This is not because your audience is more stupid than you think, but because your words are far less helpful than you think. You're way way overshooting the target. Aim several major gradations lower, and you may hit your mark.

PS: I know and do confess that I need to work on taking my own advice.

Addendum: With his gracious permission: The eminent scientist was Ralph Merkle.

Referenced by: Double Illusion of Transparency

Original with comments: Explainers Shoot High. Aim Low!

Double Illusion of Transparency

Eliezer Yudkowsky, 24 October 2007 11:06PM

Followup to: Explainers Shoot High, Illusion of Transparency

My first true foray into Bayes For Everyone was writing An Intuitive Explanation of Bayesian Reasoning, still one of my most popular works. This is the Intuitive Explanation's origin story.

In December of 2002, I'd been sermonizing in a habitual IRC channels about what seemed to me like a very straightforward idea: How words, like all other useful forms of thought, are secretly a disguised form of Bayesian inference. I thought I was explaining clearly, and yet there was one fellow, it seemed, who didn't get it. This worried me, because this was someone who'd been very enthusiastic about my Bayesian sermons up to that point. He'd gone around telling people that Bayes was "the secret of the universe", a phrase I'd been known to use.

So I went into a private IRC conversation to clear up the sticking point.

And he still didn't get it.

I took a step back and explained the immediate prerequisites, which I had thought would be obvious -

He didn't understand my explanation of the prerequisites.

In desperation, I recursed all the way back to Bayes's Theorem, the ultimate foundation stone of -

He didn't know how to apply Bayes's Theorem to update the probability that a fruit is a banana, after it is observed to be yellow. He kept mixing up p(b|y) and p(y|b).

It seems like a small thing, I know. It's strange how small things can trigger major life-realizations. Any former TAs among my readers are probably laughing: I hadn't realized, until then, that instructors got misleading feedback. Robin commented yesterday that the best way to aim your explanations is feedback from the intended audience, "an advantage teachers often have". But what if self-anchoring also causes you to overestimate how much understanding appears in your feedback?

I fell prey to a double illusion of transparency. First, I assumed that my words meant what I intended them to mean - that my listeners heard my intentions as though they were transparent. Second, when someone repeated back my sentences using slightly different word orderings, I assumed that what I heard was what they had intended to say. As if all words were transparent windows into thought, in both directions.

I thought that if I said, "Hey, guess what I noticed today! Bayes's Theorem is the secret of the universe!", and someone else said, "Yes! Bayes's Theorem is the secret of the universe!", then this was what a successful teacher-student interaction looked like: knowledge conveyed and verified. I'd read Pirsig and I knew, in theory, about how students learn to repeat back what the teacher says in slightly different words. But I thought of that as a deliberate tactic to get good grades, and I wasn't grading anyone.

This may sound odd, but until that very day, I hadn't realized why there were such things as universities. I'd thought it was just rent-seekers who'd gotten a lock on the credentialing system. Why would you need teachers to learn? That was what books were for.

But now a great and terrible light was dawning upon me. Genuinely explaining complicated things took months or years, and an entire university infrastructure with painstakingly crafted textbooks and professional instructors. You couldn't just tell people.

You're laughing at me right now, academic readers; but think back and you'll realize that academics are generally very careful not to tell the general population how difficult it is to explain things, because it would come across as condescending. Physicists can't just say, "What we do is beyond your comprehension, foolish mortal" when Congress is considering their funding. Richard Feynman once said that if you really understand something in physics you should be able to explain it to your grandmother. I believed him. I was shocked to discover it wasn't true.

But once I realized, it became horribly clear why no one had picked up and run with any of the wonderful ideas I'd been telling about Artificial Intelligence.

If I wanted to explain all these marvelous ideas I had, I'd have to go back, and back, and back. I'd have to start with the things I'd figured out before I was even thinking about Artificial Intelligence, the foundations without which nothing else would make sense.

Like all that stuff I'd worked out about human rationality, back at the dawn of time.

Which I'd considerably reworked after receiving my Bayesian Enlightenment. But either way, I had to start with the foundations. Nothing I said about AI was going to make sense unless I started at the beginning. My listeners would just decide that emergence was a better explanation.

And the beginning of all things in the reworked version was Bayes, to which there didn't seem to be any decent online introduction for newbies. Most sources just stated Bayes's Theorem and defined the terms. This, I now realized, was not going to be sufficient. The online sources I saw didn't even say why Bayes's Theorem was important. E. T. Jaynes seemed to get it, but Jaynes spoke only in calculus - no hope for novices there.

So I mentally consigned everything I'd written before 2003 to the trash heap - it was mostly obsolete in the wake of my Bayesian Enlightenment, anyway - and started over at what I fondly conceived to be the beginning.

(It wasn't.)

And I would explain it so clearly that even grade school students would get it.

(They didn't.)

I had, and have, much left to learn about explaining. But that's how it all began.

Referenced by: No One Knows What Science Doesn't Know • Practical Advice Backed By Deep Theories

Original with comments: Double Illusion of Transparency

No One Knows What Science Doesn't Know

Eliezer Yudkowsky, 25 October 2007 11:47PM

At a family party some years ago, one of my uncles remarked on how little science really knows. For example, we still have no idea how gravity works - why things fall down.

"Actually, we do know how gravity works," I said. (My father, a Ph.D. physicist, was also present; but he wasn't even touching this one.)

"We do?" said my uncle.

"Yes," I said, "Gravity is the curvature of spacetime." At this point I had still swallowed Feynman's line about being able to explain physics to one's grandmother, so I continued: "You could say that the Earth goes around the Sun in a straight line. Imagine a graph that shows both space and time, so that a straight line shows steady movement and a curved line shows acceleration. Then curve the graph paper itself. When you try to draw a straight line on the curved paper, you'll get what looks like acceleration -"

"I never heard about anything like that," said my uncle.

When was the last time, in history, when it was possible for a single human to know the knowledge of the most advanced civilization? I've seen various estimates for this - usually in the form of polymaths nominated for the position of "last person to know everything". One plausible candidate is Leonardo da Vinci, who died in 1519 - shortly after the printing press began to become popular, and shortly before Copernicus inaugurated the scientific revolution.

In the ancestral environment it was possible to know everything, and nearly everyone did. In hunter-gatherer bands of less than 200 people, with no written literature, all background knowledge was universal knowledge. If one person, in a world containing 200 people total, discovered how gravity worked, you could certainly expect to hear about it.

In a world of 6 billion people, there is not one person alive who can say with certainty that science does not know a thing. There is too much science. Our current lifetimes are too short to learn more than a tiny fraction of it, and more is being produced all the time.

Even if last week's technical journal doesn't contain the answer to a mystery, that doesn't mean that no one knows it. Maybe someone out there is typing up the paper at this very moment. You can't generalize over all 6 billion people in the world because you haven't talked to all of them - which is a non-ancestral condition! For the vast majority of humanity's evolutionary history, it was possible to meet everyone in your little world. Now there's 6 billion people who might know the answer to any question you care to ask, and you can't ask all of them.

No one knows anymore what no one knows.

My uncle is not an isolated phenomenon. I've met people who think that science knows nothing about the brain, that thought is a complete mystery unto us. (My favorite was the fellow who confidently asserted that neuroscience had been unable to assign any function "to the cerebral cortex".) As Tom McCabe put it: "Anyone who claims that the brain is a total mystery should be slapped upside the head with the MIT Encyclopedia of the Cognitive Sciences. All one thousand ninety-six pages of it."

I haven't seen the movie What The Bleep Do We Know, but if the horror stories are true, it's one long celebration of imaginary ignorance. Particularly the "mysterious effect of conscious observation" in quantum physics, which was explained away as ordinary decoherence in the 1950s, but let's not get into that again.

Ignorance should not be celebrated in the first place; I've made this point before. It is a corruption of curiosity to prefer the question to its answer. Yet people seem to get a tremendous emotional kick out of not knowing something. Worse, they think that the mysteriousness of a mysterious phenomena indicates a special quality of the phenomenon itself, inferring that it is surely different-in-kind from phenomena labeled "understood". If we are ignorant about a phenomenon, that is a fact about our state of mind, not a fact about the phenomenon itself.

In the ancestral environment, there was a certain permanence to the division between ignorance and knowledge. If none of your fellow hunter-gatherers knew what made rain fall, it was likely that no one would ever find out in your grandchildren's lifetimes. Today, the absence of knowledge is a fragile and temporary condition, like the darkness in a closet whose door happens to be shut. A single thought can shatter the absence of thought. Every scientific discovery ever made, destroyed an ancient absence-of-knowledge dating back to the dawn of time. No one knows what 6 billion people don't know today, and still less does anyone know what 7 billion people will know tomorrow.

Referenced by: Evolutions Are Stupid (But Work Anyway) • Categorizing Has Consequences • Can You Prove Two Particles Are Identical? • The Conscious Sorites Paradox • Bell's Theorem: No EPR "Reality" • Many Worlds, One Best Guess • Einstein's Speed • Why Quantum? • Detached Lever Fallacy • Crisis of Faith • Entangled Truths, Contagious Lies • Intelligence in Economics • Building Something Smarter • Cascades, Cycles, Insight... • Recursive Self-Improvement • An Especially Elegant Evpsych Experiment • The Sacred Mundane • Incremental Progress and the Valley • That Crisis thing seems pretty useful • Bystander Apathy

Original with comments: No One Knows What Science Doesn't Know

Why Are Individual IQ Differences OK?

Eliezer Yudkowsky, 26 October 2007 09:50PM

Idang Alibi of Abuja, Nigeria writes on the James Watson affair:

A few days ago, the Nobel Laureate, Dr. James Watson, made a remark that is now generating worldwide uproar, especially among blacks. He said what to me looks like a self-evident truth. He told The Sunday Times of London in an interview that in his humble opinion, black people are less intelligent than the White people...

An intriguing opening. Is Idang Alibi about to take a position on the real heart of the uproar?

I do not know what constitutes intelligence. I leave that to our so-called scholars. But I do know that in terms of organising society for the benefit of the people living in it, we blacks have not shown any intelligence in that direction at all. I am so ashamed of this and sometimes feel that I ought to have belonged to another race...

Darn, it's just a lecture on personal and national responsibility. Of course, for African nationals, taking responsibility for their country's problems is the most productive attitude regardless. But it doesn't engage with the controversies that got Watson fired.

Later in the article came this:

As I write this, I do so with great pains in my heart because I know that God has given intelligence in equal measure to all his children irrespective of the colour of their skin.

This intrigued me for two reasons: First, I'm always on the lookout for yet another case of theology making a falsifiable experimental prediction. And second, the prediction follows obviously if God is just, but what does skin colour have to do with it at all?

A great deal has already been said about the Watson affair, and I suspect that in most respects I have little to contribute that has not been said before.

But why is it that the rest of the world seems to think that individual genetic differences are okay, whereas racial genetic differences in intelligence are not? Am I the only one who's every bit as horrified by the proposition that there's any way whatsoever to be screwed before you even start, whether it's genes or lead-based paint or Down's Syndrome? What difference does skin colour make? At all?

This is only half a rhetorical question. Race adds extra controversy to anything; in that sense, it's obvious what difference skin colour makes politically. However, just because this attitude is common, should not cause us to overlook its insanity. Some kind of different psychological processing is taking place around individually-unfair intelligence distributions, and group-unfair intelligence distributions.

So, in defiance of this psychological difference, and in defiance of politics, let me point out that a group injustice has no existence apart from injustice to individuals. It's individuals who have brains to experience suffering. It's individuals who deserve, and often don't get, a fair chance at life. If God has not given intelligence in equal measure to all his children, God stands convicted of a crime against humanity, period. Skin colour has nothing to do with it, nothing at all.

And I don't think there's any serious scholar of intelligence who disputes that God has been definitively shown to be most terribly unfair. Never mind the airtight case that intelligence has a hereditary genetic component among individuals; if you think that being born with Down's Syndrome doesn't impact life outcomes, then you are on crack. What about lead-based paint? Does it not count, because parents theoretically could have prevented it but didn't? In the beginning no one knew that it was damaging. How is it just for such a tiny mistake to have such huge, irrevocable consequences? And regardless, would not a just God damn us for only our own choices? Kids don't choose to live in apartments with lead-based paint.

So much for God being "just", unless you count the people whom God has just screwed over. Maybe that's part of the fuel in the burning controversy - that people do realize, on some level, the implications for religion. They can rationalize away the implications of a child born with no legs, but not a child born with no possibility of ever understanding calculus. But then this doesn't help explain the original observation, which is that people, for some odd reason, think that adding race makes it worse somehow.

And why is my own perspective, apparently, unusual? Perhaps because I also think that intelligence deficits will be fixable given sufficiently advanced technology, biotech or nanotech. When truly huge horrors are believed unfixable, the mind's eye tends to just skip over the hideous unfairness - for much the same reason you don't deliberately rest your hand on a hot stoveburner; it hurts.

Referenced by: Of Gender and Rationality

Original with comments: Why Are Individual IQ Differences OK?

Bay Area Bayesians Unite!

Eliezer Yudkowsky, 28 October 2007 12:07AM

Robin Hanson has his fellow GMU economists to talk to, but I'm not associated with a university and I live way out in the boondocks: the echoing emptiness of, er, Silicon Valley.

Overcoming Bias gets over 2000 visitors per day. Surely some of you are from the Bay Area. Would you be interested in a Bay Area meetup of Overcoming Bias readers?

Polls after the jump.
If you're interested at all, please vote in at least the closest-city poll.
Polls will be processed for a best-compromise value, not a binding modal result.
If I get at least 30 responses, I'll start looking into meetup locations.

Opinion Polls & Market Research

Opinion Polls&Market Research

Opinion Polls&Market Research

Referenced by: Classic Sichuan in Millbrae, Thu Feb 21, 7pm

Original with comments: Bay Area Bayesians Unite!

Motivated Stopping and Motivated Continuation

Eliezer Yudkowsky, 28 October 2007 11:10PM

Followup to: The Third Alternative, The Meditation on Curiosity

While I disagree with some views of the Fast and Frugal crowd - IMO they make a few too many lemons into lemonade - it also seems to me that they tend to develop the most psychologically realistic models of any school of decision theory. Most experiments present the subjects with options, and the subject chooses an option, and that's the experimental result. The frugalists realized that in real life, you have to generate your options, and they studied how subjects did that.

Likewise, although many experiments present evidence on a silver platter, in real life you have to gather evidence, which may be costly, and at some point decide that you have enough evidence to stop and choose. When you're buying a house, you don't get exactly 10 houses to choose from, and you aren't led on a guided tour of all of them before you're allowed to decide anything. You look at one house, and another, and compare them to each other; you adjust your aspirations - reconsider how much you really need to be close to your workplace and how much you're really willing to pay; you decide which house to look at next; and at some point you decide that you've seen enough houses, and choose.

Gilovich's distinction between motivated skepticism and motivated credulity highlights how conclusions a person does not want to believe are held to a higher standard than conclusions a person wants to believe. A motivated skeptic asks if the evidence compels them to accept the conclusion; a motivated credulist asks if the evidence allows them to accept the conclusion.

I suggest that an analogous bias in psychologically realistic search is motivated stopping and motivated continuation: when we have a hidden motive for choosing the "best" current option, we have a hidden motive to stop, and choose, and reject consideration of any more options. When we have a hidden motive to reject the current best option, we have a hidden motive to suspend judgment pending additional evidence, to generate more options - to find something, anything, to do instead of coming to a conclusion.

A major historical scandal in statistics was R. A. Fisher, an eminent founder of the field, insisting that no causal link had been established between smoking and lung cancer. "Correlation is not causation", he testified to Congress. Perhaps smokers had a gene which both predisposed them to smoke and predisposed them to lung cancer.

Or maybe Fisher being employed as a consultant for tobacco firms gave him a hidden motive to decide that the evidence already gathered was insufficient to come to a conclusion, and it was better to keep looking. Fisher was also a smoker himself, and died of colon cancer in 1962.

(Ad hominem note: Fisher was a frequentist. Bayesians are more reasonable about inferring probable causality.)

Like many other forms of motivated skepticism, motivated continuation can try to disguise itself as virtuous rationality. Who can argue against gathering more evidence ? I can. Evidence is often costly, and worse, slow, and there is certainly nothing virtuous about refusing to integrate the evidence you already have. You can always change your mind later. (Apparent contradiction resolved as follows: Spending one hour discussing the problem with your mind carefully cleared of all conclusions, is different from waiting ten years on another $20 million study.)

As for motivated stopping, it appears in every place a third alternative is feared, and wherever you have an argument whose obvious counterargument you would rather not see, and in other places as well. It appears when you pursue a course of action that makes you feel good just for acting, and so you'd rather not investigate how well your plan really worked, for fear of destroying the warm glow of moral satisfaction you paid good money to purchase. It appears wherever your beliefs and anticipations get out of sync, so you have a reason to fear any new evidence gathered.

The moral is that the decision to terminate a search procedure (temporarily or permanently) is, like the search procedure itself, subject to bias and hidden motives. You should suspect motivated stopping when you close off search, after coming to a comfortable conclusion, and yet there's a lot of fast cheap evidence you haven't gathered yet - Web sites you could visit, counter-counter arguments you could consider, or you haven't closed your eyes for five minutes by the clock trying to think of a better option. You should suspect motivated continuation when some evidence is leaning in a way you don't like, but you decide that more evidence is needed - expensive evidence that you know you can't gather anytime soon, as opposed to something you're going to look up on Google in 30 minutes - before you'll have to do anything uncomfortable.

Sequence: Against Rationalization

Avoiding Your Belief's Real Weak Points

A Case Study of Motivated Continuation

Referenced by: A Case Study of Motivated Continuation • Fake Justification • The Tragedy of Group Selectionism • Fake Utility Functions • Disputing Definitions • A Genius for Destruction • Contaminated by Optimism • Dreams of AI Design • Crisis of Faith • Ethical Injunctions • Total Nano Domination • Can Humanism Match Religion's Output? • That Crisis thing seems pretty useful

Original with comments: Motivated Stopping and Motivated Continuation

Torture vs. Dust Specks

Eliezer Yudkowsky, 30 October 2007 02:50AM

"What's the worst that can happen?" goes the optimistic saying. It's probably a bad question to ask anyone with a creative imagination. Let's consider the problem on an individual level: it's not really the worst that can happen, but would nonetheless be fairly bad, if you were horribly tortured for a number of years. This is one of the worse things that can realistically happen to one person in today's world.

What's the least bad, bad thing that can happen? Well, suppose a dust speck floated into your eye and irritated it just a little, for a fraction of a second, barely enough to make you notice before you blink and wipe away the dust speck.

For our next ingredient, we need a large number. Let's use 3^^^3, written in Knuth's up-arrow notation:

- 3^3 = 27.

- 3^^3 = (3^(3^3)) = 3^27 = 7625597484987.

- 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = (3^(3^(3^(... 7625597484987 times ...)))).

3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know.

Now here's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:

Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 people get dust specks in their eyes?

I think the answer is obvious. How about you?

Referenced by: A Case Study of Motivated Continuation • A Terrifying Halloween Costume • Not for the Sake of Happiness (Alone) • Infinite Certainty • Circular Altruism • Ethics Notes • Complexity and Intelligence • Continuous Improvement • (Moral) Truth in Fiction? • The Lifespan Dilemma

Original with comments: Torture vs. Dust Specks

A Case Study of Motivated Continuation

Eliezer Yudkowsky, 31 October 2007 01:27AM

I am not wholly unsympathetic to the many commenters in Torture vs. Dust Specks who argued that it is preferable to inflict dust specks upon the eyes of 3^^^3 (amazingly huge but finite number of) people, rather than torture one person for 50 years. If you think that a dust speck is simply of no account unless it has other side effects - if you literally do not prefer zero dust specks to one dust speck - then your position is consistent. (Though I suspect that many speckers would have expressed a preference if they hadn't known about the dilemma's sting.)

So I'm on board with the commenters who chose TORTURE, and I can understand the commenters who chose SPECKS.

But some of you said the question was meaningless; or that all morality was arbitrary and subjective; or that you needed more information before you could decide; or you talked about some other confusing aspect of the problem; and then you didn't go on to state a preference.

Sorry. I can't back you on that one.

If you actually answer the dilemma, then no matter which option you choose, you're giving something up. If you say SPECKS, you're giving up your claim on a certain kind of utilitarianism; you may worry that you're not being rational enough, or that others will accuse you of failing to comprehend large numbers. If you say TORTURE, you're accepting an outcome that has torture in it.

I falsifiably predict that of the commenters who dodged, most of them saw some specific answer - either TORTURE or SPECKS - that they flinched away from giving. Maybe for just a fraction of a second before the question-confusing operation took over, but I predict the flinch was there. (To be specific: I'm not predicting that you knew, and selected, and have in mind right now, some particular answer you're deliberately not giving. I'm predicting that your thinking trended toward a particular uncomfortable answer, for at least one fraction of a second before you started finding reasons to question the dilemma itself.)

In "bioethics" debates, you very often see experts on bioethics discussing what they see as the pros and cons of, say, stem-cell research; and then, at the conclusion of their talk, they gravely declare that more debate is urgently needed, with participation from all stakeholders. If you actually come to a conclusion, if you actually argue for banning stem cells, then people with relatives dying of Parkinson's will scream at you. If you come to a conclusion and actually endorse stem cells, religious fundamentalists will scream at you. But who can argue with a call to debate?

Uncomfortable with the way the evidence is trending on Darwinism versus creationism? Consider the issue soberly, and decide that you need more evidence; you want archaeologists to dig up another billion fossils before you come to a conclusion. That way you neither say something sacrilegious, nor relinquish your self-image as a rationalist. Keep on doing this with all issues that look like they might be trending in an uncomfortable direction, and you can maintain a whole religion in your mind.

Real life is often confusing, and we have to choose anyway, because refusing to choose is also a choice. The null plan is still a plan. We always do something, even if it's nothing. As Russell and Norvig put it, "Refusing to choose is like refusing to allow time to pass."

Ducking uncomfortable choices is a dangerous habit of mind. There are certain times when it's wise to suspend judgment (for an hour, not a year). When you're facing a dilemma all of whose answers seem uncomfortable, is not one of those times! Pick one of the uncomfortable answers as the best of an unsatisfactory lot. If there's missing information, fill in the blanks with plausible assumptions or probability distributions. Whatever it takes to overcome the basic flinch away from discomfort. Then you can search for an escape route.

Until you pick one interim best guess, the discomfort will consume your attention, distract you from the search, tempt you to confuse the issue whenever your analysis seems to trend in a particular direction.

In real life, when people flinch away from uncomfortable choices, they often hurt others as well as themselves. Refusing to choose is often one of the worst choices you can make. Motivated continuation is not a habit of thought anyone can afford, egoist or altruist. The cost of comfort is too high. It's important to acquire that habit of gritting your teeth and choosing - just as important as looking for escape routes afterward.

Sequence: Against Rationalization

Motivated Stopping and Motivated Continuation

Fake Justification

Original with comments: A Case Study of Motivated Continuation

A Terrifying Halloween Costume

Eliezer Yudkowsky, 01 November 2007 02:54AM

After the jump, you can see me dressed up as something so horrifyingly dreadful that it surpasses the comprehension of a mortal human mind.

Original with comments: A Terrifying Halloween Costume

Fake Justification

Eliezer Yudkowsky, 01 November 2007 03:57AM

Many Christians who've stopped really believing now insist that they revere the Bible as a source of ethical advice. The standard atheist reply is given by Sam Harris: "You and I both know that it would take us five minutes to produce a book that offers a more coherent and compassionate morality than the Bible does." Similarly, one may try to insist that the Bible is valuable as a literary work. Then why not revere Lord of the Rings, a vastly superior literary work? And despite the standard criticisms of Tolkien's morality, Lord of the Rings is at least superior to the Bible as a source of ethics. So why don't people wear little rings around their neck, instead of crosses? Even Harry Potter is superior to the Bible, both as a work of literary art and as moral philosophy. If I really wanted to be cruel, I would compare the Bible to Jacqueline Carey's Kushiel series.

"How can you justify buying a $1 million gem-studded laptop," you ask your friend, "when so many people have no laptops at all?" And your friend says, "But think of the employment that this will provide - to the laptop maker, the laptop maker's advertising agency - and then they'll buy meals and haircuts - it will stimulate the economy and eventually many people will get their own laptops." But it would be even more efficient to buy 5,000 OLPC laptops, thus providing employment to the OLPC manufacturers and giving out laptops directly.

I've touched before on the failure to look for third alternatives. But this is not really motivated stopping. Calling it "motivated stopping" would imply that there was a search carried out in the first place.

In The Bottom Line, I observed that only the real determinants of our beliefs can ever influence our real-world accuracy, only the real determinants of our actions can influence our effectiveness in achieving our goals. Someone who buys a million-dollar laptop was really thinking, "Ooh, shiny" and that was the one true causal history of their decision to buy a laptop. No amount of "justification" can change this, unless the justification is a genuine, newly running search process that can change the conclusion. Really change the conclusion. Most criticism carried out from a sense of duty is more of a token inspection than anything else. Free elections in a one-party country.

To genuinely justify the Bible as a lauding-object by reference to its literary quality, you would have to somehow perform a neutral reading through candidate books until you found the book of highest literary quality. Renown is one reasonable criteria for generating candidates, so I suppose you could legitimately end up reading Shakespeare, the Bible, and Godel, Escher, Bach. (Otherwise it would be quite a coincidence to find the Bible as a candidate, among a million other books.) The real difficulty is in that "neutral reading" part. Easy enough if you're not a Christian, but if you are...

But of course nothing like this happened. No search ever occurred. Writing the justification of "literary quality" above the bottom line of "I <heart> the Bible" is a historical misrepresentation of how the bottom line really got there, like selling cat milk as cow milk. That is just not where the bottom line really came from. That is just not what originally happened to produce that conclusion.

If you genuinely subject your conclusion to a criticism that can potentially de-conclude it - if the criticism genuinely has that power - then that does modify "the real algorithm behind" your conclusion. It changes the entanglement of your conclusion over possible worlds. But people overestimate, by far, how likely they really are to change their minds.

With all those open minds out there, you'd think there'd be more belief-updating.

Let me guess: Yes, you admit that you originally decided you wanted to buy a million-dollar laptop by thinking, "Ooh, shiny". Yes, you concede that this isn't a decision process consonant with your stated goals. But since then, you've decided that you really ought to spend your money in such fashion as to provide laptops to as many laptopless wretches as possible. And yet you just couldn't find any more efficient way to do this than buying a million-dollar diamond-studded laptop - because, hey, you're giving money to a laptop store and stimulating the economy! Can't beat that!

My friend, I am damned suspicious of this amazing coincidence. I am damned suspicious that the best answer under this lovely, rational, altruistic criterion X, is also the idea that just happened to originally pop out of the unrelated indefensible process Y. If you don't think that rolling dice would have been likely to produce the correct answer, then how likely is it to pop out of any other irrational cognition?

It's improbable that you used mistaken reasoning, yet made no mistakes.

Sequence: Against Rationalization

A Case Study of Motivated Continuation

Fake Optimization Criteria

Referenced by: An Alien God • The Tragedy of Group Selectionism • Fake Selfishness • Fake Optimization Criteria • Lost Purposes • Fake Fake Utility Functions • Fake Utility Functions • Anthropomorphic Optimism

Original with comments: Fake Justification

An Alien God

Eliezer Yudkowsky, 02 November 2007 06:57AM

"A curious aspect of the theory of evolution," said Jacques Monod, "is that everybody thinks he understands it."

A human being, looking at the natural world, sees a thousand times purpose. A rabbit's legs, built and articulated for running; a fox's jaws, built and articulated for tearing. But what you see is not exactly what is there...

In the days before Darwin, the cause of all this apparent purposefulness was a very great puzzle unto science. The Goddists said "God did it", because you get 50 bonus points each time you use the word "God" in a sentence. Yet perhaps I'm being unfair. In the days before Darwin, it seemed like a much more reasonable hypothesis. Find a watch in the desert, said William Paley, and you can infer the existence of a watchmaker.

But when you look at all the apparent purposefulness in Nature, rather than picking and choosing your examples, you start to notice things that don't fit the Judeo-Christian concept of one benevolent God. Foxes seem well-designed to catch rabbits. Rabbits seem well-designed to evade foxes. Was the Creator having trouble making up Its mind?

When I design a toaster oven, I don't design one part that tries to get electricity to the coils and a second part that tries to prevent electricity from getting to the coils. It would be a waste of effort. Who designed the ecosystem, with its predators and prey, viruses and bacteria? Even the cactus plant, which you might think well-designed to provide water fruit to desert animals, is covered with inconvenient spines.

The ecosystem would make much more sense if it wasn't designed by a unitary Who, but, rather, created by a horde of deities - say from the Hindu or Shinto religions. This handily explains both the ubiquitous purposefulnesses, and the ubiquitous conflicts: More than one deity acted, often at cross-purposes. The fox and rabbit were both designed, but by distinct competing deities. I wonder if anyone ever remarked on the seemingly excellent evidence thus provided for Hinduism over Christianity. Probably not.

Similarly, the Judeo-Christian God is alleged to be benevolent - well, sort of. And yet much of nature's purposefulness seems downright cruel. Darwin suspected a non-standard Creator for studying Ichneumon wasps, whose paralyzing stings preserve its prey to be eaten alive by its larvae: "I cannot persuade myself," wrote Darwin, "that a beneficent and omnipotent God would have designedly created the Ichneumonidae with the express intention of their feeding within the living bodies of Caterpillars, or that a cat should play with mice." I wonder if any earlier thinker remarked on the excellent evidence thus provided for Manichaen religions over monotheistic ones.

By now we all know the punchline: You just say "evolution".

I worry that's how some people are absorbing the "scientific" explanation, as a magical purposefulness factory in Nature. I've previously discussed the case of Storm from the movie X-Men, who in one mutation gets the ability to throw lightning bolts. Why? Well, there's this thing called "evolution" that somehow pumps a lot of purposefulness into Nature, and the changes happen through "mutations". So if Storm gets a really large mutation, she can be redesigned to throw lightning bolts. Radioactivity is a popular super origin: radiation causes mutations, so more powerful radiation causes more powerful mutations. That's logic.

But evolution doesn't allow just any kind of purposefulness to leak into Nature. That's what makes evolution a success as an empirical hypothesis. If evolutionary biology could explain a toaster oven, not just a tree, it would be worthless. There's a lot more to evolutionary theory than pointing at Nature and saying, "Now purpose is allowed," or "Evolution did it!" The strength of a theory is not what it allows, but what it prohibits; if you can invent an equally persuasive explanation for any outcome, you have zero knowledge.

"Many non-biologists," observed George Williams, "think that it is for their benefit that rattles grow on rattlesnake tails." Bzzzt! This kind of purposefulness is not allowed. Evolution doesn't work by letting flashes of purposefulness creep in at random - reshaping one species for the benefit of a random recipient.

Evolution is powered by a systematic correlation between the different ways that different genes construct organisms, and how many copies of those genes make it into the next generation. For rattles to grow on rattlesnake tails, rattle-growing genes must become more and more frequent in each successive generation. (Actually genes for incrementally more complex rattles, but if I start describing all the fillips and caveats to evolutionary biology, we really will be here all day.)

There isn't an Evolution Fairy that looks over the current state of Nature, decides what would be a "good idea", and chooses to increase the frequency of rattle-constructing genes.

I suspect this is where a lot of people get stuck, in evolutionary biology. They understand that "helpful" genes become more common, but "helpful" lets any sort of purpose leak in. They don't think there's an Evolution Fairy, yet they ask which genes will be "helpful" as if a rattlesnake gene could "help" non-rattlesnakes.

The key realization is that there is no Evolution Fairy. There's no outside force deciding which genes ought to be promoted. Whatever happens, happens because of the genes themselves.

Genes for constructing (incrementally better) rattles, must have somehow ended up more frequent in the rattlesnake gene pool, because of the rattle. In this case it's probably because rattlesnakes with better rattles survive more often - rather than mating more successfully, or having brothers that reproduce more successfully, etc.

Maybe predators are wary of rattles and don't step on the snake. Or maybe the rattle diverts attention from the snake's head. (As George Williams suggests, "The outcome of a fight between a dog and a viper would depend very much on whether the dog initially seized the reptile by the head or by the tail.")

But that's just a snake's rattle. There are much more complicated ways that a gene can cause copies of itself to become more frequent in the next generation. Your brother or sister shares half your genes. A gene that sacrifices one unit of resources to bestow three units of resource on a brother, may promote some copies of itself by sacrificing one of its constructed organisms. (If you really want to know all the details and caveats, buy a book on evolutionary biology; there is no royal road.)

The main point is that the gene's effect must cause copies of that gene to become more frequent in the next generation. There's no Evolution Fairy that reaches in from outside. There's nothing which decides that some genes are "helpful" and should, therefore, increase in frequency. It's just cause and effect, starting from the genes themselves.

This explains the strange conflicting purposefulness of Nature, and its frequent cruelty. It explains even better than a horde of Shinto deities.

Why is so much of Nature at war with other parts of Nature? Because there isn't one Evolution directing the whole process. There's as many different "evolutions" as reproducing populations. Rabbit genes are becoming more or less frequent in rabbit populations. Fox genes are becoming more or less frequent in fox populations. Fox genes which construct foxes that catch rabbits, insert more copies of themselves in the next generation. Rabbit genes which construct rabbits that evade foxes are naturally more common in the next generation of rabbits. Hence the phrase "natural selection".

Why is Nature cruel? You, a human, can look at an Ichneumon wasp, and decide that it's cruel to eat your prey alive. You can decide that if you're going to eat your prey alive, you can at least have the decency to stop it from hurting. It would scarcely cost the wasp anything to anesthetize its prey as well as paralyze it. Or what about old elephants, who die of starvation when their last set of teeth fall out? These elephants aren't going to reproduce anyway. What would it cost evolution - the evolution of elephants, rather - to ensure that the elephant dies right away, instead of slowly and in agony? What would it cost evolution to anesthetize the elephant, or give it pleasant dreams before it dies? Nothing; that elephant won't reproduce more or less either way.

If you were talking to a fellow human, trying to resolve a conflict of interest, you would be in a good negotiating position - would have an easy job of persuasion. It would cost so little to anesthetize the prey, to let the elephant die without agony! Oh please, won't you do it, kindly... um...

There's no one to argue with.

Human beings fake their justifications, figure out what they want using one method, and then justify it using another method. There's no Evolution of Elephants Fairy that's trying to (a) figure out what's best for elephants, and then (b) figure out how to justify it to the Evolutionary Overseer, who (c) doesn't want to see reproductive fitness decreased, but is (d) willing to go along with the painless-death idea, so long as it doesn't actually harm any genes.

There's no advocate for the elephants anywhere in the system.

Humans, who are often deeply concerned for the well-being of animals, can be very persuasive in arguing how various kindnesses wouldn't harm reproductive fitness at all. Sadly, the evolution of elephants doesn't use a similar algorithm; it doesn't select nice genes that can plausibly be argued to help reproductive fitness. Simply: genes that replicate more often become more frequent in the next generation. Like water flowing downhill, and equally benevolent.

A human, looking over Nature, starts thinking of all the ways we would design organisms. And then we tend to start rationalizing reasons why our design improvements would increase reproductive fitness - a political instinct, trying to sell your own preferred option as matching the boss's favored justification.

And so, amateur evolutionary biologists end up making all sorts of wonderful and completely mistaken predictions. Because the amateur biologists are drawing their bottom line - and more importantly, locating their prediction in hypothesis-space - using a different algorithm than evolutions use to draw their bottom lines.

A human engineer would have designed human taste buds to measure how much of each nutrient we had, and how much we needed. When fat was scarce, almonds or cheeseburgers would taste delicious. But if you started to become obese, or if vitamins were lacking, lettuce would taste delicious. But there is no Evolution of Humans Fairy, which intelligently planned ahead and designed a general system for every contingency. It was a reliable invariant of humans' ancestral environment that calories were scarce. So genes whose organisms loved calories, became more frequent. Like water flowing downhill.

We are simply the embodied history of which organisms did in fact survive and reproduce, not which organisms ought prudentially to have survived and reproduced.

The human retina is constructed backward: The light-sensitive cells are at the back, and the nerves emerge from the front and go back through the retina into the brain. Hence the blind spot. To a human engineer, this looks simply stupid - and other organisms have independently evolved retinas the right way around. Why not redesign the retina?

The problem is that no single mutation will reroute the whole retina simultaneously. A human engineer can redesign multiple parts simultaneously, or plan ahead for future changes. But if a single mutation breaks some vital part of the organism, it doesn't matter what wonderful things a Fairy could build on top of it - the organism dies and the genes decreases in frequency.

If you turn around the retina's cells without also reprogramming the nerves and optic cable, the system as a whole won't work. It doesn't matter that, to a Fairy or a human engineer, this is one step forward in redesigning the retina. The organism is blind. Evolution has no foresight, it is simply the frozen history of which organisms did in fact reproduce. Evolution is as blind as a halfway-redesigned retina.

Find a watch in a desert, said William Paley, and you can infer the watchmaker. There were once those who denied this, who thought that life "just happened" without need of an optimization process, mice being spontaneously generated from straw and dirty shirts.

If we ask who was more correct - the theologians who argued for a Creator-God, or the intellectually unfulfilled atheists who argued that mice spontaneously generated - then the theologians must be declared the victors: evolution is not God, but it is closer to God than it is to pure random entropy. Mutation is random, but selection is non-random. This doesn't mean an intelligent Fairy is reaching in and selecting. It means there's a non-zero statistical correlation between the gene and how often the organism reproduces. Over a few million years, that non-zero statistical correlation adds up to something very powerful. It's not a god, but it's more closely akin to a god than it is to snow on a television screen.

In a lot of ways, evolution is like unto theology. "Gods are ontologically distinct from creatures," said Damien Broderick, "or they're not worth the paper they're written on." And indeed, the Shaper of Life is not itself a creature. Evolution is bodiless, like the Judeo-Christian deity. Omnipresent in Nature, immanent in the fall of every leaf. Vast as a planet's surface. Billions of years old. Itself unmade, arising naturally from the structure of physics. Doesn't that all sound like something that might have been said about God?

And yet the Maker has no mind, as well as no body. In some ways, its handiwork is incredibly poor design by human standards. It is internally divided. Most of all, it isn't nice.

In a way, Darwin discovered God - a God that failed to match the preconceptions of theology, and so passed unheralded. If Darwin had discovered that life was created by an intelligent agent - a bodiless mind that loves us, and will smite us with lightning if we dare say otherwise - people would have said "My gosh! That's God!"

But instead Darwin discovered a strange alien God - not comfortably "ineffable", but really genuinely different from us. Evolution is not a God, but if it were, it wouldn't be Jehovah. It would be H. P. Lovecraft's Azathoth, the blind idiot God burbling chaotically at the center of everything, surrounded by the thin monotonous piping of flutes.

Which you might have predicted, if you had really looked at Nature.

So much for the claim some religionists make, that they believe in a vague deity with a correspondingly high probability. Anyone who really believed in a vague deity, would have recognized their strange inhuman creator when Darwin said "Aha!"

So much for the claim some religionists make, that they are waiting innocently curious for Science to discover God. Science has already discovered the sort-of-godlike maker of humans - but it wasn't what the religionists wanted to hear. They were waiting for the discovery of their God, the highly specific God they want to be there. They shall wait forever, for the great discovery has already taken place, and the winner is Azathoth.

Well, more power to us humans. I like having a Creator I can outwit. Beats being a pet. I'm glad it was Azathoth and not Odin.

(This post begins the Evolution sequence.)

Sequence: Evolution

The Wonder of Evolution

Referenced by: "Science" as Curiosity-Stopper • The Wonder of Evolution • Evolutions Are Stupid (But Work Anyway) • Natural Selection's Speed Limit and Complexity Bound • The Tragedy of Group Selectionism • Fake Optimization Criteria • Evolutionary Psychology • Thou Art Godshatter • Conjuring An Evolution To Serve You • Fake Fake Utility Functions • To Spread Science, Keep It Secret • Zombies! Zombies? • Where Philosophy Meets Science • My Childhood Role Model • A Premature Word on AI • Heading Toward Morality • Optimization and the Singularity • The Psychological Unity of Humankind • Where Recursive Justification Hits Bottom • Fundamental Doubts • Rebelling Within Nature • The Gift We Give To Tomorrow • Humans in Funny Suits • The Bedrock of Morality: Arbitrary? • Excluding the Supernatural • Above-Average AI Scientists • Why Does Power Corrupt? • Entangled Truths, Contagious Lies • Dark Side Epistemology • Ethical Inhibitions • Aiming at the Target • Failure By Analogy • ...Recursion, Magic • Total Nano Domination • Recursive Self-Improvement • Sympathetic Minds • Value is Fragile • The Thing That I Protect • You're Entitled to Arguments, But Not (That Particular) Proof

Original with comments: An Alien God

The Wonder of Evolution

Eliezer Yudkowsky, 02 November 2007 08:49PM

Followup to: An Alien God

The wonder of evolution is that it works at all.

I mean that literally: If you want to marvel at evolution, that's what's marvel-worthy.

How does optimization first arise in the universe? If an intelligent agent designed Nature, who designed the intelligent agent? Where is the first design that has no designer? The puzzle is not how the first stage of the bootstrap can be super-clever and super-efficient; the puzzle is how it can happen at all.

Evolution resolves the infinite regression, not by being super-clever and super-efficient, but by being stupid and inefficient and working anyway. This is the marvel.

For professional reasons, I often have to discuss the slowness, randomness, and blindness of evolution. Afterward someone says: "You just said that evolution can't plan simultaneous changes, and that evolution is very inefficient because mutations are random. Isn't that what the creationists say? That you couldn't assemble a watch by randomly shaking the parts in a box?"

But the reply to creationists is not that you can assemble a watch by shaking the parts in a box. The reply is that this is not how evolution works. If you think that evolution does work by whirlwinds assembling 747s, then the creationists have successfully misrepresented biology to you; they've sold the strawman.

The real answer is that complex machinery evolves either incrementally, or by adapting previous complex machinery used for a new purpose. Squirrels jump from treetop to treetop using just their muscles, but the length they can jump depends to some extent on the aerodynamics of their bodies. So now there are flying squirrels, so aerodynamic they can glide short distances. If birds were wiped out, the descendants of flying squirrels might reoccupy that ecological niche in ten million years, gliding membranes transformed into wings. And the creationists would say, "What good is half a wing? You'd just fall down and splat. How could squirrelbirds possibly have evolved incrementally?"

That's how one complex adaptation can jump-start a new complex adaptation. Complexity can also accrete incrementally, starting from a single mutation.

First comes some gene A which is simple, but at least a little useful on its own, so that A increases to universality in the gene pool. Now along comes gene B, which is only useful in the presence of A, but A is reliably present in the gene pool, so there's a reliable selection pressure in favor of B. Now a modified version of A* arises, which depends on B, but doesn't break B's dependency on A/A*. Then along comes C, which depends on A* and B, and B*, which depends on A* and C. Soon you've got "irreducibly complex" machinery that breaks if you take out any single piece.

And yet you can still visualize the trail backward to that single piece: you can, without breaking the whole machine, make one piece less dependent on another piece, and do this a few times, until you can take out one whole piece without breaking the machine, and so on until you've turned a ticking watch back into a crude sundial.

Here's an example: DNA stores information very nicely, in a durable format that allows for exact duplication. A ribosome turns that stored information into a sequence of amino acids, a protein, which folds up into a variety of chemically active shapes. The combined system, DNA and ribosome, can build all sorts of protein machinery. But what good is DNA, without a ribosome that turns DNA information into proteins? What good is a ribosome, without DNA to tell it which proteins to make?

Organisms don't always leave fossils, and evolutionary biology can't always figure out the incremental pathway. But in this case we do know how it happened. RNA shares with DNA the property of being able to carry information and replicate itself, although RNA is less durable and copies less accurately. And RNA also shares the ability of proteins to fold up into chemically active shapes, though it's not as versatile as the amino acid chains of proteins. Almost certainly, RNA is the single A which predates the mutually dependent A* and B.

It's just as important to note that RNA does the combined job of DNA and proteins poorly, as that it does the combined job at all. It's amazing enough that a single molecule can both store information and manipulate chemistry. For it to do the job well would be a wholly unnecessary miracle.

What was the very first replicator ever to exist? It may well have been an RNA strand, because by some strange coincidence, the chemical ingredients of RNA are chemicals that would have arisen naturally on the prebiotic Earth of 4 billion years ago. Please note: evolution does not explain the origin of life; evolutionary biology is not supposed to explain the first replicator, because the first replicator does not come from another replicator. Evolution describes statistical trends in replication. The first replicator wasn't a statistical trend, it was a pure accident. The notion that evolution should explain the origin of life is a pure strawman - more creationist misrepresentation.

If you'd been watching the primordial soup on the day of the first replicator, the day that reshaped the Earth, you would not have been impressed by how well the first replicator replicated. The first replicator probably copied itself like a drunken monkey on LSD. It would have exhibited none of the signs of careful fine-tuning embodied in modern replicators, because the first replicator was an accident. It was not needful for that single strand of RNA, or chemical hypercycle, or pattern in clay, to replicate gracefully. It just had to happen at all. Even so, it was probably very improbable, considered in an isolated event - but it only had to happen once, and there were a lot of tide pools. A few billions of years later, the replicators are walking on the moon.

The first accidental replicator was the most important molecule in the history of time. But if you praised it too highly, attributing to it all sorts of wonderful replication-aiding capabilities, you would be missing the whole point.

Don't think that, in the political battle between evolutionists and creationists, whoever praises evolution must be on the side of science. Science has a very exact idea of the capabilities of evolution. If you praise evolution one millimeter higher than this, you're not "fighting on evolution's side" against creationism. You're being scientifically inaccurate, full stop. You're falling into a creationist trap by insisting that, yes, a whirlwind does have the power to assemble a 747! Isn't that amazing! How wonderfully intelligent is evolution, how praiseworthy! Look at me, I'm pledging my allegiance to science! The more nice things I say about evolution, the more I must be on evolution's side against the creationists!

But to praise evolution too highly destroys the real wonder, which is not how well evolution designs things, but that a naturally occurring process manages to design anything at all.

So let us dispose of the idea that evolution is a wonderful designer, or a wonderful conductor of species destinies, which we human beings ought to imitate. For human intelligence to imitate evolution as a designer, would be like a sophisticated modern bacterium trying to imitate the first replicator as a biochemist. As T. H. Huxley, "Darwin's Bulldog", put it:

Let us understand, once and for all, that the ethical progress of society depends, not on imitating the cosmic process, still less in running away from it, but in combating it.

Huxley didn't say that because he disbelieved in evolution, but because he understood it all too well.

Sequence: Evolution

An Alien God

Evolutions Are Stupid (But Work Anyway)

Referenced by: Evolutions Are Stupid (But Work Anyway) • Natural Selection's Speed Limit and Complexity Bound • No Evolutions for Corporations or Nanodevices • Evaluability (And Cheap Holiday Shopping) • Uncritical Supercriticality • Fake Fake Utility Functions • Heading Toward Morality • The Psychological Unity of Humankind • Selecting Rationalist Groups

Original with comments: The Wonder of Evolution

Evolutions Are Stupid (But Work Anyway)

Eliezer Yudkowsky, 03 November 2007 03:45PM

Followup to: An Alien God, The Wonder of Evolution

Yesterday, I wrote:

Science has a very exact idea of the capabilities of evolution. If you praise evolution one millimeter higher than this, you're not "fighting on evolution's side" against creationism. You're being scientifically inaccurate, full stop.

In this post I describe some well-known inefficiencies and limitations of evolutions. I say "evolutions", plural, because fox evolution works at cross-purposes to rabbit evolution, and neither can talk to snake evolution to learn how to build venomous fangs.

So I am talking about limitations of evolution here, but this does not mean I am trying to sneak in creationism. This is standard Evolutionary Biology 201. (583 if you must derive the equations.) Evolutions, thus limited, can still explain observed biology; in fact the limitations are necessary to make sense of it. Remember that the wonder of evolutions is not how well they work, but that they work at all.

Human intelligence is so complicated that no one has any good way to calculate how efficient it is. Natural selection, though not simple, is simpler than a human brain; and correspondingly slower and less efficient, as befits the first optimization process ever to exist. In fact, evolutions are simple enough that we can calculate exactly how stupid they are.

Evolutions are slow. How slow? Suppose there's a beneficial mutation which conveys a fitness advantage of 3%: on average, bearers of this gene have 1.03 times as many children as non-bearers. Assuming that the mutation spreads at all, how long will it take to spread through the whole population? That depends on the population size. A gene conveying a 3% fitness advantage, spreading through a population of 100,000, would require an average of 768 generations to reach universality in the gene pool. A population of 500,000 would require 875 generations. The general formula is

- Generations to fixation = 2 ln(N) / s

where N is the population size, and (1 + s) is the fitness. (If each bearer of the gene has 1.03 times as many children as a non-bearer, s = 0.03.)

Thus, if the population size were 1,000,000 - the estimated population in hunter-gatherer times - then it would require 2763 generations for a gene conveying a 1% advantage to spread through the gene pool.1

This should not be surprising; genes have to do all their own work of spreading. There's no Evolution Fairy who can watch the gene pool and say, "Hm, that gene seems to be spreading rapidly - I should distribute it to everyone." In a human market economy, someone who is legitimately getting 20% returns on investment - especially if there's an obvious, clear mechanism behind it - can rapidly acquire more capital from other investors; and others will start duplicate enterprises. Genes have to spread without stock markets or banks or imitators - as if Henry Ford had to make one car, sell it, buy the parts for 1.01 more cars (on average), sell those cars, and keep doing this until he was up to a million cars.

All this assumes that the gene spreads in the first place. Here the equation is simpler and ends up not depending at all on population size:

- Probability of fixation = 2s

A mutation conveying a 3% advantage (which is pretty darned large, as mutations go) has a 6% chance of spreading, at least on that occasion.2 Mutations can happen more than once, but in a population of a million with a copying fidelity of 10^-8 errors per base per generation, you may have to wait a hundred generations for another chance, and then it still has an only 6% chance of fixating.

Still, in the long run, an evolution has a good shot at getting there eventually. (This is going to be a running theme.)

Complex adaptations take a very long time to evolve. First comes allele A, which is advantageous of itself, and requires a thousand generations to fixate in the gene pool. Only then can another allele B, which depends on A, begin rising to fixation. A fur coat is not a strong advantage unless the environment has a statistically reliable tendency to throw cold weather at you. Well, genes form part of the environment of other genes, and if B depends on A, B will not have a strong advantage unless A is reliably present in the genetic environment.

Let's say that B confers a 5% advantage in the presence of A, no advantage otherwise. Then while A is still at 1% frequency in the population, B only confers its advantage 1 out of 100 times, so the average fitness advantage of B is 0.05%, and B's probability of fixation is 0.1%. With a complex adaptation, first A has to evolve over a thousand generations, then B has to evolve over another thousand generations, then A* evolves over another thousand generations... and several million years later, you've got a new complex adaptation.

Then other evolutions don't imitate it. If snake evolution develops an amazing new venom, it doesn't help fox evolution or lion evolution.

Contrast all this to a human programmer, who can design a new complex mechanism with a hundred interdependent parts over the course of a single afternoon. How is this even possible? I don't know all the answer, and my guess is that neither does science; human brains are much more complicated than evolutions. I could wave my hands and say something like "goal-directed backward chaining using combinatorial modular representations", but you would not thereby be enabled to design your own human. Still: Humans can foresightfully design new parts in anticipation of later designing other new parts; produce coordinated simultaneous changes in interdependent machinery; learn by observing single test cases; zero in on problem spots and think abstractly about how to solve them; and prioritize which tweaks are worth trying, rather than waiting for a cosmic ray strike to produce a good one. By the standards of natural selection, this is simply magic.

Humans can do things that evolutions probably can't do period over the expected lifetime of the universe. As the eminent biologist Cynthia Kenyon once put it at a dinner I had the honor of attending, "One grad student can do things in an hour that evolution could not do in a billion years." According to biologists' best current knowledge, evolutions have invented a fully rotating wheel on a grand total of three occasions.

And don't forget the part where the programmer posts the code snippet to the Internet.

Yes, some evolutionary handiwork is impressive even by comparison to the best technology of Homo sapiens. But our Cambrian explosion only started, we only really began accumulating knowledge, around... what, four hundred years ago? In some ways, biology still excels over the best human technology: we can't build a self-replicating system the size of a butterfly. In other ways, human technology leaves biology in the dust. We got wheels, we got steel, we got guns, we got knives, we got pointy sticks; we got rockets, we got transistors, we got nuclear power plants. With every passing decade, that balance tips further.

So, once again: for a human to look to natural selection as inspiration on the art of design, is like a sophisticated modern bacterium trying to imitate the first awkward replicator's biochemistry. The first replicator would be eaten instantly if it popped up in today's competitive ecology. The same fate would accrue to any human planner who tried making random point mutations to their strategies and waiting 768 iterations of testing to adopt a 3% improvement.

Don't praise evolutions one millimeter more than they deserve.

Coming tomorrow: More exciting mathematical bounds on evolution!

1 Graur, D. and Li, W.H. 2000. Fundamentals of Molecular Evolution, 2nd edition. Sinauer Associates, Sunderland, MA.

2 Haldane, J. B. S. 1927. A mathematical theory of natural and artificial selection. IV. Proc. Camb. Philos. Soc. 23:607-615.

Sequence: Evolution

The Wonder of Evolution

Adaptation-Executers, not Fitness-Maximizers

Referenced by: Natural Selection's Speed Limit and Complexity Bound • Beware of Stephen J. Gould • Fake Optimization Criteria • Adaptation-Executers, not Fitness-Maximizers • Protein Reinforcement and DNA Consequentialism • Thou Art Godshatter • Evolving to Extinction • No Evolutions for Corporations or Nanodevices • Fake Fake Utility Functions • Searching for Bayes-Structure • Heading Toward Morality • Optimization and the Singularity • The Psychological Unity of Humankind • The Opposite Sex • Fundamental Doubts • The Meaning of Right • Interpersonal Morality • The Bedrock of Morality: Arbitrary? • Excluding the Supernatural • Optimization • Efficient Cross-Domain Optimization • Intelligence in Economics • Failure By Analogy • Surprised by Brains • ...Recursion, Magic • Recursive Self-Improvement • Artificial Mysterious Intelligence • True Sources of Disagreement • Growing Up is Hard • Value is Fragile

Original with comments: Evolutions Are Stupid (But Work Anyway)

Natural Selection's Speed Limit and Complexity Bound

Eliezer Yudkowsky, 04 November 2007 04:54PM

Followup to: An Alien God, The Wonder of Evolution, Evolutions Are Stupid

Yesterday, I wrote:

Humans can do things that evolutions probably can't do period over the expected lifetime of the universe. As the eminent biologist Cynthia Kenyon once put it at a dinner I had the honor of attending, "One grad student can do things in an hour that evolution could not do in a billion years." According to biologists' best current knowledge, evolutions have invented a fully rotating wheel on a grand total of three occasions.

But then, natural selection has not been running for a mere million years. It's been running for 3.85 billion years. That's enough to do something natural selection "could not do in a billion years" three times. Surely the cumulative power of natural selection is beyond human intelligence?

Not necessarily. There's a limit on how much complexity an evolution can support against the degenerative pressure of copying errors.

(Warning: A simulation I wrote to verify the following arguments did not return the expected results. See addendum and comments.)

(Addendum 2: This discussion has now been summarized in the Less Wrong Wiki. I recommend reading that instead.)

The vast majority of mutations are either neutral or detrimental; here we are focusing on detrimental mutations. At equilibrium, the rate at which a detrimental mutation is introduced by copying errors, will equal the rate at which it is eliminated by selection.

A copying error introduces a single instantiation of the mutated gene. A death eliminates a single instantiation of the mutated gene. (We'll ignore the possibility that it's a homozygote, etc; a failure to mate also works, etc.) If the mutation is severely detrimental, it will be eliminated very quickly - the embryo might just fail to develop. But if the mutation only leads to a 0.01% probability of dying, it might spread to 10,000 people before one of them died. On average, one detrimental mutation leads to one death; the weaker the selection pressure against it, the more likely it is to spread. Again, at equilibrium, copying errors will introduce mutations at the same rate that selection eliminates them. One mutation, one death.

This means that you need the same amount of selection pressure to keep a gene intact, whether it's a relatively important gene or a relatively unimportant one. The more genes are around, the more selection pressure required. Under too much selection pressure - too many children eliminated in each generation - a species will die out.

We can quantify selection pressure as follows: Suppose that 2 parents give birth to an average of 16 children. On average all but 2 children must either die or fail to reproduce. Otherwise the species population very quickly goes to zero or infinity. From 16 possibilities, all but 2 are eliminated - we can call this 3 bits of selection pressure. Not bits like bytes on a hard drive, but mathematician's bits, information-theoretical bits; one bit is the ability to eliminate half the possibilities. This is the speed limit on evolution.

Among mammals, it's safe to say that the selection pressure per generation is on the rough order of 1 bit. Yes, many mammals give birth to more than 4 children, but neither does selection perfectly eliminate all but the most fit organisms. The speed limit on evolution is an upper bound, not an average.

This 1 bit per generation has to be divided up among all the genetic variants being selected on, for the whole population. It's not 1 bit per organism per generation, it's 1 bit per gene pool per generation. Suppose there's some amazingly beneficial mutation making the rounds, so that organisms with the mutation have 50% more offspring. And suppose there's another less beneficial mutation, that only contributes 1% to fitness. Very often, an organism that lacks the 1% mutation, but has the 50% mutation, will outreproduce another who has the 1% mutation but not the 50% mutation.

There are limiting forces on variance; going from 10 to 20 children is harder than going from 1 to 2 children. There's only so much selection to go around, and beneficial mutations compete to be promoted by it (metaphorically speaking). There's an upper bound, a speed limit to evolution: If Nature kills off a grand total of half the children, then the gene pool of the next generation can acquire a grand total of 1 bit of information.

I am informed that this speed limit holds even with semi-isolated breeding subpopulations, sexual reproduction, chromosomal linkages, and other complications.

Let's repeat that. It's worth repeating. A mammalian gene pool can acquire at most 1 bit of information per generation.

Among mammals, the rate of DNA copying errors is roughly 10^-8 per base per generation. Copy a hundred million DNA bases, and on average, one will copy incorrectly. One mutation, one death; each non-junk base of DNA soaks up the same amount of selection pressure to counter the degenerative pressure of copying errors. It's a truism among biologists that most selection pressure goes toward maintaining existing genetic information, rather than promoting new mutations.

Natural selection probably hit its complexity bound no more than a hundred million generations after multicellular organisms got started. Since then, over the last 600 million years, evolutions have substituted new complexity for lost complexity, rather than accumulating adaptations. Anyone who doubts this should read George Williams's classic "Adaptation and Natural Selection", which treats the point at much greater length.

In material terms, a Homo sapiens genome contains roughly 3 billion bases. We can see, however, that mammalian selection pressures aren't going to support 3 billion bases of useful information. This was realized on purely mathematical grounds before "junk DNA" was discovered, before the Genome Project announced that humans probably had only 20-25,000 protein-coding genes. Yes, there's genetic information that doesn't code for proteins - all sorts of regulatory regions and such. But it is an excellent bet that nearly all the DNA which appears to be junk, really is junk. Because, roughly speaking, an evolution isn't going to support more than 10^8 meaningful bases with 1 bit of selection pressure and a 10^-8 error rate.

Each base is 2 bits. A byte is 8 bits. So the meaningful DNA specifying a human must fit into at most 25 megabytes.

(Pause.)

Yes. Really.

And the Human Genome Project gave the final confirmation. 25,000 genes plus regulatory regions will fit in 100,000,000 bases with lots of room to spare.

Amazing, isn't it?

Addendum: genetics.py, a simple Python program that simulates mutation and selection in a sexually reproducing population, is failing to match the result described above. Sexual recombination is random, each pair of parents have 4 children, and the top half of the population is selected each time. Wei Dai rewrote the program in C++ and reports that the supportable amount of genetic information increases as the inverse square of the mutation rate(?!) which if generally true would make it possible for the entire human genome to be meaningful.

In the above post, George Williams's arguments date back to 1966, and the result that the human genome contains <25,000 protein-coding regions comes from the Genome Project. The argument that 2 parents having 16 children with 2 surviving implies a speed limit of 3 bits per generation was found here, and I understand that it dates back to Kimura's work in the 1950s. However, the attempt to calculate a specific bound of 25 megabytes was my own.

It's possible that the simulation contains a bug, or that I used unrealistic assumptions. If the entire human genome of 3 billion DNA bases could be meaningful, it's not clear why it would contain <25,000 genes. Empirically, an average of O(1) bits of genetic information per generation seems to square well with observed evolutionary times; we don't actually see species gaining thousands of bits per generation. There is also no reason to believe that a dog has greater morphological or biochemical complexity than a dinosaur. In short, only the math I tried to calculate myself should be regarded as having failed, not the beliefs that are wider currency in evolutionary biology. But until I understand what's going on, I would suggest citing only George Williams's arguments and the Genome Project result, not the specific mathematical calculation shown above.

Referenced by: Beware of Stephen J. Gould • The Simple Math of Everything • Observing Optimization

Original with comments: Natural Selection's Speed Limit and Complexity Bound

Beware of Stephen J. Gould

Eliezer Yudkowsky, 06 November 2007 05:22AM

Followup to: Natural Selection's Speed Limit and Complexity Bound

If you've read anything Stephen J. Gould has ever said about evolutionary biology, I have some bad news for you. In the field of evolutionary biology at large, Gould's reputation is mud. Not because he was wrong. Many honest scientists have made honest mistakes. What Gould did was much worse, involving deliberate misrepresentation of science.

In his 1996 book Full House: The Spread of Excellence from Plato to Darwin, Stephen J. Gould explains how modern evolutionary biology is very naive about evolutionary progress. Foolish evolutionary biologists, says Gould, believe that evolution has a preferred tendency toward progress and the accumulation of complexity. But of course - Gould kindly explains - this is simply a statistical illusion, bolstered by the tendency to cite hand-picked sequences like bacteria, fern, dinosaurs, dog, man. You could equally well explain this apparent progress by supposing that evolution is undergoing a random walk, sometimes losing complexity and sometimes gaining it. If so, Gould says, there will be a left bound, a minimum at zero complexity, but no right bound, and the most complex organisms will seem to grow more complex over time. Even though it's really just a random walk with no preference in either direction, the distribution widens and the tail gets longer.

What romantics, ha ha, those silly evolutionary biologists, believing in progress! It's a good thing we had a statistically sophisticated thinker like Stephen J. Gould to keep their misconceptions from infecting the general public. Indeed, Stephen J. Gould was a hero - a martyr - because evolutionary biologists don't like it when you challenge their romantic preconceptions, and they persecuted him. Or so Gould represented himself to the public.

There's just one problem: It's extremely unlikely that any modern evolutionary theorist, however much a romantic, would believe that evolution was accumulating complexity.

There was once a time when many evolutionary biologists had a romantic conception of progress, evolution climbing ever-higher mountains of complexity, dinosaur to dog to man. And there was a hero who challenged that widespread misconception. The hero was George Williams, his challenge was successful, and his reputation rests securely in evolutionary biology today.

In a population at equilibrium, harmful mutations will be eliminated by death (or failure to reproduce) at the same rate they are introduced by copying errors. A very severe mutation may be eliminated by an embryo that fails to develop, but a mutation that's lethal only one time out of 10,000 may spread to 10,000 people before it starts to be eliminated. It takes the same amount of selection pressure to support minor or major adaptations; whether the adaptation was a big one or a small one, at equilibrium, mutations must be eliminated at the same rate they are introduced by copying errors.

A population cannot sustain too high a selection pressure - too many deaths or failures to reproduce - without dying out. And it requires the same amount of selection to support any given amount of DNA against the degenerative pressure of copying errors. This, in turn, implies an upper bound on the amount of DNA that can be sustained by selection against the degenerative pressure of copying errors.

The upshot, as George Williams wrote:

A certain amount of information is added by selection every generation. At the same time, a certain amount is subtracted by randomizing processes. The more information is already stored, the more would mutation and other random forces reduce it in a given time interval. It is reasonable to suppose that there would be a maximum level of information content that could be maintained by selection in opposition to randomizing forces...

The view suggested here is that all organisms of above a certain low level organization - perhaps that of the simpler invertebrates - and beyond a certain geological period - perhaps the Cambrian - may have much the same amounts of [meaningful] information in their nuclei.

Saying this did not make Williams a heroic, persecuted martyr. He simply won. His arguments were accepted and biology moved on. The book quoted above is Adaptation and Natural Selection, now showing its age but still considered a great classic. The shift to a gene's-eye-view in evolutionary theory is sometimes called the "Williams Revolution", the other founders being Hamilton, John Maynard Smith, Trivers, and Dawkins as popularizer. In short, Williams was not exactly Mr. Obscure.

And Williams wrote in 1966, thirty years before Gould wrote Full House.

If Gould had simply stolen Williams's ideas and presented them as his own, then he would have been guilty of plagiarism. And yet at least the general public would have been accurately informed; in that sense, less damage would have been done to the public understanding of science.

But Gould's actual conduct was much stranger. He wrote as if the entire Williams revolution had never occurred! Gould attacked, as if they were still current views, romantic notions that no serious biologist had put forth since the 1960s. Then Gould presented his own counterarguments to these no-longer-advocated views, and they were bizarre. Evolution is a random walk in complexity, with a minimum at zero complexity and no upper bound? But there is an upper bound! Sheer chance explains why dogs are more complex than dinosaurs? But they probably aren't!

Why did Gould behave thus? Two observations: One, to bring order to a scientific field, it must first be in chaos. Two, plagiarism is a crime that everyone understands.

Gould undid the last thirty years of progress in his depiction of the field he was criticizing, pretending that evolutionary theory was in chaos, so he could depict himself as heroically bringing order to it. If Gould had also redid the accepted solutions as his own, he would have been caught, tried, and cast out of the public eye. Newspaper editors may not be interested in mathematical arguments about evolutionary biology, but they understand stories about plagiarism and theft. Once Gould's copying had been laid out next to the original, and eminent scientists attested to the identity, it would have been over.

So instead Gould committed a crime so bizarre that it couldn't be explained to editors. He stole Williams's chaos.

(Incidentally, Gould's notion of a random walk in complexity has the same quality as the rest of his argument. A genome acquires a beneficial allele at a readily calculable speed and probability, and until the complexity reaches equilibrium, new adaptations will tend to be acquired faster than old adaptations are lost to copying errors or environmental shifts. The fewer adaptations have been acquired by a genome, the fewer are likely to be lost to a given event. If complexity starts far below the equilibrium level, it will tend to increase.)

All this that I have said, was a common pattern throughout Gould's "work". And all this that I have said, is no news to professional biologists. Here's John Maynard Smith:

"Gould occupies a rather curious position, particularly on his side of the Atlantic. Because of the excellence of his essays, he has come to be seen by non-biologists as the preeminent evolutionary theorist. In contrast, the evolutionary biologists with whom I have discussed his work tend to see him as a man whose ideas are so confused as to be hardly worth bothering with, but as one who should not be publicly criticized because he is at least on our side against the creationists. All this would not matter, were it not that he is giving non-biologists a largely false picture of the state of evolutionary theory."

John Maynard Smith was a genuinely great evolutionary biologist, the sort of man that Gould pretended to be. But some readers may have to take my word for this, since the names of eminent scientists are often less well-known to the general public than the names of fast-talking scoundrels such as Uri Geller or Stephen J. Gould.

I am not calling Gould a scoundrel because he was wrong; honest scientists can make honest mistakes. But Gould systematically misrepresented what other scientists thought; he deluded the public as to what evolutionary biologists were thinking.

It is as if someone presented geocentric epicycles as the current belief in 21st-century astronomy, sharply criticized the complexity of all those circles orbiting circles, and argued for their own simpler model of planets that move in straight lines.

Did Gould deliberately lie? If not, he executed one of the most epic feats of self-deception in the history of marketing. The eminent John Tooby speaks:

"Although Gould characterizes his critics as "anonymous" and "a tiny coterie," nearly every major evolutionary biologist of our era has weighed in in a vain attempt to correct the tangle of confusions that the higher profile Gould has inundated the intellectual world with. The point is not that Gould is the object of some criticism -- so properly are we all -- it is that his reputation as a credible and balanced authority about evolutionary biology is non-existent among those who are in a professional position to know...
These [major evolutionary biologists] include Ernst Mayr, John Maynard Smith, George Williams, Bill Hamilton, Richard Dawkins, E.O. Wilson, Tim Clutton-Brock, Paul Harvey, Brian Charlesworth, Jerry Coyne, Robert Trivers, John Alcock, Randy Thornhill, and many others."

If Gould, after receiving that many corrections, managed to still not know the actually current beliefs in evolutionary biology, he must have had neutronium earplugs. I'm not saying it's impossible, though, because it's amazing what people can not-know when their reputation depends on it. But there comes a point in self-deception where it becomes morally indistinguishable from lying. Consistently self-serving scientific "error", in the face of repeated correction and without informing others of the criticism, blends over into scientific fraud.

And after all this, Gould is widely believed, by the general public and even by many scientists outside evolutionary biology, to be an evolutionary theorist of honorable reputation! It is as if Immanuel Velikovsky had managed to make himself into the public face of astronomy.

If you have read one of Gould's books, you are not to blame; but you must now do your best to un-believe it all - especially all the implied beliefs in evolutionary biology that Gould seemed to be attacking.

And so as not to be accused of plagiarism myself, many others have said much of what I said here - only in politer academic language, with longer sentences, and without that specific example. I thought it deserved a sharper presentation, for the benefit of the popular audience that Gould deluded; and a clear-cut example of Gould's "work", to show what the fuss is about. Many academic writers on Gould could not speak as sharply as Gould deserved. As I have no fear for my own reputation, I will say it plainly: One way or another, knowingly or unknowingly, Gould deceived the trusting public and committed the moral equivalent of deliberate scientific fraud.

Original with comments: Beware of Stephen J. Gould

The Tragedy of Group Selectionism

Eliezer Yudkowsky, 07 November 2007 07:47AM

Before 1966, it was not unusual to see serious biologists advocating evolutionary hypotheses that we would now regard as magical thinking. These muddled notions played an important historical role in the development of later evolutionary theory, error calling forth correction; like the folly of English kings provoking into existence the Magna Carta and constitutional democracy.

As an example of romance, Vero Wynne-Edwards, Warder Allee, and J. L. Brereton, among others, believed that predators would voluntarily restrain their breeding to avoid overpopulating their habitat and exhausting the prey population.

But evolution does not open the floodgates to arbitrary purposes. You cannot explain a rattlesnake's rattle by saying that it exists to benefit other animals who would otherwise be bitten. No outside Evolution Fairy decides when a gene ought to be promoted; the gene's effect must somehow directly cause the gene to be more prevalent in the next generation. It's clear why our human sense of aesthetics, witnessing a population crash of foxes who've eaten all the rabbits, cries "Something should've been done!" But how would a gene complex for restraining reproduction - of all things! - cause itself to become more frequent in the next generation?

A human being designing a neat little toy ecology - for entertainment purposes, like a model railroad - might be annoyed if their painstakingly constructed fox and rabbit populations self-destructed by the foxes eating all the rabbits and then dying of starvation themselves. So the human would tinker with the toy ecology - a fox-breeding-restrainer is the obvious solution that leaps to our human minds - until the ecology looked nice and neat. Nature has no human, of course, but that needn't stop us - now that we know what we want on aesthetic grounds, we just have to come up with a plausible argument that persuades Nature to want the same thing on evolutionary grounds.

Obviously, selection on the level of the individual won't produce individual restraint in breeding. Individuals who reproduce unrestrainedly will, naturally, produce more offspring than individuals who restrain themselves.

(Addendum: Individual selection will not produce individual sacrifice of breeding opportunities. Individual selection can certainly produce individuals who, after acquiring all available resources, use those resources to produce 4 big eggs instead of 8 small eggs - not to conserve social resources, but because that is the individual sweet spot for number of eggs * egg survival probability. This does not get rid of the commons problem.)

But suppose that the species population was broken up into subpopulations, which were mostly isolated, and only occasionally interbred. Then, surely, subpopulations that restrained their breeding would be less likely to go extinct, and would send out more messengers, and create new colonies to reinhabit the territories of crashed populations.

The problem with this scenario wasn't that it was mathematically impossible. The problem was that it was possible but very difficult.

The fundamental problem is that it's not only restrained breeders who reap the benefits of restrained breeding. If some foxes refrain from spawning cubs who eat rabbits, then the uneaten rabbits don't go to only cubs who carry the restrained-breeding adaptation. The unrestrained foxes, and their many more cubs, will happily eat any rabbits left unhunted. The only way the restraining gene can survive against this pressure, is if the benefits of restraint preferentially go to restrainers.

Specifically, the requirement is C/B < FST where C is the cost of altruism to the donor, B is the benefit of altruism to the recipient, and FST is the spatial structure of the population: the average relatedness between a randomly selected organism and its randomly selected neighbor, where a "neighbor" is any other fox who benefits from an altruistic fox's restraint. (I believe this is a derivation with different symbols, best one I could find online.)

So is the cost of restrained breeding sufficiently small, and the empirical benefit of less famine sufficiently large, compared to the empirical spatial structure of fox populations and rabbit populations, that the group selection argument can work?

The math suggests this is pretty unlikely. In this simulation, for example, the cost to altruists is 3% of fitness, pure altruist groups have a fitness twice as great as pure selfish groups, the subpopulation size is 25, and 20% of all deaths are replaced with messengers from another group: the result is polymorphic for selfishness and altruism. If the subpopulation size is doubled to 50, selfishness is fixed; if the cost to altruists is increased to 6%, selfishness is fixed; if the altruistic benefit is decreased by half, selfishness is fixed or in large majority. Neighborhood-groups must be very small, with only around 5 members, for group selection to operate when the cost of altruism exceeds 10%. This doesn't seem plausibly true of foxes restraining their breeding.

You can guess by now, I think, that the group selectionists ultimately lost the scientific argument. The kicker was not the mathematical argument, but empirical observation: foxes didn't restrain their breeding (I forget the exact species of dispute; it wasn't foxes and rabbits), and indeed, predator-prey systems crash all the time. Group selectionism would later revive, somewhat, in drastically different form - mathematically speaking, there is neighborhood structure, which implies nonzero group selection pressure not necessarily capable of overcoming countervailing individual selection pressure, and if you don't take it into account your math will be wrong, full stop. And evolved enforcement mechanisms (not originally postulated) change the game entirely. So why is this now-historical scientific dispute worthy material for Overcoming Bias?

A decade after the controversy, a biologist had a fascinating idea. The mathematical conditions for group selection overcoming individual selection were too extreme to be found in Nature. Why not create them artificially, in the laboratory? Michael J. Wade proceeded to do just that, repeatedly selecting populations of insects for low numbers of adults per subpopulation. And what was the result? Did the insects restrain their breeding and live in quiet peace with enough food for all?

No; the adults adapted to cannibalize eggs and larvae, especially female larvae.

Of course selecting for small subpopulation sizes would not select for individuals who restrained their own breeding; it would select for individuals who ate other individuals' children. Especially the girls.

Once you have that experimental result in hand - and it's massively obvious in retrospect - then it suddenly becomes clear how the original group selectionists allowed romanticism, a human sense of aesthetics, to cloud their predictions of Nature.

This is an archetypal example of a missed Third Alternative, resulting from a rationalization of a predetermined bottom line which produced a fake justification and then motivatedly stopped. The group selectionists didn't start with clear, fresh minds, happen upon the idea of group selection, and neutrally extrapolate forward the probable outcome. They started out with the beautiful idea of fox populations voluntarily restraining their reproduction to what the rabbit population would bear, Nature in perfect harmony; then they searched for a reason why this would happen, and came up with the idea of group selection; then, since they knew what they wanted the outcome of group selection to be, they didn't look for any less beautiful and aesthetic adaptations that group selection would be more likely to promote instead. If they'd really been trying to calmly and neutrally predict the result of selecting for small subpopulation sizes resistant to famine, they would have thought of cannibalizing other organisms' children or some similarly "ugly" outcome - long before they imagined anything so evolutionarily outré as individual restraint in breeding!

This also illustrates the point I was trying to make in Einstein's Arrogance: With large answer spaces, nearly all of the real work goes into promoting one possible answer to the point of being singled out for attention. If a hypothesis is improperly promoted to your attention - your sense of aesthetics suggests a beautiful way for Nature to be, and yet natural selection doesn't involve an Evolution Fairy who shares your appreciation - then this alone may seal your doom, unless you can manage to clear your mind entirely and start over.

In principle, the world's stupidest person may say the Sun is shining, but that doesn't make it dark out. Even if an answer is suggested by a lunatic on LSD, you should be able to neutrally calculate the evidence for and against, and if necessary, un-believe.

In practice, the group selectionists were doomed because their bottom line was originally suggested by their sense of aesthetics, and Nature's bottom line was produced by natural selection. These two processes had no principled reason for their outputs to correlate, and indeed they didn't. All the furious argument afterward didn't change that.

If you start with your own desires for what Nature should do, consider Nature's own observed reasons for doing things, and then rationalize an extremely persuasive argument for why Nature should produce your preferred outcome for Nature's own reasons, then Nature, alas, still won't listen. The universe has no mind and is not subject to clever political persuasion. You can argue all day why gravity should really make water flow uphill, and the water just ends up in the same place regardless. It's like the universe plain isn't listening. J. R. Molloy said: "Nature is the ultimate bigot, because it is obstinately and intolerantly devoted to its own prejudices and absolutely refuses to yield to the most persuasive rationalizations of humans."

I often recommend evolutionary biology to friends just because the modern field tries to train its students against rationalization, error calling forth correction. Physicists and electrical engineers don't have to be carefully trained to avoid anthropomorphizing electrons, because electrons don't exhibit mindish behaviors. Natural selection creates purposefulnesses which are alien to humans, and students of evolutionary theory are warned accordingly. It's good training for any thinker, but it is especially important if you want to think clearly about other weird mindish processes that do not work like you do.

Sequence: Evolution

Evolving to Extinction

Fake Optimization Criteria

Referenced by: Fake Selfishness • Fake Optimization Criteria • Conjuring An Evolution To Serve You • The Hidden Complexity of Wishes • Lost Purposes • Fake Fake Utility Functions • A Failed Just-So Story • Heading Toward Morality • The Psychological Unity of Humankind • Interpersonal Morality • Anthropomorphic Optimism • Contaminated by Optimism • Ethical Inhibitions • Selecting Rationalist Groups

Original with comments: The Tragedy of Group Selectionism

Fake Selfishness

Eliezer Yudkowsky, 08 November 2007 02:31AM

Followup to: Fake Justification

Once upon a time, I met someone who proclaimed himself to be purely selfish, and told me that I should be purely selfish as well. I was feeling mischievous(*) that day, so I said, "I've observed that with most religious people, at least the ones I meet, it doesn't matter much what their religion says, because whatever they want to do, they can find a religious reason for it. Their religion says they should stone unbelievers, but they want to be nice to people, so they find a religious justification for that instead. It looks to me like when people espouse a philosophy of selfishness, it has no effect on their behavior, because whenever they want to be nice to people, they can rationalize it in selfish terms."

And the one said, "I don't think that's true."

I said, "If you're genuinely selfish, then why do you want me to be selfish too? Doesn't that make you concerned for my welfare? Shouldn't you be trying to persuade me to be more altruistic, so you can exploit me?"

The one replied: "Well, if you become selfish, then you'll realize that it's in your rational self-interest to play a productive role in the economy, instead of, for example, passing laws that infringe on my private property."

And I said, "But I'm a small-L libertarian already, so I'm not going to support those laws. And since I conceive of myself as an altruist, I've taken a job that I expect to benefit a lot of people, including you, instead of a job that pays more. Would you really benefit more from me if I became selfish? Besides, is trying to persuade me to be selfish the most selfish thing you could be doing? Aren't there other things you could do with your time that would bring much more direct benefits? But what I really want to know is this: Did you start out by thinking that you wanted to be selfish, and then decide this was the most selfish thing you could possibly do? Or did you start out by wanting to convert others to selfishness, then look for ways to rationalize that as self-benefiting?"

And the one said, "You may be right about that last part," so I marked him down as intelligent.

(*) Other mischievous questions to ask self-proclaimed Selfishes: "Would you sacrifice your own life to save the entire human species?" (If they notice that their own life is strictly included within the human species, you can specify that they can choose between dying immediately to save the Earth, or living in comfort for one more year and then dying along with Earth.) Or, taking into account that scope insensitivity leads many people to be more concerned over one life than the Earth, "If you had to choose one event or the other, would you rather that you stubbed your toe, or that the stranger standing near the wall there gets horribly tortured for fifty years?" (If they say that they'd be emotionally disturbed by knowing, specify that they won't know about the torture.) "Would you steal a thousand dollars from Bill Gates if you could be guaranteed that neither he nor anyone else would ever find out about it?" (Selfish libertarians only.)

Referenced by: Fake Morality • Lost Purposes • Fake Fake Utility Functions • Science Doesn't Trust Your Rationality • Changing Your Metaethics

Original with comments: Fake Selfishness

Fake Morality

Eliezer Yudkowsky, 08 November 2007 09:32PM

Followup to: Fake Selfishness

God, say the religious fundamentalists, is the source of all morality; there can be no morality without a Judge who rewards and punishes. If we did not fear hell and yearn for heaven, then what would stop people from murdering each other left and right?

Suppose Omega makes a credible threat that if you ever step inside a bathroom between 7AM and 10AM in the morning, he'll kill you. Would you be panicked by the prospect of Omega withdrawing his threat? Would you cower in existential terror and cry: "If Omega withdraws his threat, then what's to keep me from going to the bathroom?" No; you'd probably be quite relieved at your increased opportunity to, ahem, relieve yourself.

Which is to say: The very fact that a religious person would be afraid of God withdrawing Its threat to punish them for committing murder, shows that they have a revulsion of murder which is independent of whether God punishes murder or not. If they had no sense that murder was wrong independently of divine retribution, the prospect of God not punishing murder would be no more existentially horrifying than the prospect of God not punishing sneezing.

If Overcoming Bias has any religious readers left, I say to you: it may be that you will someday lose your faith: and on that day, you will not lose all sense of moral direction. For if you fear the prospect of God not punishing some deed, that is a moral compass. You can plug that compass directly into your decision system and steer by it. You can simply not do whatever you are afraid God may not punish you for doing. The fear of losing a moral compass is itself a moral compass. Indeed, I suspect you are steering by that compass, and that you always have been. As Piers Anthony once said, "Only those with souls worry over whether or not they have them." s/soul/morality/ and the point carries.

You don't hear religious fundamentalists using the argument: "If we did not fear hell and yearn for heaven, then what would stop people from eating pork?" Yet by their assumptions - that we have no moral compass but divine reward and retribution - this argument should sound just as forceful as the other.

Even the notion that God threatens you with eternal hellfire, rather than cookies, piggybacks on a pre-existing negative value for hellfire. Consider the following, and ask which of these two philosophers is really the altruist, and which is really selfish?

"You should be selfish, because when people set out to improve society, they meddle in their neighbors' affairs and pass laws and seize control and make everyone unhappy. Take whichever job that pays the most money: the reason the job pays more is that the efficient market thinks it produces more value than its alternatives. Take a job that pays less, and you're second-guessing what the market thinks will benefit society most."

"You should be altruistic, because the world is an iterated Prisoner's Dilemma, and the strategy that fares best is Tit for Tat with initial cooperation. People don't like jerks. Nice guys really do finish first. Studies show that people who contribute to society and have a sense of meaning in their lives, are happier than people who don't; being selfish will only make you unhappy in the long run."

Blank out the recommendations of these two philosophers, and you can see that the first philosopher is using strictly prosocial criteria to justify his recommendations; to him, what validates an argument for selfishness is showing that selfishness benefits everyone. The second philosopher appeals to strictly individual and hedonic criteria; to him, what validates an argument for altruism is showing that altruism benefits him as an individual: higher social status or more intense feelings of pleasure.

So which of these two is the actual altruist? Whichever one actually holds open doors for little old ladies.

Referenced by: Leaky Generalizations • Lost Purposes • Fake Fake Utility Functions • Fake Utility Functions • Guardians of Ayn Rand • Changing Your Metaethics • Prolegomena to a Theory of Fun

Original with comments: Fake Morality

Fake Optimization Criteria

Eliezer Yudkowsky, 10 November 2007 12:10AM

Followup to: Fake Justification, The Tragedy of Group Selectionism

I've previously dwelt in considerable length upon forms of rationalization whereby our beliefs appear to match the evidence much more strongly than they actually do. And I'm not overemphasizing the point, either. If we could beat this fundamental metabias and see what every hypothesis really predicted, we would be able to recover from almost any other error of fact.

The mirror challenge for decision theory is seeing which option a choice criterion really endorses. If your stated moral principles call for you to provide laptops to everyone, does that really endorse buying a $1 million gem-studded laptop for yourself, or spending the same money on shipping 5000 OLPCs?

We seem to have evolved a knack for arguing that practically any goal implies practically any action. A phlogiston theorist explaining why magnesium gains weight when burned has nothing on an Inquisitor explaining why God's infinite love for all His children requires burning some of them at the stake.

There's no mystery about this. Politics was a feature of the ancestral environment. We are descended from those who argued most persuasively that the good of the tribe meant executing their hated rival Uglak. (We sure ain't descended from Uglak.)

And yet... is it possible to prove that if Robert Mugabe cared only for the good of Zimbabwe, he would resign from its presidency? You can argue that the policy follows from the goal, but haven't we just seen that humans can match up any goal to any policy? How do you know that you're right and Mugabe is wrong? (There are a number of reasons this is a good guess, but bear with me here.)

Human motives are manifold and obscure, our decision processes as vastly complicated as our brains. And the world itself is vastly complicated, on every choice of real-world policy. Can we even prove that human beings are rationalizing - that we're systematically distorting the link from principles to policy - when we lack a single firm place on which to stand? When there's no way to find out exactly what even a single optimization criterion implies? (Actually, you can just observe that people disagree about office politics in ways that strangely correlate to their own interests, while simultaneously denying that any such interests are at work. But again, bear with me here.)

Where is the standardized, open-source, generally intelligent, consequentialist optimization process into which we can feed a complete morality as an XML file, to find out what that morality really recommends when applied to our world? Is there even a single real-world case where we can know exactly what a choice criterion recommends? Where is the pure moral reasoner - of known utility function, purged of all other stray desires that might distort its optimization - whose trustworthy output we can contrast to human rationalizations of the same utility function?

Why, it's our old friend the alien god, of course! Natural selection is guaranteed free of all mercy, all love, all compassion, all aesthetic sensibilities, all political factionalism, all ideological allegiances, all academic ambitions, all libertarianism, all socialism, all Blue and all Green. Natural selection doesn't maximize its criterion of inclusive genetic fitness - it's not that smart. But when you look at the output of natural selection, you are guaranteed to be looking at an output that was optimized only for inclusive genetic fitness, and not the interests of the US agricultural industry.

In the case histories of evolutionary science - in, for example, The Tragedy of Group Selectionism - we can directly compare human rationalizations to the result of pure optimization for a known criterion. What did Wynne-Edwards think would be the result of group selection for small subpopulation sizes? Voluntary individual restraint in breeding, and enough food for everyone. What was the actual laboratory result? Cannibalism.

Now you might ask: Are these case histories of evolutionary science really relevant to human morality, which doesn't give two figs for inclusive genetic fitness when it gets in the way of love, compassion, aesthetics, healing, freedom, fairness, et cetera? Human societies didn't even have a concept of "inclusive genetic fitness" until the 20th century.

But I ask in return: If we can't see clearly the result of a single monotone optimization criterion - if we can't even train ourselves to hear a single pure note - then how will we listen to an orchestra? How will we see that "Always be selfish" or "Always obey the government" are poor guiding principles for human beings to adopt - if we think that even optimizing genes for inclusive fitness will yield organisms which sacrifice reproductive opportunities in the name of social resource conservation?

To train ourselves to see clearly, we need simple practice cases.

Sequence: Against Rationalization

Fake Justification

Is That Your True Rejection?

Sequence: Evolution

The Tragedy of Group Selectionism

Referenced by: Thou Art Godshatter • The Hidden Complexity of Wishes • Lost Purposes • Fake Fake Utility Functions • Fake Utility Functions • Guardians of Ayn Rand • GAZP vs. GLUT • Heading Toward Morality • Anthropomorphic Optimism

Original with comments: Fake Optimization Criteria

Adaptation-Executers, not Fitness-Maximizers

Eliezer Yudkowsky, 11 November 2007 06:39AM

"Individual organisms are best thought of as adaptation-executers rather than as fitness-maximizers."
-- John Tooby and Leda Cosmides, The Psychological Foundations of Culture.

Fifty thousand years ago, the taste buds of Homo sapiens directed their bearers to the scarcest, most critical food resources - sugar and fat. Calories, in a word. Today, the context of a taste bud's function has changed, but the taste buds themselves have not. Calories, far from being scarce (in First World countries), are actively harmful. Micronutrients that were reliably abundant in leaves and nuts are absent from bread, but our taste buds don't complain. A scoop of ice cream is a superstimulus, containing more sugar, fat, and salt than anything in the ancestral environment.

No human being with the deliberate goal of maximizing their alleles' inclusive genetic fitness, would ever eat a cookie unless they were starving. But individual organisms are best thought of as adaptation-executers, not fitness-maximizers.

A toaster, though its designer intended it to make toast, does not bear within it the intelligence of the designer - it won't automatically redesign and reshape itself if you try to cram in an entire loaf of bread. A Phillips-head screwdriver won't reconform itself to a flat-head screw. We created these tools, but they exist independently of us, and they continue independently of us.

The atoms of a screwdriver don't have tiny little XML tags inside describing their "objective" purpose. The designer had something in mind, yes, but that's not the same as what happens in the real world. If you forgot that the designer is a separate entity from the designed thing, you might think, "The purpose of the screwdriver is to drive screws" - as though this were an explicit property of the screwdriver itself, rather than a property of the designer's state of mind. You might be surprised that the screwdriver didn't reconfigure itself to the flat-head screw, since, after all, the screwdriver's purpose is to turn screws.

The cause of the screwdriver's existence is the designer's mind, which imagined an imaginary screw, and imagined an imaginary handle turning. The actual operation of the screwdriver, its actual fit to an actual screw head, cannot be the objective cause of the screwdriver's existence: The future cannot cause the past. But the designer's brain, as an actually existent thing within the past, can indeed be the cause of the screwdriver.

The consequence of the screwdriver's existence, may not correspond to the imaginary consequences in the designer's mind. The screwdriver blade could slip and cut the user's hand.

And the meaning of the screwdriver - why, that's something that exists in the mind of a user, not in tiny little labels on screwdriver atoms. The designer may intend it to turn screws. A murderer may buy it to use as a weapon. And then accidentally drop it, to be picked up by a child, who uses it as a chisel.

So the screwdriver's cause, and its shape, and its consequence, and its various meanings, are all different things; and only one of these things is found within the screwdriver itself.

Where do taste buds come from? Not from an intelligent designer visualizing their consequences, but from a frozen history of ancestry: Adam liked sugar and ate an apple and reproduced, Barbara liked sugar and ate an apple and reproduced, Charlie liked sugar and ate an apple and reproduced, and 2763 generations later, the allele became fixed in the population. For convenience of thought, we sometimes compress this giant history and say: "Evolution did it." But it's not a quick, local event like a human designer visualizing a screwdriver. This is the objective cause of a taste bud.

What is the objective shape of a taste bud? Technically, it's a molecular sensor connected to reinforcement circuitry. This adds another level of indirection, because the taste bud isn't directly acquiring food. It's influencing the organism's mind, making the organism want to eat foods that are similar to the food just eaten.

What is the objective consequence of a taste bud? In a modern First World human, it plays out in multiple chains of causality: from the desire to eat more chocolate, to the plan to eat more chocolate, to eating chocolate, to getting fat, to getting fewer dates, to reproducing less successfully. This consequence is directly opposite the key regularity in the long chain of ancestral successes which caused the taste bud's shape. But, since overeating has only recently become a problem, no significant evolution (compressed regularity of ancestry) has further influenced the taste bud's shape.

What is the meaning of eating chocolate? That's between you and your moral philosophy. Personally, I think chocolate tastes good, but I wish it were less harmful; acceptable solutions would include redesigning the chocolate or redesigning my biochemistry.

Smushing several of the concepts together, you could sort-of-say, "Modern humans do today what would have propagated our genes in a hunter-gatherer society, whether or not it helps our genes in a modern society." But this still isn't quite right, because we're not actually asking ourselves which behaviors would maximize our ancestors' inclusive fitness. And many of our activities today have no ancestral analogue. In the hunter-gatherer society there wasn't any such thing as chocolate.

So it's better to view our taste buds as an adaptation fitted to ancestral conditions that included near-starvation and apples and roast rabbit, which modern humans execute in a new context that includes cheap chocolate and constant bombardment by advertisements.

Therefore it is said: Individual organisms are best thought of as adaptation-executers, not fitness-maximizers.

Sequence: Evolution

Evolutions Are Stupid (But Work Anyway)

No Evolutions for Corporations or Nanodevices

Referenced by: Evolutionary Psychology • Thou Art Godshatter • Fake Fake Utility Functions • Fake Utility Functions • The Two-Party Swindle • Stop Voting For Nincompoops • Scarcity • Is Morality Given? • The Gift We Give To Tomorrow • The Bedrock of Morality: Arbitrary? • Three Fallacies of Teleology • The True Prisoner's Dilemma • Why Does Power Corrupt? • Ethical Inhibitions • Continuous Improvement • In Praise of Boredom • Value is Fragile • An Especially Elegant Evpsych Experiment

Original with comments: Adaptation-Executers, not Fitness-Maximizers

Evolutionary Psychology

Eliezer Yudkowsky, 11 November 2007 08:41PM

Followup to: An Alien God, Adaptation-Executers not Fitness-Maximizers

Like "IRC chat" or "TCP/IP protocol", the phrase "reproductive organ" is redundant. All organs are reproductive organs. Where do a bird's wings come from? An Evolution-of-Birds Fairy who thinks that flying is really neat? The bird's wings are there because they contributed to the bird's ancestors' reproduction. Likewise the bird's heart, lungs, and genitals. At most we might find it worthwhile to distinguish between directly reproductive organs and indirectly reproductive organs.

This observation holds true also of the brain, the most complex organ system known to biology. Some brain organs are directly reproductive, like lust; others are indirectly reproductive, like anger.

Where does the human emotion of anger come from? An Evolution-of-Humans Fairy who thought that anger was a worthwhile feature? The neural circuitry of anger is a reproductive organ as surely as your liver. Anger exists in Homo sapiens because angry ancestors had more kids. There's no other way it could have gotten there.

This historical fact about the origin of anger confuses all too many people. They say, "Wait, are you saying that when I'm angry, I'm subconsciously trying to have children? That's not what I'm thinking after someone punches me in the nose."

No. No. No. NO!

Individual organisms are best thought of as adaptation-executers, not fitness-maximizers. The cause of an adaptation, the shape of an adaptation, and the consequence of an adaptation, are all separate things. If you built a toaster, you wouldn't expect the toaster to reshape itself when you tried to cram in a whole loaf of bread; yes, you intended it to make toast, but that intention is a fact about you, not a fact about the toaster. The toaster has no sense of its own purpose.

But a toaster is not an intention-bearing object. It is not a mind at all, so we are not tempted to attribute goals to it. If we see the toaster as purposed, we don't think the toaster knows it, because we don't think the toaster knows anything.

It's like the old test of being asked to say the color of the letters in "blue". It takes longer for subjects to name this color, because of the need to untangle the meaning of the letters and the color of the letters. You wouldn't have similar trouble naming the color of the letters in "wind".

But a human brain, in addition to being an artifact historically produced by evolution, is also a mind capable of bearing its own intentions, purposes, desires, goals, and plans. Both a bee and a human are designs, but only a human is a designer. The bee is "wind", the human is "blue".

Cognitive causes are ontologically distinct from evolutionary causes. They are made out of a different kind of stuff. Cognitive causes are made of neurons. Evolutionary causes are made of ancestors.

The most obvious kind of cognitive cause is deliberate, like an intention to go to the supermarket, or a plan for toasting toast. But an emotion also exists physically in the brain, as a train of neural impulses or a cloud of spreading hormones. Likewise an instinct, or a flash of visualization, or a fleetingly suppressed thought; if you could scan the brain in three dimensions and you understood the code, you would be able to see them.

Even subconscious cognitions exist physically in the brain. "Power tends to corrupt," observed Lord Acton. Stalin may or may not have believed himself an altruist, working toward the greatest good for the greatest number. But it seems likely that, somewhere in Stalin's brain, there were neural circuits that reinforced pleasurably the exercise of power, and neural circuits that detected anticipations of increases and decreases in power. If there were nothing in Stalin's brain that correlated to power - no little light that went on for political command, and off for political weakness - then how could Stalin's brain have known to be corrupted by power?

Evolutionary selection pressures are ontologically distinct from the biological artifacts they create. The evolutionary cause of a bird's wings is millions of ancestor-birds who reproduced more often than other ancestor-birds, with statistical regularity owing to their possession of incrementally improved wings compared to their competitors. We compress this gargantuan historical-statistical macrofact by saying "evolution did it".

Natural selection is ontologically distinct from creatures; evolution is not a little furry thing lurking in an undiscovered forest. Evolution is a causal, statistical regularity in the reproductive history of ancestors.

And this logic applies also to the brain. Evolution has made wings that flap, but do not understand flappiness. It has made legs that walk, but do not understand walkyness. Evolution has carved bones of calcium ions, but the bones themselves have no explicit concept of strength, let alone inclusive genetic fitness. And evolution designed brains themselves capable of designing; yet these brains had no more concept of evolution than a bird has of aerodynamics. Until the 20th century, not a single human brain explicitly represented the complex abstract concept of inclusive genetic fitness.

When we're told that "The evolutionary purpose of anger is to increase inclusive genetic fitness," there's a tendency to slide to "The purpose of anger is reproduction" to "The cognitive purpose of anger is reproduction." No! The statistical regularity of ancestral history isn't in the brain, even subconsciously, any more than the designer's intentions of toast are in a toaster!

Thinking that your built-in anger-circuitry embodies an explicit desire to reproduce, is like thinking your hand is an embodied mental desire to pick things up.

Your hand is not wholly cut off from your mental desires. In particular circumstances, you can control the flexing of your fingers by an act of will. If you bend down and pick up a penny, then this may represent an act of will; but it is not an act of will that made your hand grow in the first place.

One must distinguish a one-time event of particular anger (anger-1, anger-2, anger-3) from the underlying neural circuitry for anger. An anger-event is a cognitive cause, and an anger-event may have cognitive causes, but you didn't will the anger-circuitry to be wired into the brain.

So you have to distinguish the event of anger, from the circuitry of anger, from the gene complex which laid down the neural template, from the ancestral macrofact which explains the gene complex's presence.

If there were ever a discipline that genuinely demanded X-Treme Nitpicking, it is evolutionary psychology.

Consider, O my readers, this sordid and joyful tale: A man and a woman meet in a bar. The man is attracted to her clear complexion and firm breasts, which would have been fertility cues in the ancestral environment, but which in this case result from makeup and a bra. This does not bother the man; he just likes the way she looks. His clear-complexion-detecting neural circuitry does not know that its purpose is to detect fertility, any more than the atoms in his hand contain tiny little XML tags reading "<purpose>pick things up</purpose>". The woman is attracted to his confident smile and firm manner, cues to high status, which in the ancestral environment would have signified the ability to provide resources for children. She plans to use birth control, but her confident-smile-detectors don't know this any more than a toaster knows its designer intended it to make toast. She's not concerned philosophically with the meaning of this rebellion, because her brain is a creationist and denies vehemently that evolution exists. He's not concerned philosophically with the meaning of this rebellion, because he just wants to get laid. They go to a hotel, and undress. He puts on a condom, because he doesn't want kids, just the dopamine-noradrenaline rush of sex, which reliably produced offspring 50,000 years ago when it was an invariant feature of the ancestral environment that condoms did not exist. They have sex, and shower, and go their separate ways. The main objective consequence is to keep the bar and the hotel and condom-manufacturer in business; which was not the cognitive purpose in their minds, and has virtually nothing to do with the key statistical regularities of reproduction 50,000 years ago which explain how they got the genes that built their brains that executed all this behavior.

To reason correctly about evolutionary psychology you must simultaneously consider many complicated abstract facts that are strongly related yet importantly distinct, without a single mixup or conflation.

Referenced by: Protein Reinforcement and DNA Consequentialism • Thou Art Godshatter • The Hidden Complexity of Wishes • Lost Purposes • Resist the Happy Death Spiral • Fake Fake Utility Functions • Fake Utility Functions • The Robbers Cave Experiment • Lonely Dissent • The Two-Party Swindle • Stop Voting For Nincompoops • Rational vs. Scientific Ev-Psych • Mind Projection Fallacy • Savanna Poets • Faster Than Science • Principles of Disagreement • Heading Toward Morality • The Psychological Unity of Humankind • The Moral Void • Is Morality Preference? • Is Morality Given? • Where Recursive Justification Hits Bottom • Rebelling Within Nature • Probability is Subjectively Objective • The Gift We Give To Tomorrow • Changing Your Metaethics • Interpersonal Morality • Detached Lever Fallacy • Three Fallacies of Teleology • Dreams of Friendliness • Why Does Power Corrupt? • Ethical Inhibitions • Efficient Cross-Domain Optimization • Dunbar's Function • Emotional Involvement • Sympathetic Minds • Interpersonal Entanglement • Value is Fragile • Cynicism in Ev-Psych (and Econ?) • Your Price for Joining • Collective Apathy and the Internet • Of Gender and Rationality

Original with comments: Evolutionary Psychology

Protein Reinforcement and DNA Consequentialism

Eliezer Yudkowsky, 13 November 2007 01:34AM

Followup to: Evolutionary Psychology

It takes hundreds of generations for a simple beneficial mutation to promote itself to universality in a gene pool. Thousands of generations, or even millions, to create complex interdependent machinery.

That's some slow learning there. Let's say you're building a squirrel, and you want the squirrel to know locations for finding nuts. Individual nut trees don't last for the thousands of years required for natural selection. You're going to have to learn using proteins. You're going to have to build a brain.

Protein computers and sensors can learn by looking, much faster than DNA can learn by mutation and selection. And yet (until very recently) the protein learning machines only learned in narrow, specific domains. Squirrel brains learn to find nut trees, but not to build gliders - as flying squirrel DNA is slowly learning to do. The protein computers learned faster than DNA, but much less generally.

How the heck does a double-stranded molecule that fits inside a cell nucleus, come to embody truths that baffle a whole damn squirrel brain?

Consider the high-falutin' abstract thinking that modern evolutionary theorists do in order to understand how adaptations increase inclusive genetic fitness. Reciprocal altruism, evolutionarily stable strategies, deterrence, costly signaling, sexual selection - how many humans explicitly represent this knowledge? Yet DNA can learn it without a protein computer.

There's a long chain of causality whereby a male squirrel, eating a nut today, produces more offspring months later: Chewing and swallowing food, to digesting food, to burning some calories today and turning others into fat, to burning the fat through the winter, to surviving the winter, to mating with a female, to the sperm fertilizing an egg inside the female, to the female giving birth to an offspring that shares 50% of the squirrel's genes.

With the sole exception of humans, no protein brain can imagine chains of causality that long, that abstract, and crossing that many domains. With one exception, no protein brain is even capable of drawing the consequential link from chewing and swallowing to inclusive reproductive fitness.

Yet natural selection exploits links between local actions and distant reproductive benefits. In wide generality, across domains, and through levels of abstraction that confuse some humans. Because - of course - the basic evolutionary idiom works through the actual real-world consequences, avoiding the difficulty of having a brain imagine them.

Naturally, this also misses the efficiency of having a brain imagine consequences. It takes millions of years and billions of dead bodies to build complex machines this way. And if you want to memorize the location of a nut tree, you're out of luck.

Gradually DNA acquired the ability to build protein computers, brains, that could learn small modular facets of reality like the location of nut trees. To call these brains "limited" implies that a speed limit was tacked onto a general learning device, which isn't what happened. It's just that the incremental successes of particular mutations tended to build out into domain-specific nut-tree-mapping programs. (If you know how to program, you can verify for yourself that it's easier to build a nut-tree-mapper than an Artificial General Intelligence.)

One idiom that brain-building DNA seems to have hit on, over and over, is reinforcement learning - repeating policies similar to policies previously rewarded. If a food contains lots of calories and doesn't make you sick, then eat more foods that have similar tastes. This doesn't require a brain that visualizes the whole chain of digestive causality.

Reinforcement learning isn't trivial: You've got to chop up taste space into neighborhoods of similarity, and stick a sensor in the stomach to detect calories or indigestion, and do some kind of long-term-potentiation that strengthens the eating impulse. But it seems much easier for evolution to hit on reinforcement learning, than a brain that accurately visualizes the digestive system, let alone a brain that accurately visualizes the reproductive consequences N months later.

(This efficiency does come at a price: If the environment changes, making food no longer scarce and famines improbable, the organisms may go on eating food until they explode.)

Similarly, a bird doesn't have to cognitively model the airflow over its wings. It just has to track which wing-flapping policies cause it to lurch.

Why not learn to like food based on reproductive success, so that you'll stop liking the taste of candy if it stops leading to reproductive success? Why don't birds wait and see which wing-flapping policies result in more eggs, not just more stability?

Because it takes too long. Reinforcement learning still requires you to wait for the detected consequences before you learn.

Now, if a protein brain could imagine the consequences, accurately, it wouldn't need a reinforcement sensor that waited for them to actually happen.

Put a food reward in a transparent box. Put the corresponding key, which looks unique and uniquely corresponds to that box, in another transparent box. Put the key to that box in another box. Do this with five boxes. Mix in another sequence of five boxes that doesn't lead to a food reward. Then offer a choice of two keys, one which starts the sequence of five boxes leading to food, one which starts the sequence leading nowhere.

Chimpanzees can learn to do this. (Dohl 1970.) So consequentialist reasoning, backward chaining from goal to action, is not strictly limited to Homo sapiens.

But as far as I know, no non-primate species can pull that trick. And working with a few transparent boxes is nothing compared to the kind of high-falutin' cross-domain reasoning you would need to causally link food to inclusive fitness. (Never mind linking reciprocal altruism to inclusive fitness). Reinforcement learning seems to evolve a lot more easily.

When natural selection builds a digestible-calorie-sensor linked by reinforcement learning to taste, then the DNA itself embodies the implicit belief that calories lead to reproduction. So the long-term, complicated, cross-domain, distant link from calories to reproduction, is learned by natural selection - it's implicit in the reinforcement learning mechanism that uses calories as a reward signal.

Only short-term consequences, which the protein brains can quickly observe and easily learn from, get hooked up to protein learning. The DNA builds a protein computer that seeks calories, rather than, say, chewiness. Then the protein computer learns which tastes are caloric. (Oversimplified, I know. Lots of inductive hints embedded in this machinery.)

But the DNA had better hope that its protein computer never ends up in an environment where calories are bad for it... or where sexual pleasure stops correlating to reproduction... or where there are marketers that intelligently reverse-engineer reward signals...

Referenced by: Thou Art Godshatter • Fake Fake Utility Functions • Optimization and the Singularity • Ethical Inhibitions • Emotional Involvement

Original with comments: Protein Reinforcement and DNA Consequentialism

Thou Art Godshatter

Eliezer Yudkowsky, 13 November 2007 07:38PM

Followup to: An Alien God, Adaptation-Executers not Fitness-Maximizers, Evolutionary Psychology

Before the 20th century, not a single human being had an explicit concept of "inclusive genetic fitness", the sole and absolute obsession of the blind idiot god. We have no instinctive revulsion of condoms or oral sex. Our brains, those supreme reproductive organs, don't perform a check for reproductive efficacy before granting us sexual pleasure.

Why not? Why aren't we consciously obsessed with inclusive genetic fitness? Why did the Evolution-of-Humans Fairy create brains that would invent condoms? "It would have been so easy," thinks the human, who can design new complex systems in an afternoon.

The Evolution Fairy, as we all know, is obsessed with inclusive genetic fitness. When she decides which genes to promote to universality, she doesn't seem to take into account anything except the number of copies a gene produces. (How strange!)

But since the maker of intelligence is thus obsessed, why not create intelligent agents - you can't call them humans - who would likewise care purely about inclusive genetic fitness? Such agents would have sex only as a means of reproduction, and wouldn't bother with sex that involved birth control. They could eat food out of an explicitly reasoned belief that food was necessary to reproduce, not because they liked the taste, and so they wouldn't eat candy if it became detrimental to survival or reproduction. Post-menopausal women would babysit grandchildren until they became sick enough to be a net drain on resources, and would then commit suicide.

It seems like such an obvious design improvement - from the Evolution Fairy's perspective.

Now it's clear, as was discussed yesterday, that it's hard to build a powerful enough consequentialist. Natural selection sort-of reasons consequentially, but only by depending on the actual consequences. Human evolutionary theorists have to do really high-falutin' abstract reasoning in order to imagine the links between adaptations and reproductive success.

But human brains clearly can imagine these links in protein. So when the Evolution Fairy made humans, why did It bother with any motivation except inclusive genetic fitness?

It's been less than two centuries since a protein brain first represented the concept of natural selection. The modern notion of "inclusive genetic fitness" is even more subtle, a highly abstract concept. What matters is not the number of shared genes. Chimpanzees share 95% of your genes. What matters is shared genetic variance, within a reproducing population - your sister is one-half related to you, because any variations in your genome, within the human species, are 50% likely to be shared by your sister.

Only in the last century - arguably only in the last fifty years - have evolutionary biologists really begun to understand the full range of causes of reproductive success, things like reciprocal altruism and costly signaling. Without all this highly detailed knowledge, an intelligent agent that set out to "maximize inclusive fitness" would fall flat on its face.

So why not preprogram protein brains with the knowledge? Why wasn't a concept of "inclusive genetic fitness" programmed into us, along with a library of explicit strategies? Then you could dispense with all the reinforcers. The organism would be born knowing that, with high probability, fatty foods would lead to fitness. If the organism later learned that this was no longer the case, it would stop eating fatty foods. You could refactor the whole system. And it wouldn't invent condoms or cookies.

This looks like it should be quite possible in principle. I occasionally run into people who don't quite understand consequentialism, who say, "But if the organism doesn't have a separate drive to eat, it will starve, and so fail to reproduce." So long as the organism knows this very fact, and has a utility function that values reproduction, it will automatically eat. In fact, this is exactly the consequentialist reasoning that natural selection itself used to build automatic eaters.

What about curiosity? Wouldn't a consequentialist only be curious when it saw some specific reason to be curious? And wouldn't this cause it to miss out on lots of important knowledge that came with no specific reason for investigation attached? Again, a consequentialist will investigate given only the knowledge of this very same fact. If you consider the curiosity drive of a human - which is not undiscriminating, but responds to particular features of problems - then this complex adaptation is purely the result of consequentialist reasoning by DNA, an implicit representation of knowledge: Ancestors who engaged in this kind of inquiry left more descendants.

So in principle, the pure reproductive consequentialist is possible. In principle, all the ancestral history implicitly represented in cognitive adaptations can be converted to explicitly represented knowledge, running on a core consequentialist.

But the blind idiot god isn't that smart. Evolution is not a human programmer who can simultaneously refactor whole code architectures. Evolution is not a human programmer who can sit down and type out instructions at sixty words per minute.

For millions of years before hominid consequentialism, there was reinforcement learning. The reward signals were events that correlated reliably to reproduction. You can't ask a nonhominid brain to foresee that a child eating fatty foods now will live through the winter. So the DNA builds a protein brain that generates a reward signal for eating fatty food. Then it's up to the organism to learn which prey animals are tastiest.

DNA constructs protein brains with reward signals that have a long-distance correlation to reproductive fitness, but a short-distance correlation to organism behavior. You don't have to figure out that eating sugary food in the fall will lead to digesting calories that can be stored fat to help you survive the winter so that you mate in spring to produce offspring in summer. An apple simply tastes good, and your brain just has to plot out how to get more apples off the tree.

And so organisms evolve rewards for eating, and building nests, and scaring off competitors, and helping siblings, and discovering important truths, and forming strong alliances, and arguing persuasively, and of course having sex...

When hominid brains capable of cross-domain consequential reasoning began to show up, they reasoned consequentially about how to get the existing reinforcers. It was a relatively simple hack, vastly simpler than rebuilding an "inclusive fitness maximizer" from scratch. The protein brains plotted how to acquire calories and sex, without any explicit cognitive representation of "inclusive fitness".

A human engineer would have said, "Whoa, I've just invented a consequentialist! Now I can take all my previous hard-won knowledge about which behaviors improve fitness, and declare it explicitly! I can convert all this complicated reinforcement learning machinery into a simple declarative knowledge statement that 'fatty foods and sex usually improve your inclusive fitness'. Consequential reasoning will automatically take care of the rest. Plus, it won't have the obvious failure mode where it invents condoms!"

But then a human engineer wouldn't have built the retina backward, either.

The blind idiot god is not a unitary purpose, but a many-splintered attention. Foxes evolve to catch rabbits, rabbits evolve to evade foxes; there are as many evolutions as species. But within each species, the blind idiot god is purely obsessed with inclusive genetic fitness. No quality is valued, not even survival, except insofar as it increases reproductive fitness. There's no point in an organism with steel skin if it ends up having 1% less reproductive capacity.

Yet when the blind idiot god created protein computers, its monomaniacal focus on inclusive genetic fitness was not faithfully transmitted. Its optimization criterion did not successfully quine. We, the handiwork of evolution, are as alien to evolution as our Maker is alien to us. One pure utility function splintered into a thousand shards of desire.

Why? Above all, because evolution is stupid in an absolute sense. But also because the first protein computers weren't anywhere near as general as the blind idiot god, and could only utilize short-term desires.

In the final analysis, asking why evolution didn't build humans to maximize inclusive genetic fitness, is like asking why evolution didn't hand humans a ribosome and tell them to design their own biochemistry. Because evolution can't refactor code that fast, that's why. But maybe in a billion years of continued natural selection that's exactly what would happen, if intelligence were foolish enough to allow the idiot god continued reign.

The Mote in God's Eye by Niven and Pournelle depicts an intelligent species that stayed biological a little too long, slowly becoming truly enslaved by evolution, gradually turning into true fitness maximizers obsessed with outreproducing each other. But thankfully that's not what happened. Not here on Earth. At least not yet.

So humans love the taste of sugar and fat, and we love our sons and daughters. We seek social status, and sex. We sing and dance and play. We learn for the love of learning.

A thousand delicious tastes, matched to ancient reinforcers that once correlated with reproductive fitness - now sought whether or not they enhance reproduction. Sex with birth control, chocolate, the music of long-dead Bach on a CD.

And when we finally learn about evolution, we think to ourselves: "Obsess all day about inclusive genetic fitness? Where's the fun in that?"

The blind idiot god's single monomaniacal goal splintered into a thousand shards of desire. And this is well, I think, though I'm a human who says so. Or else what would we do with the future? What would we do with the billion galaxies in the night sky? Fill them with maximally efficient replicators? Should our descendants deliberately obsess about maximizing their inclusive genetic fitness, regarding all else only as a means to that end?

Being a thousand shards of desire isn't always fun, but at least it's not boring. Somewhere along the line, we evolved tastes for novelty, complexity, elegance, and challenge - tastes that judge the blind idiot god's monomaniacal focus, and find it aesthetically unsatisfying.

And yes, we got those very same tastes from the blind idiot's godshatter. So what?

Referenced by: Terminal Values and Instrumental Values • Conjuring An Evolution To Serve You • Not for the Sake of Happiness (Alone) • Leaky Generalizations • The Hidden Complexity of Wishes • Fake Fake Utility Functions • Heading Toward Morality • Optimization and the Singularity • Moral Complexities • Is Morality Given? • The Gift We Give To Tomorrow • Setting Up Metaethics • The Meaning of Right • Contaminated by Optimism • Morality as Fixed Computation • Inseparably Right; or, Joy in the Merely Good • Mirrors and Paintings • Magical Categories • Qualitative Strategies of Friendliness • Emotional Involvement • In Praise of Boredom

Original with comments: Thou Art Godshatter

Terminal Values and Instrumental Values

Eliezer Yudkowsky, 15 November 2007 07:56AM

On a purely instinctive level, any human planner behaves as if they distinguish between means and ends. Want chocolate? There's chocolate at the Publix supermarket. You can get to the supermarket if you drive one mile south on Washington Ave. You can drive if you get into the car. You can get into the car if you open the door. You can open the door if you have your car keys. So you put your car keys into your pocket, and get ready to leave the house...

...when suddenly the word comes on the radio that an earthquake has destroyed all the chocolate at the local Publix. Well, there's no point in driving to the Publix if there's no chocolate there, and no point in getting into the car if you're not driving anywhere, and no point in having car keys in your pocket if you're not driving. So you take the car keys out of your pocket, and call the local pizza service and have them deliver a chocolate pizza. Mm, delicious.

I rarely notice people losing track of plans they devised themselves. People usually don't drive to the supermarket if they know the chocolate is gone. But I've also noticed that when people begin explicitly talking about goal systems instead of just wanting things, mentioning "goals" instead of using them, they oft become confused. Humans are experts at planning, not experts on planning, or there'd be a lot more AI developers in the world.

In particularly, I've noticed people get confused when - in abstract philosophical discussions rather than everyday life - they consider the distinction between means and ends; more formally, between "instrumental values" and "terminal values".

(Another long post needed as a reference.)

Part of the problem, it seems to me, is that the human mind uses a rather ad-hoc system to keep track of its goals - it works, but not cleanly. English doesn't embody a sharp distinction between means and ends: "I want to save my sister's life" and "I want to administer penicillin to my sister" use the same word "want".

Can we describe, in mere English, the distinction that is getting lost?

As a first stab:

"Instrumental values" are desirable strictly conditional on their anticipated consequences. "I want to administer penicillin to my sister", not because a penicillin-filled sister is an intrinsic good, but in anticipation of penicillin curing her flesh-eating pneumonia. If instead you anticipated that injecting penicillin would melt your sister into a puddle like the Wicked Witch of the West, you'd fight just as hard to keep her penicillin-free.

"Terminal values" are desirable without conditioning on other consequences: "I want to save my sister's life" has nothing to do with your anticipating whether she'll get injected with penicillin after that.

This first attempt suffers from obvious flaws. If saving my sister's life would cause the Earth to be swallowed up by a black hole, then I would go off and cry for a while, but I wouldn't administer penicillin. Does this mean that saving my sister's life was not a "terminal" or "intrinsic" value, because it's theoretically conditional on its consequences? Am I only trying to save her life because of my belief that a black hole won't consume the Earth afterward? Common sense should say that's not what's happening.

So forget English. We can set up a mathematical description of a decision system in which terminal values and instrumental values are separate and incompatible types - like integers and floating-point numbers, in a programming language with no automatic conversion between them.

An ideal Bayesian decision system can be set up using only four elements:

- Outcomes : type Outcome[]

- list of possible outcomes

- {sister lives, sister dies}

- Actions: type Action[]

- list of possible actions

- {administer penicillin, don't administer penicillin}

- Utility_function : type Outcome -> Utility

- utility function that maps each outcome onto a utility

- (a utility being representable as a real number between negative and positive infinity)

- {sister lives: 1, sister dies: 0}

- Conditional_probability_function : type Action -> Outcome -> Probability

- conditional probability function that maps each action onto a probability distribution over outcomes

- (a probability being representable as a real number between 0 and 1)

- {administer penicillin: sister lives, .9; sister dies, .1 ;; don't administer penicillin: sister lives, 0.3; sister dies, 0.7}

If you can't read the type system directly, don't worry, I'll always translate into English. For programmers, seeing it described in distinct statements helps to set up distinct mental objects.

And the decision system itself?

- Expected_Utility : Action A -> (Sum O in Outcomes: Utility(O) * Probability(O|A))

- The "expected utility" of an action equals the sum, over all outcomes, of the utility of that outcome times the conditional probability of that outcome given that action.

- {EU(administer penicillin) = 0.9 ; EU(don't administer penicillin) = 0.3}

- Choose : -> (Argmax A in Actions: Expected_Utility(A))

- Pick an action whose "expected utility" is maximal

- {return: administer penicillin}

For every action, calculate the conditional probability of all the consequences that might follow, then add up the utilities of those consequences times their conditional probability. Then pick the best action.

This is a mathematically simple sketch of a decision system. It is not an efficient way to compute decisions in the real world.

Suppose, for example, that you need a sequence of acts to carry out a plan? The formalism can easily represent this by letting each Action stand for a whole sequence. But this creates an exponentially large space, like the space of all sentences you can type in 100 letters. As a simple example, if one of the possible acts on the first turn is "Shoot my own foot off", a human planner will decide this is a bad idea generally - eliminate all sequences beginning with this action. But we've flattened this structure out of our representation. We don't have sequences of acts, just flat "actions".

So, yes, there are a few minor complications. Obviously so, or we'd just run out and build a real AI this way. In that sense, it's much the same as Bayesian probability theory itself.

But this is one of those times when it's a surprisingly good idea to consider the absurdly simple version before adding in any high-falutin' complications.

Consider the philosopher who asserts, "All of us are ultimately selfish; we care only about our own states of mind. The mother who claims to care about her son's welfare, really wants to believe that her son is doing well - this belief is what makes the mother happy. She helps him for the sake of her own happiness, not his." You say, "Well, suppose the mother sacrifices her life to push her son out of the path of an oncoming truck. That's not going to make her happy, just dead." The philosopher stammers for a few moments, then replies, "But she still did it because she valued that choice above others - because of the feeling of importance she attached to that decision."

So you say, "TYPE ERROR: No constructor found for Expected_Utility -> Utility."

Allow me to explain that reply.

Even our simple formalism illustrates a sharp distinction between expected utility, which is something that actions have; and utility, which is something that outcomes have. Sure, you can map both utilities and expected utilities onto real numbers. But that's like observing that you can map wind speed and temperature onto real numbers. It doesn't make them the same thing.

The philosopher begins by arguing that all your Utilities must be over Outcomes consisting of your state of mind. If this were true, your intelligence would operate as an engine to steer the future into regions where you were happy Future states would be distinguished only by your state of mind; you would be indifferent between any two futures in which you had the same state of mind.

And you would, indeed, be rather unlikely to sacrifice your own life to save another.

When we object that people sometimes do sacrifice their lives, the philosopher's reply shifts to discussing Expected Utilities over Actions: "The feeling of importance she attached to that decision." This is a drastic jump that should make us leap out of our chairs in indignation. Trying to convert an Expected_Utility into a Utility would cause an outright error in our programming language. But in English it all sounds the same.

The choices of our simple decision system are those with highest Expected Utility, but this doesn't say anything whatsoever about where it steers the future. It doesn't say anything about the utilities the decider assigns, or which real-world outcomes are likely to happen as a result. It doesn't say anything about the mind's function as an engine.

The physical cause of a physical action is a cognitive state, in our ideal decider an Expected_Utility, and this expected utility is calculated by evaluating a utility function over imagined consequences. To save your son's life, you must imagine the event of your son's life being saved, and this imagination is not the event itself. It's a quotation, like the difference between "snow" and snow. But that doesn't mean that what's inside the quote marks must itself be a cognitive state. If you choose the action that leads to the future that you represent with "my son is still alive", then you have functioned as an engine to steer the future into a region where your son is still alive. Not an engine that steers the future into a region where you represent the sentence "my son is still alive". To steer the future there, your utility function would have to return a high utility when fed ""my son is still alive"", the quotation of the quotation, your imagination of yourself imagining. Recipes make poor cake when you grind them up and toss them in the batter.

And that's why it's helpful to consider the simple decision systems first. Mix enough complications into the system, and formerly clear distinctions become harder to see.

So now let's look at some complications. Clearly the Utility function (mapping Outcomes onto Utilities) is meant to formalize what I earlier referred to as "terminal values", values not contingent upon their consequences. What about the case where saving your sister's life leads to Earth's destruction by a black hole? In our formalism, we've flattened out this possibility. Outcomes don't lead to Outcomes, only Actions lead to Outcomes. Your sister recovering from pneumonia followed by the Earth being devoured by a black hole would be flattened into a single "possible outcome".

And where are the "instrumental values" in this simple formalism? Actually, they've vanished entirely! You see, in this formalism, actions lead directly to outcomes with no intervening events. There's no notion of throwing a rock that flies through the air and knocks an apple off a branch so that it falls to the ground. Throwing the rock is the Action, and it leads straight to the Outcome of the apple lying on the ground - according to the conditional probability function that turns an Action directly into a Probability distribution over Outcomes.

In order to actually compute the conditional probability function, and in order to separately consider the utility of a sister's pneumonia and a black hole swallowing Earth, we would have to represent the network structure of causality - the way that events lead to other events.

And then the instrumental values would start coming back. If the causal network was sufficiently regular, you could find a state B that tended to lead to C regardless of how you achieved B. Then if you wanted to achieve C for some reason, you could plan efficiently by first working out a B that led to C, and then an A that led to B. This would be the phenomenon of "instrumental value" - B would have "instrumental value" because it led to C. C itself might be terminally valued - a term in the utility function over the total outcome. Or C might just be an instrumental value, a node that was not directly valued by the utility function.

Instrumental value, in this formalism, is purely an aid to the efficient computation of plans. It can and should be discarded wherever this kind of regularity does not exist.

Suppose, for example, that there's some particular value of B that doesn't lead to C. Would you choose an A which led to that B? Or never mind the abstract philosophy: If you wanted to go to the supermarket to get chocolate, and you wanted to drive to the supermarket, and you needed to get into your car, would you gain entry by ripping off the car door with a steam shovel? (No.) Instrumental value is a "leaky abstraction", as we programmers say; you sometimes have to toss away the cached value and compute out the actual expected utility. Part of being efficient without being suicidal is noticing when convenient shortcuts break down. Though this formalism does give rise to instrumental values, it does so only where the requisite regularity exists, and strictly as a convenient shortcut in computation.

But if you complicate the formalism before you understand the simple version, then you may start thinking that instrumental values have some strange life of their own, even in a normative sense. That, once you say B is usually good because it leads to C, you've committed yourself to always try for B even in the absence of C. People make this kind of mistake in abstract philosophy, even though they would never, in real life, rip open their car door with a steam shovel. You may start thinking that there's no way to develop a consequentialist that maximizes only inclusive genetic fitness, because it will starve unless you include an explicit terminal value for "eating food". People make this mistake even though they would never stand around opening car doors all day long, for fear of being stuck outside their cars if they didn't have a terminal value for opening car doors.

Instrumental values live in (the network structure of) the conditional probability function. This makes instrumental value strictly dependent on beliefs-of-fact given a fixed utility function. If I believe that penicillin causes pneumonia, and that the absence of penicillin cures pneumonia, then my perceived instrumental value of penicillin will go from high to low. Change the beliefs of fact - change the conditional probability function that associates actions to believed consequences - and the instrumental values will change in unison.

In moral arguments, some disputes are about instrumental consequences, and some disputes are about terminal values. If your debating opponent says that banning guns will lead to lower crime, and you say that banning guns lead to higher crime, then you agree about a superior instrumental value (crime is bad), but you disagree about which intermediate events lead to which consequences. But I do not think an argument about female circumcision is really a factual argument about how to best achieve a shared value of treating women fairly or making them happy.

This important distinction often gets flushed down the toilet in angry arguments. People with factual disagreements and shared values, each decide that their debating opponents must be sociopaths. As if your hated enemy, gun control / rights advocates, really wanted to kill people, which should be implausible as realistic psychology.

I fear the human brain does not strongly type the distinction between terminal moral beliefs and instrumental moral beliefs. "We should ban guns" and "We should save lives" don't feel different, as moral beliefs, the way that sight feels different from sound. Despite all the other ways that the human goal system complicates everything in sight, this one distinction it manages to collapse into a mishmash of things-with-conditional-value.

To extract out the terminal values we have to inspect this mishmash of valuable things, trying to figure out which ones are getting their value from somewhere else. It's a difficult project! If you say that you want to ban guns in order to reduce crime, it may take a moment to realize that "reducing crime" isn't a terminal value, it's a superior instrumental value with links to terminal values for human lives and human happinesses. And then the one who advocates gun rights may have links to the superior instrumental value of "reducing crime" plus a link to a value for "freedom", which might be a terminal value unto them, or another instrumental value...

We can't print out our complete network of values derived from other values. We probably don't even store the whole history of how values got there. By considering the right moral dilemmas, "Would you do X if Y", we can often figure out where our values came from. But even this project itself is full of pitfalls; misleading dilemmas and gappy philosophical arguments. We don't know what our own values are, or where they came from, and can't find out except by undertaking error-prone projects of cognitive archaeology. Just forming a conscious distinction between "terminal value" and "instrumental value", and keeping track of what it means, and using it correctly, is hard work. Only by inspecting the simple formalism can we see how easy it ought to be, in principle.

And that's to say nothing of all the other complications of the human reward system - the whole use of reinforcement architecture, and the way that eating chocolate is pleasurable, and anticipating eating chocolate is pleasurable, but they're different kinds of pleasures...

But I don't complain too much about the mess.

Being ignorant of your own values may not always be fun, but at least it's not boring.

Referenced by: Not for the Sake of Happiness (Alone) • Leaky Generalizations • The Hidden Complexity of Wishes • Lost Purposes • Fake Fake Utility Functions • Fake Utility Functions • The Allais Paradox • Zut Allais! • Against Discount Rates • The Quotation is not the Referent • Angry Atoms • Possibility and Could-ness • Heading Toward Morality • Moral Complexities • Is Morality Preference? • Is Morality Given? • The Gift We Give To Tomorrow • The Meaning of Right • Interpersonal Morality • Contaminated by Optimism • Moral Error and Moral Disagreement • Invisible Frameworks • Three Fallacies of Teleology • Qualitative Strategies of Friendliness • Optimization • Aiming at the Target • Measuring Optimization Power • Recognizing Intelligence • In Praise of Boredom • Value is Fragile

Original with comments: Terminal Values and Instrumental Values

Evolving to Extinction

Eliezer Yudkowsky, 16 November 2007 07:18AM

Followup to: Evolutions Are Stupid

It is a very common misconception that an evolution works for the good of its species. Can you remember hearing someone talk about two rabbits breeding eight rabbits and thereby "contributing to the survival of their species"? A modern evolutionary biologist would never say such a thing; they'd sooner breed with a rabbit.

It's yet another case where you've got to simultaneously consider multiple abstract concepts and keep them distinct. Evolution doesn't operate on particular individuals; individuals keep whatever genes they're born with. Evolution operates on a reproducing population, a species, over time. There's a natural tendency to think that if an Evolution Fairy is operating on the species, she must be optimizing for the species. But what really changes are the gene frequencies, and frequencies don't increase or decrease according to how much the gene helps the species as a whole. As we shall later see, it's quite possible for a species to evolve to extinction.

Why are boys and girls born in roughly equal numbers? (Leaving aside crazy countries that use artificial gender selection technologies.) To see why this is surprising, consider that 1 male can impregnate 2, 10, or 100 females; it wouldn't seem that you need the same number of males as females to ensure the survival of the species. This is even more surprising in the vast majority of animal species where the male contributes very little to raising the children - humans are extraordinary, even among primates, for their level of paternal investment. Balanced gender ratios are found even in species where the male impregnates the female and vanishes into the mist.

Consider two groups on different sides of a mountain; in group A, each mother gives birth to 2 males and 2 females; in group B, each mother gives birth to 3 females and 1 male. Group A and group B will have the same number of children, but group B will have 50% more grandchildren and 125% more great-grandchildren. You might think this would be a significant evolutionary advantage.

But consider: The rarer males become, the more reproductively valuable they become - not to the group, but to the individual parent. Every child has one male and one female parent. Then in every generation, the total genetic contribution from all males equals the total genetic contribution from all females. The fewer males, the greater the individual genetic contribution per male. If all the females around you are doing what's good for the group, what's good for the species, and birthing 1 male per 10 females, you can make a genetic killing by birthing all males, each of whom will have (on average) ten times as many grandchildren as their female cousins.

So while group selection ought to favor more girls, individual selection favors equal investment in male and female offspring. Looking at the statistics of a maternity ward, you can see at a glance that the quantitative balance between group selection forces and individual selection forces is overwhelmingly tilted in favor of individual selection in Homo sapiens.

(Technically, this isn't quite a glance. Individual selection favors equal parental investments in male and female offspring. If males cost half as much to birth and/or raise, twice as many males as females will be born at the evolutionarily stable equilibrium. If the same number of males and females were born in the population at large, but males were twice as cheap to birth, then you could again make a genetic killing by birthing more males. So the maternity ward should reflect the balance of parental opportunity costs, in a hunter-gatherer society, between raising boys and raising girls; and you'd have to assess that somehow. But ya know, it doesn't seem all that much more reproductive-opportunity-costly for a hunter-gatherer family to raise a girl, so it's kinda suspicious that around the same number of boys are born as girls.)

Natural selection isn't about groups, or species, or even individuals. In a sexual species, an individual organism doesn't evolve; it keeps whatever genes it's born with. An individual is a once-off collection of genes that will never reappear; how can you select on that? When you consider that nearly all of your ancestors are dead, it's clear that "survival of the fittest" is a tremendous misnomer. "Replication of the fitter" would be more accurate, although technically, fitness is defined only in terms of replication.

Natural selection is really about gene frequencies. To get a complex adaptation, a machine with multiple dependent parts, each new gene as it evolves depends on the other genes being reliably present in its genetic environment. They must have high frequencies. The more complex the machine, the higher the frequencies must be. The signature of natural selection occurring is a gene rising from 0.00001% of the gene pool to 99% of the gene pool. This is the information, in an information-theoretic sense; and this is what must happen for large complex adaptations to evolve.

The real struggle in natural selection is not the competition of organisms for resources; this is an ephemeral thing when all the participants will vanish in another generation. The real struggle is the competition of alleles for frequency in the gene pool. This is the lasting consequence that creates lasting information. The two rams bellowing and locking horns are only passing shadows.

It's perfectly possible for an allele to spread to fixation by outcompeting an alternative allele which was "better for the species". If the Flying Spaghetti Monster magically created a species whose gender mix was perfectly optimized to ensure the survival of the species - the optimal gender mix to bounce back reliably from near-extinction events, adapt to new niches, etcetera - then the evolution would rapidly degrade this species optimum back into the individual-selection optimum of equal parental investment in males and females.

Imagine a "Frodo gene" that sacrifices its vehicle to save its entire species from an extinction event. What happens to the allele frequency as a result? It goes down. Kthxbye.

If species-level extinction threats occur regularly (call this a "Buffy environment") then the Frodo gene will systematically decrease in frequency and vanish, and soon thereafter, so will the species. A hypothetical example? Maybe. If the human species was going to stay biological for another century, it would be a good idea to start cloning Gandhi.

In viruses, there's the tension between individual viruses replicating as fast as possible, versus the benefit of leaving the host alive long enough to transmit the illness. This is a good real-world example of group selection, and if the virus evolves to a point on the fitness landscape where the group selection pressures fail to overcome individual pressures, the virus could vanish shortly thereafter. I don't know if a disease has ever been caught in the act of evolving to extinction, but it's probably happened any number of times.

Segregation-distorters subvert the mechanisms that usually guarantee fairness of sexual reproduction. For example, there is a segregation-distorter on the male sex chromosome of some mice which causes only male children to be born, all carrying the segregation-distorter. Then these males impregnate females, who give birth to only male children, and so on. You might cry "This is cheating!" but that's a human perspective; the reproductive fitness of this allele is extremely high, since it produces twice as many copies of itself in the succeeding generation as its nonmutant alternative. Even as females become rarer and rarer, males carrying this gene are no less likely to mate than any other male, and so the segregation-distorter remains twice as fit as its alternative allele. It's speculated that real-world group selection may have played a role in keeping the frequency of this gene as low as it seems to be. In which case, if mice were to evolve the ability to fly and migrate for the winter, they would probably form a single reproductive population, and would evolve to extinction as the segregation-distorter evolved to fixation.

Around 50% of the total genome of maize consists of transposons, DNA elements whose primary function is to copy themselves into other locations of DNA. A class of transposons called "P elements" seem to have first appeared in Drosophila only in the middle of the 20th century, and spread to every population of the species within 50 years. The "Alu sequence" in humans, a 300-base transposon, is repeated between 300,000 and a million times in the human genome. This may not extinguish a species, but it doesn't help it; transposons cause more mutations which are as always mostly harmful, decrease the effective copying fidelity of DNA. Yet such cheaters are extremely fit.

Suppose that in some sexually reproducing species, a perfect DNA-copying mechanism is invented. Since most mutations are detrimental, this gene complex is an advantage to its holders. Now you might wonder about beneficial mutations - they do happen occasionally, so wouldn't the unmutable be at a disadvantage? But in a sexual species, a beneficial mutation that began in a mutable can spread to the descendants of unmutables as well. The mutables suffer from degenerate mutations in each generation; and the unmutables can sexually acquire, and thereby benefit from, any beneficial mutations that occur in the mutables. Thus the mutables have a pure disadvantage. The perfect DNA-copying mechanism rises in frequency to fixation. Ten thousand years later there's an ice age and the species goes out of business. It evolved to extinction.

The "bystander effect" is that, when someone is in trouble, solitary individuals are more likely to intervene than groups. A college student apparently having an epileptic seizure was helped 85% of the time by a single bystander, and 31% of the time by five bystanders. I speculate that even if the kinship relation in a hunter-gatherer tribe was strong enough to create a selection pressure for helping individuals not directly related, when several potential helpers were present, a genetic arms race might occur to be the last one to step forward. Everyone delays, hoping that someone else will do it. Humanity is facing multiple species-level extinction threats right now, and I gotta tell ya, there ain't a lot of people steppin' forward. If we lose this fight because virtually no one showed up on the battlefield, then - like a probably-large number of species which we don't see around today - we will have evolved to extinction.

Cancerous cells do pretty well in the body, prospering and amassing more resources, far outcompeting their more obedient counterparts. For a while.

Multicellular organisms can only exist because they've evolved powerful internal mechanisms to outlaw evolution. If the cells start evolving, they rapidly evolve to extinction: the organism dies.

So praise not evolution for the solicitous concern it shows for the individual; nearly all of your ancestors are dead. Praise not evolution for the solicitous concern it shows for a species; no one has ever found a complex adaptation which can only be interpreted as operating to preserve a species, and the mathematics would seem to indicate that this is virtually impossible. Indeed, it's perfectly possible for a species to evolve to extinction. Humanity may be finishing up the process right now. You can't even praise evolution for the solicitous concern it shows for genes; the battle between two alternative alleles at the same location is a zero-sum game for frequency.

Fitness is not always your friend.

Sequence: Evolution

No Evolutions for Corporations or Nanodevices

The Tragedy of Group Selectionism

Referenced by: No Evolutions for Corporations or Nanodevices • Conjuring An Evolution To Serve You • Fake Fake Utility Functions • A Failed Just-So Story • Optimization and the Singularity • The Gift We Give To Tomorrow • Interpersonal Morality • Three Fallacies of Teleology • Excluding the Supernatural • Ethical Inhibitions • Observing Optimization

Original with comments: Evolving to Extinction

No Evolutions for Corporations or Nanodevices

Eliezer Yudkowsky, 17 November 2007 02:24AM

"The laws of physics and the rules of math don't cease to apply. That leads me to believe that evolution doesn't stop. That further leads me to believe that nature -- bloody in tooth and claw, as some have termed it -- will simply be taken to the next level...
"[Getting rid of Darwinian evolution is] like trying to get rid of gravitation. So long as there are limited resources and multiple competing actors capable of passing on characteristics, you have selection pressure."
-- Perry Metzger, predicting that the reign of natural selection would continue into the indefinite future.

In evolutionary biology, as in many other fields, it is important to think quantitatively rather than qualitatively. Does a beneficial mutation "sometimes spread, but not always"? Well, a psychic power would be a beneficial mutation, so you'd expect it to spread, right? Yet this is qualitative reasoning, not quantitative - if X is true, then Y is true; if psychic powers are beneficial, they may spread. In Evolutions Are Stupid, I described the equations for a beneficial mutation's probability of fixation, roughly twice the fitness advantage (6% for a 3% advantage). Only this kind of numerical thinking is likely to make us realize that mutations which are only rarely useful are extremely unlikely to spread, and that it is practically impossible for complex adaptations to arise without constant use. If psychic powers really existed, we should expect to see everyone using them all the time - not just because they would be so amazingly useful, but because otherwise they couldn't have evolved in the first place.

"So long as there are limited resources and multiple competing actors capable of passing on characteristics, you have selection pressure." This is qualitative reasoning. How much selection pressure?

While there are several candidates for the most important equation in evolutionary biology, I would pick Price's Equation, which in its simplest formulation reads:

change in average characteristic = covariance(relative fitness, characteristic)

This is a very powerful and general formula. For example, a particular gene for height can be the Z, the characteristic that changes, in which case Price's Equation says that the change in the probability of possessing this gene equals the covariance of the gene with reproductive fitness. Or you can consider height in general as the characteristic Z, apart from any particular genes, and Price's Equation says that the change in height in the next generation will equal the covariance of height with relative reproductive fitness.

(At least, this is true so long as height is straightforwardly heritable. If nutrition improves, so that a fixed genotype becomes taller, you have to add a correction term to Price's Equation. If there are complex nonlinear interactions between many genes, you have to either add a correction term, or calculate the equation in such a complicated way that it ceases to enlighten.)

Many enlightenments may be attained by studying the different forms and derivations of Price's Equation. For example, the final equation says that the average characteristic changes according to its covariance with relative fitness, rather than its absolute fitness. This means that if a Frodo gene saves its whole species from extinction, the average Frodo characteristic does not increase, since Frodo's act benefited all genotypes equally and did not covary with relative fitness.

It is said that Price became so disturbed with the implications of his equation for altruism that he committed suicide, though he may have had other issues. (Overcoming Bias does not advocate committing suicide after studying Price's Equation.)

One of the enlightenments which may be gained by meditating upon Price's Equation is that "limited resources" and "multiple competing actors capable of passing on characteristics" are not sufficient to give rise to an evolution. "Things that replicate themselves" is not a sufficient condition. Even "competition between replicating things" is not sufficient.

Do corporations evolve? They certainly compete. They occasionally spin off children. Their resources are limited. They sometimes die.

But how much does the child of a corporation resemble its parents? Much of the personality of a corporation derives from key officers, and CEOs cannot divide themselves by fission. Price's Equation only operates to the extent that characteristics are heritable across generations. If great-great-grandchildren don't much resemble their great-great-grandparents, you won't get more than four generations' worth of cumulative selection pressure - anything that happened more than four generations ago will blur itself out. Yes, the personality of a corporation can influence its spinoff - but that's nothing like the heritability of DNA, which is digital rather than analog, and can transmit itself with 10^-8 errors per base per generation.

With DNA you have heritability lasting for millions of generations. That's how complex adaptations can arise by pure evolution - the digital DNA lasts long enough for a gene conveying 3% advantage to spread itself over 768 generations, and then another gene dependent on it can arise. Even if corporations replicated with digital fidelity, they would currently be at most ten generations into the RNA World.

Now, corporations are certainly selected, in the sense that incompetent corporations go bust. This should logically make you more likely to observe corporations with features contributing to competence. And in the same sense, any star that goes nova shortly after it forms, is less likely to be visible when you look up at the night sky. But if an accident of stellar dynamics makes one star burn longer than another star, that doesn't make it more likely that future stars will also burn longer - the feature will not be copied onto other stars. We should not expect future astrophysicists to discover complex internal features of stars which seem designed to help them burn longer. That kind of mechanical adaptation requires much larger cumulative selection pressures than a once-off winnowing.

Think of the principle introduced in Einstein's Arrogance - that the vast majority of the evidence required to think of General Relativity had to go into raising that one particular equation to the level of Einstein's personal attention; the amount of evidence required to raise it from a deliberately considered possibility to 99.9% certainty was trivial by comparison. In the same sense, complex features of corporations which require hundreds of bits to specify, are produced primarily by human intelligence, not a handful of generations of low-fidelity evolution. In biology, the mutations are purely random and evolution supplies thousands of bits of cumulative selection pressure. In corporations, humans offer up thousand-bit intelligently designed complex "mutations", and then the further selection pressure of "Did it go bankrupt or not?" accounts for a handful of additional bits in explaining what you see.

Advanced molecular nanotechnology - the artificial sort, not biology - should be able to copy itself with digital fidelity through thousands of generations. Would Price's Equation thereby gain a foothold?

Correlation is covariance divided by variance, so if A is highly predictive of B, there can be a strong "correlation" between them even if A is ranging from 0 to 9 and B is only ranging from 50.0001 and 50.0009. Price's Equation runs on covariance of characteristics with reproduction - not correlation! If you can compress variance in characteristics into a tiny band, the covariance goes way down, and so does the cumulative change in the characteristic.

The Foresight Institute suggests, among other sensible proposals, that the replication instructions for any nanodevice should be encrypted. Moreover, encrypted such that flipping a single bit of the encoded instructions will entirely scramble the decrypted output. If all nanodevices produced are precise molecular copies, and moreover, any mistakes on the assembly line are not heritable because the offspring got a digital copy of the original encrypted instructions for use in making grandchildren, then your nanodevices ain't gonna be doin' much evolving.

You'd still have to worry about prions - self-replicating assembly errors apart from the encrypted instructions, where a robot arm fails to grab a carbon atom that is used in assembling a homologue of itself, and this causes the offspring's robot arm to likewise fail to grab a carbon atom, etc., even with all the encrypted instructions remaining constant. But how much correlation is there likely to be, between this sort of transmissible error, and a higher reproductive rate? Let's say that one nanodevice produces a copy of itself every 1000 seconds, and the new nanodevice is magically more efficient (it not only has a prion, it has a beneficial prion) and copies itself every 999.99999 seconds. It needs one less carbon atom attached, you see. That's not a whole lot of variance in reproduction, so it's not a whole lot of covariance either.

And how often will these nanodevices need to replicate? Unless they've got more atoms available than exist in the solar system, or for that matter, the visible Universe, only a small number of generations will pass before they hit the resource wall. "Limited resources" are not a sufficient condition for evolution; you need the frequently iterated death of a substantial fraction of the population to free up resources. Indeed, "generations" is not so much an integer as an integral over the fraction of the population that consists of newly created individuals.

This is, to me, the most frightening thing about grey goo or nanotechnological weapons - that they could eat the whole Earth and then that would be it, nothing interesting would happen afterward. Diamond is stabler than proteins held together by van der Waals forces, so the goo would only need to reassemble some pieces of itself when an asteroid hit. Even if prions were a powerful enough idiom to support evolution at all - evolution is slow enough with digital DNA! - less than 1.0 generations might pass between when the goo ate the Earth and when the Sun died.

To sum up, if you have all of the following properties:

- Entities that replicate

- Substantial variation in their characteristics

- Substantial variation in their reproduction

- Persistent correlation between the characteristics and reproduction

- High-fidelity long-range heritability in characteristics

- Frequent birth of a significant fraction of the breeding population

- And all this remains true through many iterations

Then you will have significant cumulative selection pressures, enough to produce complex adaptations by the force of evolution.

Sequence: Evolution

Adaptation-Executers, not Fitness-Maximizers

Evolving to Extinction

Referenced by: The Simple Math of Everything • Conjuring An Evolution To Serve You • Fake Fake Utility Functions • Optimization and the Singularity • Interpersonal Morality • When Anthropomorphism Became Stupid • Failure By Analogy

Original with comments: No Evolutions for Corporations or Nanodevices

The Simple Math of Everything

Eliezer Yudkowsky, 17 November 2007 10:42PM

I am not a professional evolutionary biologist. I only know a few equations, very simple ones by comparison to what can be found in any textbook on evolutionary theory with math, and on one memorable occasion I used one incorrectly. For me to publish an article in a highly technical ev-bio journal would be as impossible as corporations evolving. And yet when I'm dealing with almost anyone who's not a professional evolutionary biologist...

It seems to me that there's a substantial advantage in knowing the drop-dead basic fundamental embarrassingly simple mathematics in as many different subjects as you can manage. Not, necessarily, the high-falutin' complicated damn math that appears in the latest journal articles. Not unless you plan to become a professional in the field. But for people who can read calculus, and sometimes just plain algebra, the drop-dead basic mathematics of a field may not take that long to learn. And it's likely to change your outlook on life more than the math-free popularizations or the highly technical math.

Not Jacobean matrices for frequency-dependent gene selection; just Haldane's calculation of time to fixation. Not quantum physics; just the wave equation for sound in air. Not the maximum entropy solution using Lagrange multipliers; just Bayes's Rule.

The Simple Math of Everything, written for people who are good at math, might not be all that weighty a volume. How long does it take to explain Bayes's Rule to someone who's good at math? Damn would I like to buy that book and send it back in time to my 16-year-old self. But there's no way I have time to write this book, so I'm tossing the idea out there.

Even in reading popular works on science, there is yet power. You don't want to end up like those poor souls in that recent interview (I couldn't Google) where a well-known scientist in field XYZ thinks the universe is 100 billion years old. But it seems to me that there's substantially more power in pushing until you encounter some basic math. Not complicated math, just basic math. F=ma is too simple, though. You should take the highest low-hanging fruit you can reach.

Yes, there are sciences whose soul is not in their math, yet which are nonetheless incredibly important and enlightening. Evolutionary psychology, for example. But even there, if you kept pushing until you encountered equations, you would be well-served by that heuristic, even if the equations didn't seem all that enlightening compared to the basic results.

I remember when I finally picked up and started reading through my copy of the Feynman Lectures on Physics, even though I couldn't think of any realistic excuse for how this was going to help my AI work, because I just got fed up with not knowing physics. And - you can guess how this story ends - it gave me a new way of looking at the world, which all my earlier reading in popular physics (including Feynman's QED) hadn't done. Did that help inspire my AI research? Hell yes. (Though it's a good thing I studied neuroscience, evolutionary psychology, evolutionary biology, Bayes, and physics in that order - physics alone would have been terrible inspiration for AI research.)

In academia (or so I am given to understand) there's a huge pressure to specialize, to push your understanding of one subject all the way out to the frontier of the latest journal articles, so that you can write your own journal articles and get tenure. Well, one may certainly have to learn the far math of one field, but why avoid the simple math of others? Is it too embarrassing to learn just a little math, and then stop? Is there an unwritten rule which says that once you start learning any math, you are obligated to finish it all? Could that be why the practice isn't more common?

I know that I'm much more embarrassed to know a few simple equations of physics, than I was to know only popular physics. It feels wronger to know a few simple equations of evolutionary biology than to know only qualitative evolutionary biology. Even mentioning how useful it's been seems wrong, as if I'm boasting about something that no one should boast about. It feels like I'm a dilettante - but how would I be diletting less if I hadn't studied even the simple math?

Referenced by: Conjuring An Evolution To Serve You • The Beauty of Settled Science • An Intuitive Explanation of Quantum Mechanics • Possibility and Could-ness

Original with comments: The Simple Math of Everything

Conjuring An Evolution To Serve You

Eliezer Yudkowsky, 19 November 2007 05:55AM

GreyThumb.blog offers an interesting analogue between research on animal breeding and the fall of Enron. Before 1995, the way animal breeding worked was that you would take the top individual performers in each generation and breed from them, or their parents. A cockerel doesn't lay eggs, so you have to observe daughter hens to determine which cockerels to breed. Sounds logical, right? If you take the hens who lay the most eggs in each generation, and breed from them, you should get hens who lay more and more eggs.

Behold the awesome power of making evolution work for you! The power that made butterflies - now constrained to your own purposes! And it worked, too. Per-cow milk output in the US doubled between 1905 and 1965, and has doubled again since then.

Yet conjuring Azathoth oft has unintended consequences, as some researchers realized in the 1990s. In the real world, sometimes you have more than animal per farm. You see the problem, right? If you don't, you should probably think twice before trying to conjure an evolution to serve you - magic is not for the unparanoid.

Selecting the hen who lays the most eggs doesn't necessarily get you the most efficient egg-laying metabolism. It may get you the most dominant hen, that pecked its way to the top of the pecking order at the expense of other hens. Individual selection doesn't necessarily work to the benefit of the group, but a farm's productivity is determined by group outputs.

Indeed, for some strange reason, the individual breeding programs which had been so successful at increasing egg production now required hens to have their beaks clipped, or be housed in individual cages, or they would peck each other to death.

While the conditions for group selection are only rarely right in Nature, one can readily impose genuine group selection in the laboratory. After only 6 generations of artificially imposed group selection - breeding from the hens in the best groups, rather than the best individual hens - average days of survival increased from 160 to 348, and egg mass per bird increased from 5.3 to 13.3 kg. At 58 weeks of age, the selected line had 20% mortality compared to the control group at 54%. A commercial line of hens, allowed to grow up with unclipped beaks, had 89% mortality at 58 weeks.

And the fall of Enron? Jeff Skilling fancied himself an evolution-conjurer, it seems. (Not that he, like, knew any evolutionary math or anything.) Every year, every Enron employee's performance would be evaluated, and the bottom 10% would get fired, and the top performers would get huge raises and bonuses. Unfortunately, as GreyThumb points out:

"Everyone knows that there are many things you can do in any corporate environment to give the appearance and impression of being productive. Enron's corporate environment was particularly conducive to this: its principal business was energy trading, and it had large densely populated trading floors peopled by high-powered traders that would sit and play the markets all day. There were, I'm sure, many things that a trader could do to up his performance numbers, either by cheating or by gaming the system. This gaming of the system probably included gaming his fellow traders, many of whom were close enough to rub elbows with.

"So Enron was applying selection at the individual level according to metrics like individual trading performance to a group system whose performance was, like the henhouses, an emergent property of group dynamics as well as a result of individual fitness. The result was more or less the same. Instead of increasing overall productivity, they got mean chickens and actual productivity declined. They were selecting for traits like aggressiveness, sociopathic tendencies, and dishonesty."

And the moral of the story is: Be careful when you set forth to conjure the blind idiot god. People look at a pretty butterfly (note selectivity) and think: "Evolution designed them - how pretty - I should get evolution to do things for me, too!" But this is qualitative reasoning, as if evolution were either present or absent. Applying 10% selection for 10 generations is not going to get you the same amount of cumulative selection pressure as 3.85 billion years of natural selection.

I have previously emphasized that the evolution-of-foxes works at cross-purposes to the evolution-of-rabbits; there is no unitary Evolution God to praise for every beauty of Nature. Azathoth has ten million hands. When you conjure, you don't get the evolution, the Maker of Butterflies. You get an evolution, with characteristics and strength that depend on your exact conjuration. If you just take everything you see in Nature and attribute it to "evolution", you'll start thinking that some cute little conjuration which runs for 20 generations will get you artifacts on the order of butterflies. Try 3.85 billion years.

Same caveat with the wonders of simulated evolution on computers, producing a radio antenna better than a human design, etcetera. These are sometimes human-competitive (more often not) when it comes to optimizing a continuous design over 57 performance criteria, or breeding a design with 57 elements. Anything beyond that, and modern evolutionary algorithms are defeated by the same exponential explosion that consumes the rest of AI. Yes, evolutionary algorithms have a legitimate place in AI. Consult a machine-learning expert, who knows when to use them and when not to. Even biologically inspired genetic algorithms with sexual mixing, rarely perform better than beam searches and other non-biologically-inspired techniques on the same problem.

And for this weakness, let us all be thankful. If the blind idiot god did not take a million years in which to do anything complicated, It would be bloody scary. 3.85 billion years of natural selection produced molecular nanotechnology (cells) and Artificial General Intelligence (brains), which even we humans aren't going to get for a few more decades. If there were an alien demideity, morality-and-aesthetics-free, often blindly suicidal, capable of wielding nanotech and AGI in real time, I'd put aside all other concerns and figure out how to kill it. Assuming that I hadn't already been enslaved beyond all desire of escape. Look at the trouble we're having with bacteria, which go through generations fast enough that their evolutions are learning to evade our antibiotics after only a few decades' respite.

You really don't want to conjure Azathoth at full power. You really, really don't. You'll get more than pretty butterflies.

Referenced by: The Hidden Complexity of Wishes • Fake Fake Utility Functions • Selecting Rationalist Groups

Original with comments: Conjuring An Evolution To Serve You

Artificial Addition

Eliezer Yudkowsky, 20 November 2007 07:58AM

Followup to: The Simple Truth

Suppose that human beings had absolutely no idea how they performed arithmetic. Imagine that human beings had evolved, rather than having learned, the ability to count sheep and add sheep. People using this built-in ability have no idea how it worked, the way Aristotle had no idea how his visual cortex supported his ability to see things. Peano Arithmetic as we know it has not been invented. There are philosophers working to formalize numerical intuitions, but they employ notations such as

Plus-Of(Seven, Six) = Thirteen

to formalize the intuitively obvious fact that when you add "seven" plus "six", of course you get "thirteen".

In this world, pocket calculators work by storing a giant lookup table of arithmetical facts, entered manually by a team of expert Artificial Arithmeticians, for starting values that range between zero and one hundred. While these calculators may be helpful in a pragmatic sense, many philosophers argue that they're only simulating addition, rather than really adding. No machine can really count - that's why humans have to count thirteen sheep before typing "thirteen" into the calculator. Calculators can recite back stored facts, but they can never know what the statements mean - if you type in "two hundred plus two hundred" the calculator says "Error: Outrange", when it's intuitively obvious, if you know what the words mean, that the answer is "four hundred".

Philosophers, of course, are not so naive as to be taken in by these intuitions. Numbers are really a purely formal system - the label "thirty-seven" is meaningful, not because of any inherent property of the words themselves, but because the label refers to thirty-seven sheep in the external world. A number is given this referential property by its semantic network of relations to other numbers. That's why, in computer programs, the LISP token for "thirty-seven" doesn't need any internal structure - it's only meaningful because of reference and relation, not some computational property of "thirty-seven" itself.

No one has ever developed an Artificial General Arithmetician, though of course there are plenty of domain-specific, narrow Artificial Arithmeticians that work on numbers between "twenty" and "thirty", and so on. And if you look at how slow progress has been on numbers in the range of "two hundred", then it becomes clear that we're not going to get Artificial General Arithmetic any time soon. The best experts in the field estimate it will be at least a hundred years before calculators can add as well as a human twelve-year-old.

But not everyone agrees with this estimate, or with merely conventional beliefs about Artificial Arithmetic. It's common to hear statements such as the following:

- "It's a framing problem - what 'twenty-one plus' equals depends on whether it's 'plus three' or 'plus four'. If we can just get enough arithmetical facts stored to cover the common-sense truths that everyone knows, we'll start to see real addition in the network."

- "But you'll never be able to program in that many arithmetical facts by hiring experts to enter them manually. What we need is an Artificial Arithmetician that can learn the vast network of relations between numbers that humans acquire during their childhood by observing sets of apples."

- "No, what we really need is an Artificial Arithmetician that can understand natural language, so that instead of having to be explicitly told that twenty-one plus sixteen equals thirty-seven, it can get the knowledge by exploring the Web."

- "Frankly, it seems to me that you're just trying to convince yourselves that you can solve the problem. None of you really know what arithmetic is, so you're floundering around with these generic sorts of arguments. 'We need an AA that can learn X', 'We need an AA that can extract X from the Internet'. I mean, it sounds good, it sounds like you're making progress, and it's even good for public relations, because everyone thinks they understand the proposed solution - but it doesn't really get you any closer to general addition, as opposed to domain-specific addition. Probably we will never know the fundamental nature of arithmetic. The problem is just too hard for humans to solve."

- "That's why we need to develop a general arithmetician the same way Nature did - evolution."

- "Top-down approaches have clearly failed to produce arithmetic. We need a bottom-up approach, some way to make arithmetic emerge. We have to acknowledge the basic unpredictability of complex systems."

- "You're all wrong. Past efforts to create machine arithmetic were futile from the start, because they just didn't have enough computing power. If you look at how many trillions of synapses there are in the human brain, it's clear that calculators don't have lookup tables anywhere near that large. We need calculators as powerful as a human brain. According to Moore's Law, this will occur in the year 2031 on April 27 between 4:00 and 4:30 in the morning."

- "I believe that machine arithmetic will be developed when researchers scan each neuron of a complete human brain into a computer, so that we can simulate the biological circuitry that performs addition in humans."

- "I don't think we have to wait to scan a whole brain. Neural networks are just like the human brain, and you can train them to do things without knowing how they do them. We'll create programs that will do arithmetic without we, our creators, ever understanding how they do arithmetic."

- "But Gödel's Theorem shows that no formal system can ever capture the basic properties of arithmetic. Classical physics is formalizable, so to add two and two, the brain must take advantage of quantum physics."

- "Hey, if human arithmetic were simple enough that we could reproduce it in a computer, we wouldn't be able to count high enough to build computers."

- "Haven't you heard of John Searle's Chinese Calculator Experiment? Even if you did have a huge set of rules that would let you add 'twenty-one' and 'sixteen', just imagine translating all the words into Chinese, and you can see that there's no genuine addition going on. There are no real numbers anywhere in the system, just labels that humans use for numbers..."

There is more than one moral to this parable, and I have told it with different morals in different contexts. It illustrates the idea of levels of organization, for example - a CPU can add two large numbers because the numbers aren't black-box opaque objects, they're ordered structures of 32 bits.

But for purposes of overcoming bias, let us draw two morals:

- First, the danger of believing assertions you can't regenerate from your own knowledge.

- Second, the danger of trying to dance around basic confusions.

Lest anyone accuse me of generalizing from fictional evidence, both lessons may be drawn from the real history of Artificial Intelligence as well.

The first danger is the object-level problem that the AA devices ran into: they functioned as tape recorders playing back "knowledge" generated from outside the system, using a process they couldn't capture internally. A human could tell the AA device that "twenty-one plus sixteen equals thirty-seven", and the AA devices could record this sentence and play it back, or even pattern-match "twenty-one plus sixteen" to output "thirty-seven!", but the AA devices couldn't generate such knowledge for themselves.

Which is strongly reminiscent of believing a physicist who tells you "Light is waves", recording the fascinating words and playing them back when someone asks "What is light made of?", without being able to generate the knowledge for yourself. More on this theme tomorrow.

The second moral is the meta-level danger that consumed the Artificial Arithmetic researchers and opinionated bystanders - the danger of dancing around confusing gaps in your knowledge. The tendency to do just about anything except grit your teeth and buckle down and fill in the damn gap.

Whether you say, "It is emergent!", or whether you say, "It is unknowable!", in neither case are you acknowledging that there is a basic insight required which is possessable, but unpossessed by you.

How can you know when you'll have a new basic insight? And there's no way to get one except by banging your head against the problem, learning everything you can about it, studying it from as many angles as possible, perhaps for years. It's not a pursuit that academia is set up to permit, when you need to publish at least one paper per month. It's certainly not something that venture capitalists will fund. You want to either go ahead and build the system now, or give up and do something else instead.

Look at the comments above: none are aimed at setting out on a quest for the missing insight which would make numbers no longer mysterious, make "twenty-seven" more than a black box. None of the commenters realized that their difficulties arose from ignorance or confusion in their own minds, rather than an inherent property of arithmetic. They were not trying to achieve a state where the confusing thing ceased to be confusing.

If you read Judea Pearl's "Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference" then you will see that the basic insight behind graphical models is indispensable to problems that require it. (It's not something that fits on a T-Shirt, I'm afraid, so you'll have to go and read the book yourself. I haven't seen any online popularizations of Bayesian networks that adequately convey the reasons behind the principles, or the importance of the math being exactly the way it is, but Pearl's book is wonderful.) There were once dozens of "non-monotonic logics" awkwardly trying to capture intuitions such as "If my burglar alarm goes off, there was probably a burglar, but if I then learn that there was a small earthquake near my home, there was probably not a burglar." With the graphical-model insight in hand, you can give a mathematical explanation of exactly why first-order logic has the wrong properties for the job, and express the correct solution in a compact way that captures all the common-sense details in one elegant swoop. Until you have that insight, you'll go on patching the logic here, patching it there, adding more and more hacks to force it into correspondence with everything that seems "obviously true".

You won't know the Artificial Arithmetic problem is unsolvable without its key. If you don't know the rules, you don't know the rule that says you need to know the rules to do anything. And so there will be all sorts of clever ideas that seem like they might work, like building an Artificial Arithmetician that can read natural language and download millions of arithmetical assertions from the Internet.

And yet somehow the clever ideas never work. Somehow it always turns out that you "couldn't see any reason it wouldn't work" because you were ignorant of the obstacles, not because no obstacles existed. Like shooting blindfolded at a distant target - you can fire blind shot after blind shot, crying, "You can't prove to me that I won't hit the center!" But until you take off the blindfold, you're not even in the aiming game. When "no one can prove to you" that your precious idea isn't right, it means you don't have enough information to strike a small target in a vast answer space. Until you know your idea will work, it won't.

From the history of previous key insights in Artificial Intelligence, and the grand messes which were proposed prior to those insights, I derive an important real-life lesson: When the basic problem is your ignorance, clever strategies for bypassing your ignorance lead to shooting yourself in the foot.

Referenced by: Truly Part Of You • The Hidden Complexity of Wishes • Lost Purposes • Fake Fake Utility Functions • Fake Utility Functions • Reversed Stupidity Is Not Intelligence • Zombies! Zombies? • A Premature Word on AI • The Rhythm of Disagreement • Principles of Disagreement • Causality and Moral Responsibility • Passing the Recursive Buck • Grasping Slippery Things • Ghosts in the Machine • Surface Analogies and Deep Causes • Abstracted Idealized Dynamics • That Tiny Note of Discord • Building Something Smarter • Logical or Connectionist AI? • Teaching the Unteachable

Original with comments: Artificial Addition

Truly Part Of You

Eliezer Yudkowsky, 21 November 2007 02:18AM

Followup to: Guessing the Teacher's Password, Artificial Addition

A classic paper by Drew McDermott, "Artificial Intelligence Meets Natural Stupidity", criticized AI programs that would try to represent notions like happiness is a state of mind using a semantic network:

STATE-OF-MIND
^
| IS-A
|
HAPPINESS

And of course there's nothing inside the "HAPPINESS" node; it's just a naked LISP token with a suggestive English name.

So, McDermott says, "A good test for the disciplined programmer is to try using gensyms in key places and see if he still admires his system. For example, if STATE-OF-MIND is renamed G1073..." then we would have IS-A(HAPPINESS, G1073) "which looks much more dubious."

Or as I would slightly rephrase the idea: If you substituted randomized symbols for all the suggestive English names, you would be completely unable to figure out what G1071(G1072, 1073) meant. Was the AI program meant to represent hamburgers? Apples? Happiness? Who knows? If you delete the suggestive English names, they don't grow back.

Suppose a physicist tells you that "Light is waves", and you believe him. You now have a little network in your head that says IS-A(LIGHT, WAVES). If someone asks you "What is light made of?" you'll be able to say "Waves!"

As McDermott says, "The whole problem is getting the hearer to notice what it has been told. Not 'understand', but 'notice'." Suppose that instead the physicist told you, "Light is made of little curvy things." (Not true, btw.) Would you notice any difference of anticipated experience?

How can you realize that you shouldn't trust your seeming knowledge that "light is waves"? One test you could apply is asking, "Could I regenerate this knowledge if it were somehow deleted from my mind?"

This is similar in spirit to scrambling the names of suggestively named LISP tokens in your AI program, and seeing if someone else can figure out what they allegedly "refer" to. It's also similar in spirit to observing that while an Artificial Arithmetician can record and play back Plus-Of(Seven, Six) = Thirteen, it can't regenerate the knowledge if you delete it from memory, until another human re-enters it in the database. Just as if you forgot that "light is waves", you couldn't get back the knowledge except the same way you got the knowledge to begin with - by asking a physicist. You couldn't generate the knowledge for yourself, the way that physicists originally generated it.

The same experiences that lead us to formulate a belief, connect that belief to other knowledge and sensory input and motor output. If you see a beaver chewing a log, then you know what this thing-that-chews-through-logs looks like, and you will be able to recognize it on future occasions whether it is called a "beaver" or not. But if you acquire your beliefs about beavers by someone else telling you facts about "beavers", you may not be able to recognize a beaver when you see one.

This is the terrible danger of trying to tell an Artificial Intelligence facts which it could not learn for itself. It is also the terrible danger of trying to tell someone about physics that they cannot verify for themselves. For what physicists mean by "wave" is not "little squiggly thing" but a purely mathematical concept.

As Davidson observes, if you believe that "beavers" live in deserts, are pure white in color, and weigh 300 pounds when adult, then you do not have any beliefs about beavers, true or false. Your belief about "beavers" is not right enough to be wrong. If you don't have enough experience to regenerate beliefs when they are deleted, then do you have enough experience to connect that belief to anything at all? Wittgenstein: "A wheel that can be turned though nothing turns with it, is not part of the mechanism."

Almost as soon as I started reading about AI - even before I read McDermott - I realized it would be a really good idea to always ask myself: "How would I regenerate this knowledge if it were deleted from my mind?"

The deeper the deletion, the stricter the test. If all proofs of the Pythagorean Theorem were deleted from my mind, could I re-prove it? I think so. If all knowledge of the Pythagorean Theorem were deleted from my mind, would I notice the Pythagorean Theorem to re-prove? That's harder to boast, without putting it to the test; but if you handed me a right triangle with sides 3 and 4, and told me that the length of the hypotenuse was calculable, I think I would be able to calculate it, if I still knew all the rest of my math.

What about the notion of mathematical proof? If no one had ever told it to me, would I be able to reinvent that on the basis of other beliefs I possess? There was a time when humanity did not have such a concept. Someone must have invented it. What was it that they noticed? Would I notice if I saw something equally novel and equally important? Would I be able to think that far outside the box?

How much of your knowledge could you regenerate? From how deep a deletion? It's not just a test to cast out insufficiently connected beliefs. It's a way of absorbing a fountain of knowledge, not just one fact.

A shepherd builds a counting system that works by throwing a pebble into a bucket whenever a sheep leaves the fold, and taking a pebble out whenever a sheep returns. If you, the apprentice, do not understand this system - if it is magic that works for no apparent reason - then you will not know what to do if you accidentally drop an extra pebble into the bucket. That which you cannot make yourself, you cannot remake when the situation calls for it. You cannot go back to the source, tweak one of the parameter settings, and regenerate the output, without the source. If "Two plus four equals six" is a brute fact unto you, and then one of the elements changes to "five", how are you to know that "two plus five equals seven" when you were simply told that "two plus four equals six"?

If you see a small plant that drops a seed whenever a bird passes it, it will not occur to you that you can use this plant to partially automate the sheep-counter. Though you learned something that the original maker would use to improve on his invention, you can't go back to the source and re-create it.

When you contain the source of a thought, that thought can change along with you as you acquire new knowledge and new skills. When you contain the source of a thought, it becomes truly a part of you and grows along with you.

Strive to make yourself the source of every thought worth thinking. If the thought originally came from outside, make sure it comes from inside as well. Continually ask yourself: "How would I regenerate the thought if it were deleted?" When you have an answer, imagine that knowledge being deleted as well. And when you find a fountain, see what else it can pour.

Referenced by: Lost Purposes • Fake Fake Utility Functions • Fake Utility Functions • GAZP vs. GLUT • The Rhythm of Disagreement • Principles of Disagreement • Detached Lever Fallacy • Building Something Smarter • Living By Your Own Strength • You're Entitled to Arguments, But Not (That Particular) Proof

Original with comments: Truly Part Of You

Not for the Sake of Happiness (Alone)

Eliezer Yudkowsky, 22 November 2007 03:19AM

Followup to: Terminal Values and Instrumental Values

When I met the futurist Greg Stock some years ago, he argued that the joy of scientific discovery would soon be replaced by pills that could simulate the joy of scientific discovery. I approached him after his talk and said, "I agree that such pills are probably possible, but I wouldn't voluntarily take them."

And Stock said, "But they'll be so much better that the real thing won't be able to compete. It will just be way more fun for you to take the pills than to do all the actual scientific work."

And I said, "I agree that's possible, so I'll make sure never to take them."

Stock seemed genuinely surprised by my attitude, which genuinely surprised me.

One often sees ethicists arguing as if all human desires are reducible, in principle, to the desire for ourselves and others to be happy. (In particular, Sam Harris does this in The End of Faith, which I just finished perusing - though Harris's reduction is more of a drive-by shooting than a major topic of discussion.)

This isn't the same as arguing whether all happinesses can be measured on a common utility scale - different happinesses might occupy different scales, or be otherwise non-convertible. And it's not the same as arguing that it's theoretically impossible to value anything other than your own psychological states, because it's still permissible to care whether other people are happy.

The question, rather, is whether we should care about the things that make us happy, apart from any happiness they bring.

We can easily list many cases of moralists going astray by caring about things besides happiness. The various states and countries that still outlaw oral sex make a good example; these legislators would have been better off if they'd said, "Hey, whatever turns you on." But this doesn't show that all values are reducible to happiness; it just argues that in this particular case it was an ethical mistake to focus on anything else.

It is an undeniable fact that we tend to do things that make us happy, but this doesn't mean we should regard the happiness as the only reason for so acting. First, this would make it difficult to explain how we could care about anyone else's happiness - how we could treat people as ends in themselves, rather than instrumental means of obtaining a warm glow of satisfaction.

Second, just because something is a consequence of my action doesn't mean it was the sole justification. If I'm writing a blog post, and I get a headache, I may take an ibuprofen. One of the consequences of my action is that I experience less pain, but this doesn't mean it was the only consequence, or even the most important reason for my decision. I do value the state of not having a headache. But I can value something for its own sake and also value it as a means to an end.

For all value to be reducible to happiness, it's not enough to show that happiness is involved in most of our decisions - it's not even enough to show that happiness is the most important consequent in all of our decisions - it must be the only consequent. That's a tough standard to meet. (I originally found this point in a Sober and Wilson paper, not sure which one.)

If I claim to value art for its own sake, then would I value art that no one ever saw? A screensaver running in a closed room, producing beautiful pictures that no one ever saw? I'd have to say no. I can't think of any completely lifeless object that I would value as an end, not just a means. That would be like valuing ice cream as an end in itself, apart from anyone eating it. Everything I value, that I can think of, involves people and their experiences somewhere along the line.

The best way I can put it, is that my moral intuition appears to require both the objective and subjective component to grant full value.

The value of scientific discovery requires both a genuine scientific discovery, and a person to take joy in that discovery. It may seem difficult to disentangle these values, but the pills make it clearer.

I would be disturbed if people retreated into holodecks and fell in love with mindless wallpaper. I would be disturbed even if they weren't aware it was a holodeck, which is an important ethical issue if some agents can potentially transport people into holodecks and substitute zombies for their loved ones without their awareness. Again, the pills make it clearer: I'm not just concerned with my own awareness of the uncomfortable fact. I wouldn't put myself into a holodeck even if I could take a pill to forget the fact afterward. That's simply not where I'm trying to steer the future.

I value freedom: When I'm deciding where to steer the future, I take into account not only the subjective states that people end up in, but also whether they got there as a result of their own efforts. The presence or absence of an external puppet master can affect my valuation of an otherwise fixed outcome. Even if people wouldn't know they were being manipulated, it would matter to my judgment of how well humanity had done with its future. This is an important ethical issue, if you're dealing with agents powerful enough to helpfully tweak people's futures without their knowledge.

So my values are not strictly reducible to happiness: There are properties I value about the future that aren't reducible to activation levels in anyone's pleasure center; properties that are not strictly reducible to subjective states even in principle.

Which means that my decision system has a lot of terminal values, none of them strictly reducible to anything else. Art, science, love, lust, freedom, friendship...

And I'm okay with that. I value a life complicated enough to be challenging and aesthetic - not just the feeling that life is complicated, but the actual complications - so turning into a pleasure center in a vat doesn't appeal to me. It would be a waste of humanity's potential, which I value actually fulfilling, not just having the feeling that it was fulfilled.

Referenced by: Leaky Generalizations • The Hidden Complexity of Wishes • Lost Purposes • Fake Fake Utility Functions • The "Intuitions" Behind "Utilitarianism" • Heading Toward Morality • Setting Up Metaethics • High Challenge • Free to Optimize • Interpersonal Entanglement

Original with comments: Not for the Sake of Happiness (Alone)

Leaky Generalizations

Eliezer Yudkowsky, 22 November 2007 09:16PM

Followup to: Terminal Values and Instrumental Values

Are apples good to eat? Usually, but some apples are rotten.

Do humans have ten fingers? Most of us do, but plenty of people have lost a finger and nonetheless qualify as "human".

Unless you descend to a level of description far below any macroscopic object - below societies, below people, below fingers, below tendon and bone, below cells, all the way down to particles and fields where the laws are truly universal - then practically every generalization you use in the real world will be leaky.

(Though there may, of course, be some exceptions to the above rule...)

Mostly, the way you deal with leaky generalizations is that, well, you just have to deal. If the cookie market almost always closes at 10pm, except on Thanksgiving it closes at 6pm, and today happens to be National Native American Genocide Day, you'd better show up before 6pm or you won't get a cookie.

Our ability to manipulate leaky generalizations is opposed by need for closure, the degree to which we want to say once and for all that humans have fingers, and get frustrated when we have to tolerate continued ambiguity. Raising the value of the stakes can increase need for closure - which shuts down complexity tolerance when complexity tolerance is most needed.

Life would be complicated even if the things we wanted were simple (they aren't). The leakyness of leaky generalizations about what-to-do-next would leak in from the leaky structure of the real world. Or to put it another way:

Instrumental values often have no specification which is both compact and local.

Suppose there's a box containing a million dollars. The box is locked, not with an ordinary combination lock, but with a dozen keys controlling a machine that can open the box. If you know how the machine works, you can deduce which sequences of key-presses will open the box. There's more than one key sequence that can trigger the button. But if you press a sufficiently wrong sequence, the machine incinerates the money. And if you don't know about the machine, there's no simple rules like "Pressing any key three times opens the box" or "Pressing five different keys with no repetitions incinerates the money."

There's a compact nonlocal specification of which keys you want to press: You want to press keys such that they open the box. You can write a compact computer program that computes which key sequences are good, bad or neutral, but the computer program will need to describe the machine, not just the keys themselves.

There's likewise a local noncompact specification of which keys to press: a giant lookup table of the results for each possible key sequence. It's a very large computer program, but it makes no mention of anything except the keys.

But there's no way to describe which key sequences are good, bad, or neutral, which is both simple and phrased only in terms of the keys themselves.

It may be even worse if there are tempting local generalizations which turn out to be leaky. Pressing most keys three times in a row will open the box, but there's a particular key that incinerates the money if you press it just once. You might think you had found a perfect generalization - a locally describable class of sequences that always opened the box - when you had merely failed to visualize all the possible paths of the machine, or failed to value all the side effects.

The machine represents the complexity of the real world. The openness of the box (which is good) and the incinerator (which is bad) represent the thousand shards of desire that make up our terminal values. The keys represent the actions and policies and strategies available to us.

When you consider how many different ways we value outcomes, and how complicated are the paths we take to get there, it's a wonder that there exists any such thing as helpful ethical advice. (Of which the strangest of all advices, and yet still helpful, is that "The end does not justify the means.")

But conversely, the complicatedness of action need not say anything about the complexity of goals. You often find people who smile wisely, and say, "Well, morality is complicated, you know, female circumcision is right in one culture and wrong in another, it's not always a bad thing to torture people. How naive you are, how full of need for closure, that you think there are any simple rules."

You can say, unconditionally and flatly, that killing anyone is a huge dose of negative terminal utility. Yes, even Hitler. That doesn't mean you shouldn't shoot Hitler. It means that the net instrumental utility of shooting Hitler carries a giant dose of negative utility from Hitler's death, and an hugely larger dose of positive utility from all the other lives that would be saved as a consequence.

Many commit the type error that I warned against in Terminal Values and Instrumental Values, and think that if the net consequential expected utility of Hitler's death is conceded to be positive, then the immediate local terminal utility must also be positive, meaning that the moral principle "Death is always a bad thing" is itself a leaky generalization. But this is double counting, with utilities instead of probabilities; you're setting up a resonance between the expected utility and the utility, instead of a one-way flow from utility to expected utility.

Or maybe it's just the urge toward a one-sided policy debate: the best policy must have no drawbacks.

In my moral philosophy, the local negative utility of Hitler's death is stable, no matter what happens to the external consequences and hence to the expected utility.

Of course, you can set up a moral argument that it's an inherently a good thing to punish evil people, even with capital punishment for sufficiently evil people. But you can't carry this moral argument by pointing out that the consequence of shooting a man with a leveled gun may be to save other lives. This is appealing to the value of life, not appealing to the value of death. If expected utilities are leaky and complicated, it doesn't mean that utilities must be leaky and complicated as well. They might be! But it would be a separate argument.

Referenced by: The Hidden Complexity of Wishes • Lost Purposes • Fake Fake Utility Functions • Replace the Symbol with the Substance

Original with comments: Leaky Generalizations

The Hidden Complexity of Wishes

Eliezer Yudkowsky, 24 November 2007 12:12AM

Followup to: The Tragedy of Group Selectionism, Fake Optimization Criteria, Terminal Values and Instrumental Values, Artificial Addition, Leaky Generalizations

"I wish to live in the locations of my choice, in a physically healthy, uninjured, and apparently normal version of my current body containing my current mental state, a body which will heal from all injuries at a rate three sigmas faster than the average given the medical technology available to me, and which will be protected from any diseases, injuries or illnesses causing disability, pain, or degraded functionality or any sense, organ, or bodily function for more than ten days consecutively or fifteen days in any year..."
-- The Open-Source Wish Project, Wish For Immortality 1.1

There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent.

Suppose your aged mother is trapped in a burning building, and it so happens that you're in a wheelchair; you can't rush in yourself. You could cry, "Get my mother out of that building!" but there would be no one to hear.

Luckily you have, in your pocket, an Outcome Pump. This handy device squeezes the flow of time, pouring probability into some outcomes, draining it from others.

The Outcome Pump is not sentient. It contains a tiny time machine, which resets time unless a specified outcome occurs. For example, if you hooked up the Outcome Pump's sensors to a coin, and specified that the time machine should keep resetting until it sees the coin come up heads, and then you actually flipped the coin, you would see the coin come up heads. (The physicists say that any future in which a "reset" occurs is inconsistent, and therefore never happens in the first place - so you aren't actually killing any versions of yourself.)

Whatever proposition you can manage to input into the Outcome Pump, somehow happens, though not in a way that violates the laws of physics. If you try to input a proposition that's too unlikely, the time machine will suffer a spontaneous mechanical failure before that outcome ever occurs.

You can also redirect probability flow in more quantitative ways using the "future function" to scale the temporal reset probability for different outcomes. If the temporal reset probability is 99% when the coin comes up heads, and 1% when the coin comes up tails, the odds will go from 1:1 to 99:1 in favor of tails. If you had a mysterious machine that spit out money, and you wanted to maximize the amount of money spit out, you would use reset probabilities that diminished as the amount of money increased. For example, spitting out $10 might have a 99.999999% reset probability, and spitting out $100 might have a 99.99999% reset probability. This way you can get an outcome that tends to be as high as possible in the future function, even when you don't know the best attainable maximum.

So you desperately yank the Outcome Pump from your pocket - your mother is still trapped in the burning building, remember? - and try to describe your goal: get your mother out of the building!

The user interface doesn't take English inputs. The Outcome Pump isn't sentient, remember? But it does have 3D scanners for the near vicinity, and built-in utilities for pattern matching. So you hold up a photo of your mother's head and shoulders; match on the photo; use object contiguity to select your mother's whole body (not just her head and shoulders); and define the future function using your mother's distance from the building's center. The further she gets from the building's center, the less the time machine's reset probability.

You cry "Get my mother out of the building!", for luck, and press Enter.

For a moment it seems like nothing happens. You look around, waiting for the fire truck to pull up, and rescuers to arrive - or even just a strong, fast runner to haul your mother out of the building -

BOOM! With a thundering roar, the gas main under the building explodes. As the structure comes apart, in what seems like slow motion, you glimpse your mother's shattered body being hurled high into the air, traveling fast, rapidly increasing its distance from the former center of the building.

On the side of the Outcome Pump is an Emergency Regret Button. All future functions are automatically defined with a huge negative value for the Regret Button being pressed - a temporal reset probability of nearly 1 - so that the Outcome Pump is extremely unlikely to do anything which upsets the user enough to make them press the Regret Button. You can't ever remember pressing it. But you've barely started to reach for the Regret Button (and what good will it do now?) when a flaming wooden beam drops out of the sky and smashes you flat.

Which wasn't really what you wanted, but scores very high in the defined future function...

The Outcome Pump is a genie of the second class. No wish is safe.

If someone asked you to get their poor aged mother out of a burning building, you might help, or you might pretend not to hear. But it wouldn't even occur to you to explode the building. "Get my mother out of the building" sounds like a much safer wish than it really is, because you don't even consider the plans that you assign extreme negative values.

Consider again the Tragedy of Group Selectionism: Some early biologists asserted that group selection for low subpopulation sizes would produce individual restraint in breeding; and yet actually enforcing group selection in the laboratory produced cannibalism, especially of immature females. It's obvious in hindsight that, given strong selection for small subpopulation sizes, cannibals will outreproduce individuals who voluntarily forego reproductive opportunities. But eating little girls is such an un-aesthetic solution that Wynne-Edwards, Allee, Brereton, and the other group-selectionists simply didn't think of it. They only saw the solutions they would have used themselves.

Suppose you try to patch the future function by specifying that the Outcome Pump should not explode the building: outcomes in which the building materials are distributed over too much volume, will have ~1 temporal reset probabilities.

So your mother falls out of a second-story window and breaks her neck. The Outcome Pump took a different path through time that still ended up with your mother outside the building, and it still wasn't what you wanted, and it still wasn't a solution that would occur to a human rescuer.

If only the Open-Source Wish Project had developed a Wish To Get Your Mother Out Of A Burning Building:

"I wish to move my mother (defined as the woman who shares half my genes and gave birth to me) to outside the boundaries of the building currently closest to me which is on fire; but not by exploding the building; nor by causing the walls to crumble so that the building no longer has boundaries; nor by waiting until after the building finishes burning down for a rescue worker to take out the body..."

All these special cases, the seemingly unlimited number of required patches, should remind you of the parable of Artificial Addition - programming an Arithmetic Expert Systems by explicitly adding ever more assertions like "fifteen plus fifteen equals thirty, but fifteen plus sixteen equals thirty-one instead".

How do you exclude the outcome where the building explodes and flings your mother into the sky? You look ahead, and you foresee that your mother would end up dead, and you don't want that consequence, so you try to forbid the event leading up to it.

Your brain isn't hardwired with a specific, prerecorded statement that "Blowing up a burning building containing my mother is a bad idea." And yet you're trying to prerecord that exact specific statement in the Outcome Pump's future function. So the wish is exploding, turning into a giant lookup table that records your judgment of every possible path through time.

You failed to ask for what you really wanted. You wanted your mother to go on living, but you wished for her to become more distant from the center of the building.

Except that's not all you wanted. If your mother was rescued from the building but was horribly burned, that outcome would rank lower in your preference ordering than an outcome where she was rescued safe and sound. So you not only value your mother's life, but also her health.

And you value not just her bodily health, but her state of mind. Being rescued in a fashion that traumatizes her - for example, a giant purple monster roaring up out of nowhere and seizing her - is inferior to a fireman showing up and escorting her out through a non-burning route. (Yes, we're supposed to stick with physics, but maybe a powerful enough Outcome Pump has aliens coincidentally showing up in the neighborhood at exactly that moment.) You would certainly prefer her being rescued by the monster to her being roasted alive, however.

How about a wormhole spontaneously opening and swallowing her to a desert island? Better than her being dead; but worse than her being alive, well, healthy, untraumatized, and in continual contact with you and the other members of her social network.

Would it be okay to save your mother's life at the cost of the family dog's life, if it ran to alert a fireman but then got run over by a car? Clearly yes, but it would be better ceteris paribus to avoid killing the dog. You wouldn't want to swap a human life for hers, but what about the life of a convicted murderer? Does it matter if the murderer dies trying to save her, from the goodness of his heart? How about two murderers? If the cost of your mother's life was the destruction of every extant copy, including the memories, of Bach's Little Fugue in G Minor, would that be worth it? How about if she had a terminal illness and would die anyway in eighteen months?

If your mother's foot is crushed by a burning beam, is it worthwhile to extract the rest of her? What if her head is crushed, leaving her body? What if her body is crushed, leaving only her head? What if there's a cryonics team waiting outside, ready to suspend the head? Is a frozen head a person? Is Terry Schiavo a person? How much is a chimpanzee worth?

Your brain is not infinitely complicated; there is only a finite Kolmogorov complexity / message length which suffices to describe all the judgments you would make. But just because this complexity is finite does not make it small. We value many things, and no they are not reducible to valuing happiness or valuing reproductive fitness.

There is no safe wish smaller than an entire human morality. There are too many possible paths through Time. You can't visualize all the roads that lead to the destination you give the genie. "Maximizing the distance between your mother and the center of the building" can be done even more effectively by detonating a nuclear weapon. Or, at higher levels of genie power, flinging her body out of the Solar System. Or, at higher levels of genie intelligence, doing something that neither you nor I would think of, just like a chimpanzee wouldn't think of detonating a nuclear weapon. You can't visualize all the paths through time, any more than you can program a chess-playing machine by hardcoding a move for every possible board position.

And real life is far more complicated than chess. You cannot predict, in advance, which of your values will be needed to judge the path through time that the genie takes. Especially if you wish for something longer-term or wider-range than rescuing your mother from a burning building.

I fear the Open-Source Wish Project is futile, except as an illustration of how not to think about genie problems. The only safe genie is a genie that shares all your judgment criteria, and at that point, you can just say "I wish for you to do what I should wish for." Which simply runs the genie's should function.

Indeed, it shouldn't be necessary to say anything. To be a safe fulfiller of a wish, a genie must share the same values that led you to make the wish. Otherwise the genie may not choose a path through time which leads to the destination you had in mind, or it may fail to exclude horrible side effects that would lead you to not even consider a plan in the first place. Wishes are leaky generalizations, derived from the huge but finite structure that is your entire morality; only by including this entire structure can you plug all the leaks.

With a safe genie, wishing is superfluous. Just run the genie.

Referenced by: Lost Purposes • Fake Fake Utility Functions • Spooky Action at a Distance: The No-Communication Theorem • Ghosts in the Machine • Heading Toward Morality • Contaminated by Optimism • Morality as Fixed Computation • Unnatural Categories • Magical Categories • My Naturalistic Awakening • What I Think, If Not Why • Devil's Offers • In Praise of Boredom • Value is Fragile

Original with comments: The Hidden Complexity of Wishes

Lost Purposes

Eliezer Yudkowsky, 25 November 2007 09:01AM

It was in either kindergarten or first grade that I was first asked to pray, given a transliteration of a Hebrew prayer. I asked what the words meant. I was told that so long as I prayed in Hebrew, I didn't need to know what the words meant, it would work anyway.

That was the beginning of my break with Judaism.

As you read this, some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper.

Maybe you realized it was all madness, but I bet you did it anyway. You didn't have a choice, right?

A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation. Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level. I seem to recall (though I can't find the source) that just taking mandatory tests was 40% of classroom time in one school.

The old Soviet bureaucracy was famous for being more interested in appearances than reality. One shoe factory overfulfilled its quota by producing lots of tiny shoes. Another shoe factory reported cut but unassembled leather as a "shoe". The superior bureaucrats weren't interested in looking too hard, because they also wanted to report quota overfulfillments. All this was a great help to the comrades freezing their feet off.

It is now being suggested in several sources that an actual majority of published findings in medicine, though "statistically significant with p<0.05", are untrue. But so long as p<0.05 remains the threshold for publication, why should anyone hold themselves to higher standards, when that requires bigger research grants for larger experimental groups, and decreases the likelihood of getting a publication? Everyone knows that the whole point of science is to publish lots of papers, just as the whole point of a university is to print certain pieces of parchment, and the whole point of a school is to pass the mandatory tests that guarantee the annual budget. You don't get to set the rules of the game, and if you try to play by different rules, you'll just lose.

(Though for some reason, physics journals require a threshold of p<0.0001. It's as if they conceive of some other purpose to their existence than publishing physics papers.)

There's chocolate at the supermarket, and you can get to the supermarket by driving, and driving requires that you be in the car, which means opening your car door, which needs keys. If you find there's no chocolate at the supermarket, you won't stand around opening and slamming your car door because the car door still needs opening. I rarely notice people losing track of plans they devised themselves.

It's another matter when incentives must flow through large organizations - or worse, many different organizations and interest groups, some of them governmental. Then you see behaviors that would mark literal insanity, if they were born from a single mind. Someone gets paid every time they open a car door, because that's what's measurable; and this person doesn't care whether the driver ever gets paid for arriving at the supermarket, let alone whether the buyer purchases the chocolate, or whether the eater is happy or starving.

From a Bayesian perspective, subgoals are epiphenomena of conditional probability functions. There is no expected utility without utility. How silly would it be to think that instrumental value could take on a mathematical life of its own, leaving terminal value in the dust? It's not sane by decision-theoretical criteria of sanity.

But consider the No Child Left Behind Act. The politicians want to look like they're doing something about educational difficulties; the politicians have to look busy to voters this year, not fifteen years later when the kids are looking for jobs. The politicians are not the consumers of education. The bureaucrats have to show progress, which means that they're only interested in progress that can be measured this year. They aren't the ones who'll end up ignorant of science. The publishers who commission textbooks, and the committees that purchase textbooks, don't sit in the classrooms bored out of their skulls.

The actual consumers of knowledge are the children - who can't pay, can't vote, can't sit on the committees. Their parents care for them, but don't sit in the classes themselves; they can only hold politicians responsible according to surface images of "tough on education". Politicians are too busy being re-elected to study all the data themselves; they have to rely on surface images of bureaucrats being busy and commissioning studies - it may not work to help any children, but it works to let politicians appear caring. Bureaucrats don't expect to use textbooks themselves, so they don't care if the textbooks are hideous to read, so long as the process by which they are purchased looks good on the surface. The textbook publishers have no motive to produce bad textbooks, but they know that the textbook purchasing committee will be comparing textbooks based on how many different subjects they cover, and that the fourth-grade purchasing committee isn't coordinated with the third-grade purchasing committee, so they cram as many subjects into one textbook as possible. Teachers won't get through a fourth of the textbook before the end of the year, and then the next year's teacher will start over. Teachers might complain, but they aren't the decision-makers, and ultimately, it's not their future on the line, which puts sharp bounds on how much effort they'll spend on unpaid altruism...

It's amazing, when you look at it that way - consider all the lost information and lost incentives - that anything at all remains of the original purpose, gaining knowledge. Though many educational systems seem to be currently in the process of collapsing into a state not much better than nothing.

Want to see the problem really solved? Make the politicians go to school.

A single human mind can track a probabilistic expectation of utility as it flows through the conditional chances of a dozen intermediate events - including nonlocal dependencies, places where the expected utility of opening the car door depends on whether there's chocolate in the supermarket. But organizations can only reward today what is measurable today, what can be written into legal contract today, and this means measuring intermediate events rather than their distant consequences. These intermediate measures, in turn, are leaky generalizations - often very leaky. Bureaucrats are untrustworthy genies, for they do not share the values of the wisher.

Miyamoto Musashi said:

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him. You must thoroughly research this."

(I wish I lived in an era where I could just tell my readers they have to thoroughly research something, without giving insult.)

Why would any individual lose track of their purposes in a swordfight? If someone else had taught them to fight, if they had not generated the entire art from within themselves, they might not understand the reason for parrying at one moment, or springing at another moment; they might not realize when the rules had exceptions, fail to see the times when the usual method won't cut through.

The essential thing in the art of epistemic rationality is to understand how every rule is cutting through to the truth in the same movement. The corresponding essential of pragmatic rationality - decision theory, versus probability theory - is to always see how every expected utility cuts through to utility. You must thoroughly research this.

C. J. Cherryh said:

"Your sword has no blade. It has only your intention. When that goes astray you have no weapon."

I have seen many people go astray when they wish to the genie of an imagined AI, dreaming up wish after wish that seems good to them, sometimes with many patches and sometimes without even that pretense of caution. And they don't jump to the meta-level. They don't instinctively look-to-purpose, the instinct that started me down the track to atheism at the age of five. They do not ask, as I reflexively ask, "Why do I think this wish is a good idea? Will the genie judge likewise?" They don't see the source of their judgment, hovering behind the judgment as its generator. They lose track of the ball; they know the ball bounced, but they don't instinctively look back to see where it bounced from - the criterion that generated their judgments.

Likewise with people not automatically noticing when supposedly selfish people give altruistic arguments in favor of selfishness, or when supposedly altruistic people give selfish arguments in favor of altruism.

People can handle goal-tracking for driving to the supermarket just fine, when it's all inside their own heads, and no genies or bureaucracies or philosophies are involved. The trouble is that real civilization is immensely more complicated than this. Dozens of organizations, and dozens of years, intervene between the child suffering in the classroom, and the new-minted college graduate not being very good at their job. (But will the interviewer or manager notice, if the college graduate is good at looking busy?) With every new link that intervenes between the action and its consequence, intention has one more chance to go astray. With every intervening link, information is lost, incentive is lost. And this bothers most people a lot less than it bothers me, or why were all my classmates willing to say prayers without knowing what they meant? They didn't feel the same instinct to look-to-the-generator.

Can people learn to keep their eye on the ball? To keep their intention from going astray? To never spring or strike or touch, without knowing the higher goal they will complete in the same movement? People do often want to do their jobs, all else being equal. Can there be such a thing as a sane corporation? A sane civilization, even? That's only a distant dream, but it's what I've been getting at with all these blog posts on the flow of intentions (aka expected utility, aka instrumental value) without losing purpose (aka utility, aka terminal value). Can people learn to feel the flow of parent goals and child goals? To know consciously, as well as implicitly, the distinction between expected utility and utility?

Do you care about threats to your civilization? The worst metathreat to complex civilization is its own complexity, for that complication leads to the loss of many purposes.

I look back, and I see that more than anything, my life has been driven by an exceptionally strong abhorrence to lost purposes. I hope it can be transformed to a learnable skill.

Referenced by: Purpose and Pragmatism • Superhero Bias • Fake Fake Utility Functions • Fake Utility Functions • Effortless Technique • Zen and the Art of Rationality • Replace the Symbol with the Substance • Decoherent Essences • When (Not) To Use Probabilities • The Comedy of Behaviorism • Invisible Frameworks • Trying to Try • Emotional Involvement • 3 Levels of Rationality Verification

Original with comments: Lost Purposes

Purpose and Pragmatism

Eliezer Yudkowsky, 26 November 2007 06:51AM

Followup to: Making Beliefs Pay Rent, Lost Purposes

Thus runs the ancient parable:

If a tree falls in a forest and no one hears it, does it make a sound?
One says, "Yes it does, for it makes vibrations in the air."
Another says, "No it does not, for there is no auditory processing in any brain."

So begins a long, acrimonious battle...

The conventional resolution is that the two are fighting over the definition of a word, and such labels do not have intrinsic definitions, only agreed-upon definitions.

Yet if you need to know about the forest for any pragmatic reason - if there is anything you plan on doing with the knowledge - then the answer is no longer a matter of mutual agreement. If, for example, you need to know whether landmines will be set off by the tree falling, then you cannot make the land mines explode or unexplode by any possible amount of agreement about the meaning of the word "sound". You can get the whole world to agree, one way or the other, and it still won't make a difference.

You find yourself in an unheard-falling-tree dilemma, only when you become curious about a question with no pragmatic use, and no predictive consequences. Which suggests that you may be playing loose with your purposes.

So does this mean that truth reduces to usefulness? But this, itself, would be a purpose-loss, a subgoal stomp, a mistaking of the indicator for the indicated. Usefulness for prediction, and demonstrated powers of manipulation, is one of the best indicators of truth. This does not mean that usefulness is truth. You might as well say that the act of driving to the supermarket is eating chocolate.

There is, nonetheless, a deep similarity between the pragmatic and the epistemic arts of rationality, in the matter of keeping your eye on the ball.

In pragmatic rationality, keeping your eye on the ball means holding to your purpose: Being aware of how each act leads to its consequence, and not losing sight of utilities in leaky generalizations about expected utilities. If you hold firmly in your mind the image of a drained swamp, you will be less likely to get lost in fighting alligators.

In epistemic rationality, keeping your eye on the ball means holding to your question: Being aware of what each indicator says about its indicatee, and not losing sight of the original question in fights over indicators. If you want to know whether landmines will detonate, you will not get lost in fighting over the meaning of the word "sound".

Both cases deal with leaky generalizations about conditional probabilities: P(Y=y|X=x) is nearly but not quite 1.

In the case of pragmatic rationality: driving to the supermarket may almost always get you chocolate, but on some occasions it will not. If you forget your final purpose and think that x=y then you will not be able to deal with cases where the supermarket is out of chocolate.

In the case of epistemic rationality: seeing a "Chocolate for sale" sign in the supermarket may almost always indicate that chocolate is available, but on some occasions it will not. If you forget your original question and think that x=y then you will go on arguing "But the sign is up!" even when someone calls out to you, "Hey, they don't have any chocolate today!"

This is a deep connection between the human arts of pragmatic and epistemic rationality...

...which does not mean they are the same thing.

Referenced by: Uncritical Supercriticality • Fake Fake Utility Functions

Original with comments: Purpose and Pragmatism

The Affect Heuristic

Eliezer Yudkowsky, 27 November 2007 07:58AM

The affect heuristic is when subjective impressions of goodness/badness act as a heuristic - a source of fast, perceptual judgments. Pleasant and unpleasant feelings are central to human reasoning, and the affect heuristic comes with lovely biases - some of my favorites.

Let's start with one of the relatively less crazy biases. You're about to move to a new city, and you have to ship an antique grandfather clock. In the first case, the grandfather clock was a gift from your grandparents on your 5th birthday. In the second case, the clock was a gift from a remote relative and you have no special feelings for it. How much would you pay for an insurance policy that paid out $100 if the clock were lost in shipping? According to Hsee and Kunreuther (2000), subjects stated willingness to pay more than twice as much in the first condition. This may sound rational - why not pay more to protect the more valuable object? - until you realize that the insurance doesn't protect the clock, it just pays if the clock is lost, and pays exactly the same amount for either clock. (And yes, it was stated that the insurance was with an outside company, so it gives no special motive to the movers.)

All right, but that doesn't sound too insane. Maybe you could get away with claiming the subjects were insuring affective outcomes, not financial outcomes - purchase of consolation.

Then how about this? Yamagishi (1997) showed that subjects judged a disease as more dangerous when it was described as killing 1,286 people out of every 10,000, versus a disease that was 24.14% likely to be fatal. Apparently the mental image of a thousand dead bodies is much more alarming, compared to a single person who's more likely to survive than not.

But wait, it gets worse.

Suppose an airport must decide whether to spend money to purchase some new equipment, while critics argue that the money should be spent on other aspects of airport safety. Slovic et. al. (2002) presented two groups of subjects with the arguments for and against purchasing the equipment, with a response scale ranging from 0 (would not support at all) to 20 (very strong support). One group saw the measure described as saving 150 lives. The other group saw the measure described as saving 98% of 150 lives. The hypothesis motivating the experiment was that saving 150 lives sounds vaguely good - is that a lot? a little? - while saving 98% of something is clearly very good because 98% is so close to the upper bound of the percentage scale. Lo and behold, saving 150 lives had mean support of 10.4, while saving 98% of 150 lives had mean support of 13.6.

Or consider the report of Denes-Raj and Epstein (1994): Subjects offered an opportunity to win $1 each time they randomly drew a red jelly bean from a bowl, often preferred to draw from a bowl with more red beans and a smaller proportion of red beans. E.g., 7 in 100 was preferred to 1 in 10.

According to Denes-Raj and Epstein, these subjects reported afterward that even though they knew the probabilities were against them, they felt they had a better chance when there were more red beans. This may sound crazy to you, oh Statistically Sophisticated Reader, but if you think more carefully you'll realize that it makes perfect sense. A 7% probability versus 10% probability may be bad news, but it's more than made up for by the increased number of red beans. It's a worse probability, yes, but you're still more likely to win, you see. You should meditate upon this thought until you attain enlightenment as to how the rest of the planet thinks about probability.

Finucane et. al. (2000) tested the theory that people would conflate their judgments about particular good/bad aspects of something into an overall good or bad feeling about that thing. For example, information about a possible risk, or possible benefit, of nuclear power plants. Logically, information about risk doesn't have to bear any relation to information about benefits. If it's a physical fact about a reactor design that it's passively safe (won't go supercritical even if the surrounding coolant systems and so on break down), this doesn't imply that the reactor will necessarily generate less waste, or produce electricity at a lower cost, etcetera. All these things would be good, but they are not the same good thing. Nonetheless, Finucane et. al. found that for nuclear reactors, natural gas, and food preservatives, presenting information about high benefits made people perceive lower risks; presenting information about higher risks made people perceive lower benefits; and so on across the quadrants.

Finucane et. al. also found that time pressure greatly increased the inverse relationship between perceived risk and perceived benefit, consistent with the general finding that time pressure, poor information, or distraction all increase the dominance of perceptual heuristics over analytic deliberation.

Ganzach (2001) found the same effect in the realm of finance. According to ordinary economic theory, return and risk should correlate positively - or to put it another way, people pay a premium price for safe investments, which lowers the return; stocks deliver higher returns than bonds, but have correspondingly greater risk. When judging familiar stocks, analysts' judgments of risks and returns were positively correlated, as conventionally predicted. But when judging unfamiliar stocks, analysts tended to judge the stocks as if they were generally good or generally bad - low risk and high returns, or high risk and low returns.

For further reading I recommend the fine summary chapter in Slovic et. al. 2002: "Rational Actors or Rational Fools: Implications of the Affect Heuristic for Behavioral Economics."

Denes-Raj, V., & Epstein, S. (1994). Conflict between intuitive and rational processing: When people behave against their better judgment. Journal of Personality and Social Psychology, 66, 819-829.

Finucane, M. L., Alhakami, A., Slovic, P., & Johnson, S. M. (2000). The affect heuristic in judgments of risks and benefits. Journal of Behavioral Decision Making, 13, 1-17.

Ganzach, Y. (2001). Judging risk and return of financial assets. Organizational Behavior and Human Decision Processes, 83, 353-370.

Hsee, C. K. & Kunreuther, H. (2000). The affection effect in insurance decisions. Journal of Risk and Uncertainty, 20, 141-159.

Slovic, P., Finucane, M., Peters, E. and MacGregor, D. 2002. Rational Actors or Rational Fools: Implications of the Affect Heuristic for Behavioral Economics. Journal of Socio-Economics, 31: 329–342.

Yamagishi, K. (1997). When a 12.86% mortality is more dangerous than 24.14%: Implications for risk communication. Applied Cognitive Psychology, 11, 495-506.

Sequence: Death Spirals and the Cult Attractor

Evaluability (And Cheap Holiday Shopping)

Referenced by: Evaluability (And Cheap Holiday Shopping) • The Halo Effect • Affective Death Spirals • Fake Fake Utility Functions • Stop Voting For Nincompoops • Rational vs. Scientific Ev-Psych • Heading Toward Morality • Harder Choices Matter Less • On Doing the Impossible • Failure By Affective Analogy • Recursive Self-Improvement • Growing Up is Hard • Special Status Needs Special Support

Original with comments: The Affect Heuristic

Evaluability (And Cheap Holiday Shopping)

Eliezer Yudkowsky, 28 November 2007 12:37AM

Followup to: The Affect Heuristic

With the expensive part of the Hallowthankmas season now approaching, a question must be looming large in our readers' minds:

"Dear Overcoming Bias, are there biases I can exploit to be seen as generous without actually spending lots of money?"

I'm glad to report the answer is yes! According to Hsee (1998) - in a paper entitled "Less is better: When low-value options are valued more highly than high-value options" - if you buy someone a $45 scarf, you are more likely to be seen as generous than if you buy them a $55 coat.

This is a special case of a more general phenomenon. An earlier experiment, Hsee (1996), asked subjects how much they would be willing to pay for a second-hand music dictionary:

- Dictionary A, from 1993, with 10,000 entries, in like-new condition.

- Dictionary B, from 1993, with 20,000 entries, with a torn cover and otherwise in like-new condition.

The gotcha was that some subjects saw both dictionaries side-by-side, while other subjects only saw one dictionary...

Subjects who saw only one of these options were willing to pay an average of $24 for Dictionary A and an average of $20 for Dictionary B. Subjects who saw both options, side-by-side, were willing to pay $27 for Dictionary B and $19 for Dictionary A.

Of course, the number of entries in a dictionary is more important than whether it has a torn cover, at least if you ever plan on using it for anything. But if you're only presented with a single dictionary, and it has 20,000 entries, the number 20,000 doesn't mean very much. Is it a little? A lot? Who knows? It's non-evaluable. The torn cover, on the other hand - that stands out. That has a definite affective valence: namely, bad.

Seen side-by-side, though, the number of entries goes from non-evaluable to evaluable, because there are two compatible quantities to be compared. And, once the number of entries becomes evaluable, that facet swamps the importance of the torn cover.

From Slovic et. al. (2002): Would you prefer:

1. A 29/36 chance to win $2

2. A 7/36 chance to win $9

While the average prices (equivalence values) placed on these options were $1.25 and $2.11 respectively, their mean attractiveness ratings were 13.2 and 7.5. Both the prices and the attractiveness rating were elicited in a context where subjects were told that two gambles would be randomly selected from those rated, and they would play the gamble with the higher price or higher attractiveness rating. (Subjects had a motive to rate gambles as more attractive, or price them higher, that they would actually prefer to play.)

The gamble worth more money seemed less attractive, a classic preference reversal. The researchers hypothesized that the dollar values were more compatible with the pricing task, but the probability of payoff was more compatible with attractiveness. So (the researchers thought) why not try to make the gamble's payoff more emotionally salient - more affectively evaluable - more attractive?

And how did they do this? By adding a very small loss to the gamble. The old gamble had a 7/36 chance of winning $9. The new gamble had a 7/36 chance of winning $9 and a 29/36 chance of losing 5¢. In the old gamble, you implicitly evaluate the attractiveness of $9. The new gamble gets you to evaluate the attractiveness of winning $9 versus losing 5¢.

"The results," said Slovic. et. al., "exceeded our expectations." In a new experiment, the simple gamble with a 7/36 chance of winning $9 had a mean attractiveness rating of 9.4, while the complex gamble that included a 29/36 chance of losing 5¢ had a mean attractiveness rating of 14.9.

A follow-up experiment tested whether subjects preferred the old gamble to a certain gain of $2. Only 33% of students preferred the old gamble. Among another group asked to choose between a certain $2 and the new gamble (with the added possibility of a 5¢ loss), fully 60.8% preferred the gamble. After all, $9 isn't a very attractive amount of money, but $9/5¢ is an amazingly attractive win/loss ratio.

You can make a gamble more attractive by adding a strict loss! Isn't psychology fun? This is why no one who truly appreciates the wondrous intricacy of human intelligence wants to design a human-like AI.

Of course, it only works if the subjects don't see the two gambles side-by-side.

Similarly, which of these two ice creams do you think subjects in Hsee (1998) preferred?

Naturally, the answer depends on whether the subjects saw a single ice cream, or the two side-by-side. Subjects who saw a single ice cream were willing to pay $1.66 to Vendor H and $2.26 to Vendor L. Subjects who saw both ice creams were willing to pay $1.85 to Vendor H and $1.56 to Vendor L.

What does this suggest for your holiday shopping? That if you spend $400 on a 16GB iPod Touch, your recipient sees the most expensive MP3 player. If you spend $400 on a Nintendo Wii, your recipient sees the least expensive game machine. Which is better value for the money? Ah, but that question only makes sense if you see the two side-by-side. You'll think about them side-by-side while you're shopping, but the recipient will only see what they get.

If you have a fixed amount of money to spend - and your goal is to display your friendship, rather than to actually help the recipient - you'll be better off deliberately not shopping for value. Decide how much money you want to spend on impressing the recipient, then find the most worthless object which costs that amount. The cheaper the class of objects, the more expensive a particular object will appear, given that you spend a fixed amount. Which is more memorable, a $25 shirt or a $25 candle?

Gives a whole new meaning to the Japanese custom of buying $50 melons, doesn't it? You look at that and shake your head and say "What is it with the Japanese?". And yet they get to be perceived as incredibly generous, spendthrift even, while spending only $50. You could spend $200 on a fancy dinner and not appear as wealthy as you can by spending $50 on a melon. If only there was a custom of gifting $25 toothpicks or $10 dust specks; they could get away with spending even less.

PS: If you actually use this trick, I want to know what you bought.

Hsee, C. K. (1996). The evaluability hypothesis: An explanation for preference reversals between joint and separate evaluations of alternatives. Organizational Behavior and Human Decision Processes, 67, 242-257.

Hsee, C. K. (1998). Less is better: When low-value options are valued more highly than high-value options. Journal of Behavioral Decision Making, 11, 107-121.

Slovic, P., Finucane, M., Peters, E. and MacGregor, D. (2002.) Rational Actors or Rational Fools: Implications of the Affect Heuristic for Behavioral Economics. Journal of Socio-Economics, 31: 329–342.

Sequence: Death Spirals and the Cult Attractor

The Affect Heuristic

Unbounded Scales, Huge Jury Awards, & Futurism

Referenced by: Unbounded Scales, Huge Jury Awards, & Futurism • Fake Fake Utility Functions • Less Meta

Original with comments: Evaluability (And Cheap Holiday Shopping)

Unbounded Scales, Huge Jury Awards, & Futurism

Eliezer Yudkowsky, 29 November 2007 07:45AM

Followup to: Evaluability

"Psychophysics", despite the name, is the respectable field that links physical effects to sensory effects. If you dump acoustic energy into air - make noise - then how loud does that sound to a person, as a function of acoustic energy? How much more acoustic energy do you have to pump into the air, before the noise sounds twice as loud to a human listener? It's not twice as much; more like eight times as much.

Acoustic energy and photons are straightforward to measure. When you want to find out how loud an acoustic stimulus sounds, how bright a light source appears, you usually ask the listener or watcher. This can be done using a bounded scale from "very quiet" to "very loud", or "very dim" to "very bright". You can also use an unbounded scale, whose zero is "not audible at all" or "not visible at all", but which increases from there without limit. When you use an unbounded scale, the observer is typically presented with a constant stimulus, the modulus, which is given a fixed rating. For example, a sound that is assigned a loudness of 10. Then the observer can indicate a sound twice as loud as the modulus by writing 20.

And this has proven to be a fairly reliable technique. But what happens if you give subjects an unbounded scale, but no modulus? 0 to infinity, with no reference point for a fixed value? Then they make up their own modulus, of course. The ratios between stimuli will continue to correlate reliably between subjects. Subject A says that sound X has a loudness of 10 and sound Y has a loudness of 15. If subject B says that sound X has a loudness of 100, then it's a good guess that subject B will assign loudness in the range of 150 to sound Y. But if you don't know what subject C is using as their modulus - their scaling factor - then there's no way to guess what subject C will say for sound X. It could be 1. It could be 1000.

For a subject rating a single sound, on an unbounded scale, without a fixed standard of comparison, nearly all the variance is due to the arbitrary choice of modulus, rather than the sound itself.

"Hm," you think to yourself, "this sounds an awful lot like juries deliberating on punitive damages. No wonder there's so much variance!" An interesting analogy, but how would you go about demonstrating it experimentally?

Kahneman et. al., 1998 and 1999, presented 867 jury-eligible subjects with descriptions of legal cases (e.g., a child whose clothes caught on fire) and asked them to either

1. Rate the outrageousness of the defendant's actions, on a bounded scale

2. Rate the degree to which the defendant should be punished, on a bounded scale, or

3. Assign a dollar value to punitive damages

And, lo and behold, while subjects correlated very well with each other in their outrage ratings and their punishment ratings, their punitive damages were all over the map. Yet subjects' rank-ordering of the punitive damages - their ordering from lowest award to highest award - correlated well across subjects.

If you asked how much of the variance in the "punishment" scale could be explained by the specific scenario - the particular legal case, as presented to multiple subjects - then the answer, even for the raw scores, was .49. For the rank orders of the dollar responses, the amount of variance predicted was .51. For the raw dollar amounts, the variance explained was .06!

Which is to say: if you knew the scenario presented - the aforementioned child whose clothes caught on fire - you could take a good guess at the punishment rating, and a good guess at the rank-ordering of the dollar award relative to other cases, but the dollar award itself would be completely unpredictable.

Taking the median of twelve randomly selected responses didn't help much either.

So a jury award for punitive damages isn't so much an economic valuation as an attitude expression - a psychophysical measure of outrage, expressed on an unbounded scale with no standard modulus.

I observe that many futuristic predictions are, likewise, best considered as attitude expressions. Take the question, "How long will it be until we have human-level AI?" The responses I've seen to this are all over the map. On one memorable occasion, a mainstream AI guy said to me, "Five hundred years." (!!)

Now the reason why time-to-AI is just not very predictable, is a long discussion in its own right. But it's not as if the guy who said "Five hundred years" was looking into the future to find out. And he can't have gotten the number using the standard bogus method with Moore's Law. So what did the number 500 mean?

As far as I can guess, it's as if I'd asked, "On a scale where zero is 'not difficult at all', how difficult does the AI problem feel to you?" If this were a bounded scale, every sane respondent would mark "extremely hard" at the right-hand end. Everything feels extremely hard when you don't know how to do it. But instead there's an unbounded scale with no standard modulus. So people just make up a number to represent "extremely difficult", which may come out as 50, 100, or even 500. Then they tack "years" on the end, and that's their futuristic prediction.

"How hard does the AI problem feel?" isn't the only substitutable question. Others respond as if I'd asked "How positive do you feel about AI?", only lower numbers mean more positive feelings, and then they also tack "years" on the end. But if these "time estimates" represent anything other than attitude expressions on an unbounded scale with no modulus, I have been unable to determine it.

Kahneman, D., Schkade, D. A., and Sunstein, C. 1998. Shared Outrage and Erratic Awards: The Psychology of Punitive Damages. Journal of Risk and Uncertainty 16, 49-86.

Kahneman, D., Ritov, I. and Schkade, D. A. 1999. Economic Preferences or Attitude Expressions? An Analysis of Dollar Responses to Public Issues. Journal of Risk and Uncertainty, 19: 203-235.

Sequence: Death Spirals and the Cult Attractor

Evaluability (And Cheap Holiday Shopping)

The Halo Effect

Referenced by: Fake Fake Utility Functions

Original with comments: Unbounded Scales, Huge Jury Awards, & Futurism

The Halo Effect

Eliezer Yudkowsky, 30 November 2007 12:58AM

The affect heuristic is how an overall feeling of goodness or badness contributes to many other judgments, whether it's logical or not, whether you're aware of it or not. Subjects told about the benefits of nuclear power are likely to rate it as having fewer risks; stock analysts rating unfamiliar stocks judge them as generally good or generally bad - low risk and high returns, or high risk and low returns - in defiance of ordinary economic theory, which says that risk and return should correlate positively.

The halo effect is the manifestation of the affect heuristic in social psychology. Robert Cialdini, in Influence: Science and Practice, summarizes:

Research has shown that we automatically assign to good-looking individuals such favorable traits as talent, kindness, honesty, and intelligence (for a review of this evidence, see Eagly, Ashmore, Makhijani, & Longo, 1991). Furthermore, we make these judgments without being aware that physical attractiveness plays a role in the process. Some consequences of this unconscious assumption that "good-looking equals good" scare me. For example, a study of the 1974 Canadian federal elections found that attractive candidates received more than two and a half times as many votes as unattractive candidates (Efran & Patterson, 1976). Despite such evidence of favoritism toward handsome politicians, follow-up research demonstrated that voters did not realize their bias. In fact, 73 percent of Canadian voters surveyed denied in the strongest possible terms that their votes had been influenced by physical appearance; only 14 percent even allowed for the possibility of such influence (Efran & Patterson, 1976). Voters can deny the impact of attractiveness on electability all they want, but evidence has continued to confirm its troubling presence (Budesheim & DePaola, 1994).

A similar effect has been found in hiring situations. In one study, good grooming of applicants in a simulated employment interview accounted for more favorable hiring decisions than did job qualifications - this, even though the interviewers claimed that appearance played a small role in their choices (Mack & Rainey, 1990). The advantage given to attractive workers extends past hiring day to payday. Economists examining U.S. and Canadian samples have found that attractive individuals get paid an average of 12-14 percent more than their unattractive coworkers (Hammermesh & Biddle, 1994).

Equally unsettling research indicates that our judicial process is similarly susceptible to the influences of body dimensions and bone structure. It now appears that good-looking people are likely to receive highly favorable treatment in the legal system (see Castellow, Wuensch, & Moore, 1991; and Downs & Lyons, 1990, for reviews). For example, in a Pennsylvania study (Stewart, 1980), researchers rated the physical attractiveness of 74 separate male defendants at the start of their criminal trials. When, much later, the researchers checked court records for the results of these cases, they found that the handsome men had received significantly lighter sentences. In fact, attractive defendants were twice as likely to avoid jail as unattractive defendants. In another study - this one on the damages awarded in a staged negligence trial - a defendant who was better looking than his victim was assessed an average amount of $5,623; but when the victim was the more attractive of the two, the average compensation was $10,051. What's more, both male and female jurors exhibited the attractiveness-based favoritism (Kulka & Kessler, 1978).

Other experiments have demonstrated that attractive people are more likely to obtain help when in need (Benson, Karabenic, & Lerner, 1976) and are more persuasive in changing the opinions of an audience (Chaiken, 1979)...

The influence of attractiveness on ratings of intelligence, honesty, or kindness is a clear example of bias - especially when you judge these other qualities based on fixed text - because we wouldn't expect judgments of honesty and attractiveness to conflate for any legitimate reason. On the other hand, how much of my perceived intelligence is due to my honesty? How much of my perceived honesty is due to my intelligence? Finding the truth, and saying the truth, are not as widely separated in nature as looking pretty and looking smart...

But these studies on the halo effect of attractiveness, should make us suspicious that there may be a similar halo effect for kindness, or intelligence. Let's say that you know someone who not only seems very intelligent, but also honest, altruistic, kindly, and serene. You should be suspicious that some of these perceived characteristics are influencing your perception of the others. Maybe the person is genuinely intelligent, honest, and altruistic, but not all that kindly or serene. You should be suspicious if the people you know seem to separate too cleanly into devils and angels.

And - I know you don't think you have to do it, but maybe you should - be just a little more skeptical of the more attractive political candidates.

Cialdini, R. B. 2001. Influence: Science and Practice. Boston, MA: Allyn and Bacon.

Cialdini's references:

Benson, P. L., Karabenic, S. A., & Lerner, R. M. (1976). Pretty pleases: The effects of physical attractiveness, race, and sex on receiving help. Journal of Experimental Social Psychology, 12, 409-415.

Budesheim, T. L., & DePaola, S. J. (1994). Beauty or the beast? The effects of appearance, personality, and issue information on evaluations of political candidates. Personality and Social Psychology Bulletin, 20, 339-348.

Castellow, W. A., Wuensch, K. L., & Moore, C. H. (1990). Effects of physical attractiveness of the plaintiff and defendant in sexual harassment judgments. Journal of Social Behavior and Personality, 5, 547-562.

Chaiken, S. ((1979). Communicator physical attractiveness and persuasion. Journal of Personality and Social Psychology, 5, 547-562.

Downs, A. C., & Lyons, P. M. (1990). Natural observations of the links between attractiveness and initial legal judgments. Personality and Social Psychology Bulletin, 17, 541-547.

Eagly, A. H., Ashmore, R. D., Makhijani, M. G., & Longo, L. C. (1991). What is beautiful is good, but...: A meta-analytic review of research on the physical attractiveness stereotype. Psychological Bulletin, 110, 109-128.

Efran, M. G., & Patterson, E. W. J. (1976). The politics of appearance. Unpublished manuscript, University of Toronto.

Hammermesh, D., & Biddle, J. E. (1994). Beauty and the labor market. The American Economic Review, 84, 1174-1194.

Kulka, R. A., & Kessler, J. R. (1978). Is justice really blind? The effect of litigant physical attractiveness on judicial judgment. Journal of Applied Social Psychology, 4, 336-381.

Mack, D., & Rainey, D. (1990). Female applicants' grooming and personnel selection. Journal of Social Behavior and Personality, 5, 399-407.

Stewart, J. E. II. (1980). Defendant's attractiveness as a factor in the outcome of trials. Journal of Applied Social Psychology, 10, 348-361.

Sequence: Death Spirals and the Cult Attractor

Unbounded Scales, Huge Jury Awards, & Futurism

Superhero Bias

Referenced by: Superhero Bias • Mere Messiahs • Affective Death Spirals • Resist the Happy Death Spiral • Uncritical Supercriticality • Fake Fake Utility Functions • Every Cause Wants To Be A Cult • Reversed Stupidity Is Not Intelligence • Cultish Countercultishness • Heading Toward Morality • Failure By Affective Analogy • Why Our Kind Can't Cooperate • Special Status Needs Special Support

Original with comments: The Halo Effect

Superhero Bias

Eliezer Yudkowsky, 01 December 2007 03:14AM

Followup to: The Halo Effect

Suppose there's a heavily armed sociopath, a kidnapper with hostages, who has just rejected all requests for negotiation and announced his intent to start killing. In real life, the good guys don't usually kick down the door when the bad guy has hostages. But sometimes - very rarely, but sometimes - life imitates Hollywood to the extent of genuine good guys needing to smash through a door.

Imagine, in two widely separated realities, two heroes who charge into the room, first to confront the villain.

In one reality, the hero is strong enough to throw cars, can fire power blasts out of his nostrils, has X-ray hearing, and his skin doesn't just deflect bullets but annihilates them on contact. The villain has ensconced himself in an elementary school and taken over two hundred children hostage; their parents are waiting outside, weeping.

In another reality, the hero is a New York police officer, and the hostages are three prostitutes the villain collected off the street.

Consider this question very carefully: Who is the greater hero? And who is more likely to get their own comic book?

The halo effect is that perceptions of all positive traits are correlated. Profiles rated higher on scales of attractiveness, are also rated higher on scales of talent, kindness, honesty, and intelligence.

And so comic-book characters who seem strong and invulnerable, both positive traits, also seem to possess more of the heroic traits of courage and heroism. And yet:

"How tough can it be to act all brave and courageous when you're pretty much invulnerable?"
-- Empowered, Vol. 1

I can't remember if I read the following point somewhere, or hypothesized it myself: Fame, in particular, seems to combine additively with all other personality characteristics. Consider Gandhi. Was Gandhi the most altruistic person of the 20th century, or just the most famous altruist? Gandhi faced police with riot sticks and soldiers with guns. But Gandhi was a celebrity, and he was protected by his celebrity. What about the others in the march, the people who faced riot sticks and guns even though there wouldn't be international headlines if they were put in the hospital or gunned down?

What did Gandhi think of getting the headlines, the celebrity, the fame, the place in history, becoming the archetype for non-violent resistance, when he took less risk than any of the people marching with him? How did he feel when one of those anonymous heroes came up to him, eyes shining, and told Gandhi how wonderful he was? Did Gandhi ever visualize his world in those terms? I don't know; I'm not Gandhi.

This is not in any sense a criticism of Gandhi. The point of non-violent resistance is not to show off your courage. That can be done much more easily by going over Niagara Falls in a barrel. Gandhi couldn't help being somewhat-but-not-entirely protected by his celebrity. And Gandhi's actions did take courage - not as much courage as marching anonymously, but still a great deal of courage.

The bias I wish to point out is that Gandhi's fame score seems to get perceptually added to his justly accumulated altruism score. When you think about nonviolence, you think of Gandhi - not an anonymous protestor in one of Gandhi's marches who faced down riot clubs and guns, and got beaten, and had to be taken to the hospital, and walked with a limp for the rest of her life, and no one ever remembered her name.

Similarly, which is greater - to risk your life to save two hundred children, or to risk your life to save three adults?

The answer depends on what one means by greater. If you ever have to choose between saving three adults and saving two hundred children, then choose the latter. "Whoever saves a single life, it is as if he had saved the whole world" may be a fine applause light, but it's terrible moral advice if you've got to pick one or the other. So if you mean "greater" in the sense of "Which is more important?" or "Which is the preferred outcome?" or "Which should I choose if I have to do one or the other?" then it is greater to save two hundred than three.

But if you ask about greatness in the sense of revealed virtue, then someone who would risk their life to save only three lives, reveals more courage than someone who would risk their life to save two hundred but not three.

This doesn't mean that you can deliberately choose to risk your life to save three adults, and let the two hundred schoolchildren go hang, because you want to reveal more virtue. Someone who risks their life because they want to be virtuous has revealed far less virtue than someone who risks their life because they want to save others. Someone who chooses to save three lives rather than two hundred lives, because they think it reveals greater virtue, is so selfishly fascinated with their own "greatness" as to have committed the moral equivalent of manslaughter.

It's one of those wu wei scenarios: You cannot reveal virtue by trying to reveal virtue. Given a choice between a safe method to save the world which involves no personal sacrifice or discomfort, and a method that risks your life and requires you to endure great privation, you cannot become a hero by deliberately choosing the second path. There is nothing heroic about wanting to be a hero. It would be a lost purpose.

Truly virtuous people who are genuinely trying to save lives, rather than trying to reveal virtue, will constantly seek to save more lives with less effort, which means that less of their virtue will be revealed. It may be confusing, but it's not contradictory.

But we cannot always choose to be invulnerable to bullets. After we've done our best to reduce risk and increase scope, any remaining heroism is well and truly revealed.

The police officer who puts their life on the line with no superpowers, no X-Ray vision, no super-strength, no ability to fly, and above all no invulnerability to bullets, reveals far greater virtue than Superman - who is only a mere superhero.

Sequence: Death Spirals and the Cult Attractor

The Halo Effect

Mere Messiahs

Referenced by: Mere Messiahs • Affective Death Spirals • Fake Fake Utility Functions • Einstein's Superpowers • The Ultimate Source • Heading Toward Morality • Is Morality Preference? • Shut up and do the impossible! • Ethics Notes • Higher Purpose

Original with comments: Superhero Bias

Mere Messiahs

Eliezer Yudkowsky, 02 December 2007 12:49AM

Followup to: Superhero Bias

Yesterday I discussed how the halo effect, which causes people to see all positive characteristics as correlated - for example, more attractive individuals are also perceived as more kindly, honest, and intelligent - causes us to admire heroes more if they're super-strong and immune to bullets. Even though, logically, it takes much more courage to be a hero if you're not immune to bullets. Furthermore, it reveals more virtue to act courageously to save one life than to save the world. (Although if you have to do one or the other, of course you should save the world.)

"The police officer who puts their life on the line with no superpowers", I said, "reveals far greater virtue than Superman, who is a mere superhero."

But let's be more specific.

John Perry was a New York City police officer who also happened to be an Extropian and transhumanist, which is how I come to know his name. John Perry was due to retire shortly and start his own law practice, when word came that a plane had slammed into the World Trade Center. He died when the north tower fell. I didn't know John Perry personally, so I cannot attest to this of direct knowledge; but very few Extropians believe in God, and I expect that Perry was likewise an atheist.

Which is to say that Perry knew he was risking his very existence, every week on the job. And it's not, like most people in history, that he knew he had only a choice of how to die, and chose to make it matter - because Perry was a transhumanist; he had genuine hope. And Perry went out there and put his life on the line anyway. Not because he expected any divine reward. Not because he expected to experience anything at all, if he died. But because there were other people in danger, and they didn't have immortal souls either, and his hope of life was worth no more than theirs.

I did not know John Perry. I do not know if he saw the world this way. But the fact that an atheist and a transhumanist can still be a police officer, can still run into the lobby of a burning building, says more about the human spirit than all the martyrs who ever hoped of heaven.

So that is one specific police officer...

...and now for the superhero.

As the Christians tell the story, Jesus Christ could walk on water, calm storms, drive out demons with a word. It must have made for a comfortable life: Starvation a problem? Xerox some bread. Don't like a tree? Curse it. Romans a problem? Sic your Dad on them. Eventually this charmed life ended, when Jesus voluntarily presented himself for crucifixion. Being nailed to a cross is not a comfortable way to die. But as the Christians tell the story, it only lasted a few hours - nothing compared to the duration, or even the intensity, of the tortures the Inquisition visited upon suspected witches. As the Christians tell the story, Jesus did this knowing he would come back to life three days later, and then go to Heaven. What was the threat that moved Jesus to face a few hours' suffering followed by eternity in Heaven? Was it the life of a single person? Was it the corruption of the church of Judea, or the oppression of Rome? No: as the Christians tell the story, the eternal fate of every human went on the line before Jesus suffered himself to be temporarily nailed to a cross.

But I do not wish to condemn a man who is not truly so guilty. What if Jesus - no, let's pronounce his name correctly: Yeishu - what if Yeishu of Nazareth never walked on water, and nonetheless defied the church of Judea established by the powers of Rome?

Would that not deserve greater honor than that which adheres to Jesus Christ, who was only a mere messiah?

Alas, somehow it seems greater for a hero to have steel skin and godlike powers. Somehow it seems to reveal more virtue to die temporarily to save the whole world, than to die permanently confronting a corrupt church. It seems so common, as if many other people through history had done the same.

Comfortably ensconced two thousand years in the future, we can levy all sorts of criticisms at Yeishu, but Yeishu did what he believed to be right, confronted a church he believed to be corrupt, and died for it. Without benefit of hindsight, he could hardly be expected to predict the true impact of his life upon the world. Relative to most other prophets of his day, he was probably relatively more honest, relatively less violent, and relatively more courageous. If you strip away the unintended consequences, the worst that can be said of Yeishu is that others in history did better. (Epicurus, Buddha, and Marcus Aurelius all come to mind.) Yeishu died forever, and - from one perspective - he did it for the sake of honesty. Fifteen hundred years before science, religious honesty was not an oxymoron.

As Sam Harris said:

"It is not enough that Jesus was a man who transformed himself to such a degree that the Sermon on the Mount could be his heart's confession. He also had to be the Son of God, born of a virgin, and destined to return to earth trailing clouds of glory. The effect of such dogma is to place the example of Jesus forever out of reach. His teaching ceases to become a set of empirical claims about the linkage between ethics and spiritual insight and instead becomes a gratuitous, and rather gruesome, fairy tale. According to the dogma of Christianity, becoming just like Jesus is impossible. One can only enumerate one's sins, believe the unbelievable, and await the end of the world."

I severely doubt that Yeishu ever spoke the Sermon on the Mount. Nonetheless, Yeishu deserves honor. He deserves more honor than the Christians would grant him.

But since Yeishu probably anticipated his soul would survive, he doesn't deserve more honor than John Perry.

Sequence: Death Spirals and the Cult Attractor

Superhero Bias

Affective Death Spirals

Referenced by: Fake Fake Utility Functions • The Moral Void

Original with comments: Mere Messiahs

Affective Death Spirals

Eliezer Yudkowsky, 02 December 2007 04:44PM

Followup to: The Affect Heuristic, The Halo Effect

Many, many, many are the flaws in human reasoning which lead us to overestimate how well our beloved theory explains the facts. The phlogiston theory of chemistry could explain just about anything, so long as it didn't have to predict it in advance. And the more phenomena you use your favored theory to explain, the truer your favored theory seems - has it not been confirmed by these many observations? As the theory seems truer, you will be more likely to question evidence that conflicts with it. As the favored theory seems more general, you will seek to use it in more explanations.

If you know anyone who believes that Belgium secretly controls the US banking system, or that they can use an invisible blue spirit force to detect available parking spaces, that's probably how they got started.

(Just keep an eye out, and you'll observe much that seems to confirm this theory...)

This positive feedback cycle of credulity and confirmation is indeed fearsome, and responsible for much error, both in science and in everyday life.

But it's nothing compared to the death spiral that begins with a charge of positive affect - a thought that feels really good.

A new political system that can save the world. A great leader, strong and noble and wise. An amazing tonic that can cure upset stomachs and cancer.

Heck, why not go for all three? A great cause needs a great leader. A great leader should be able to brew up a magical tonic or two.

The halo effect is that any perceived positive characteristic (such as attractiveness or strength) increases perception of any other positive characteristic (such as intelligence or courage). Even when it makes no sense, or less than no sense.

Positive characteristics enhance perception of every other positive characteristic? That sounds a lot like how a fissioning uranium atom sends out neutrons that fission other uranium atoms.

Weak positive affect is subcritical; it doesn't spiral out of control. An attractive person seems more honest, which, perhaps, makes them seem more attractive; but the effective neutron multiplication factor is less than 1. Metaphorically speaking. The resonance confuses things a little, but then dies out.

With intense positive affect attached to the Great Thingy, the resonance touches everywhere. A believing Communist sees the wisdom of Marx in every hamburger bought at McDonalds; in every promotion they're denied that would have gone to them in a true worker's paradise; in every election that doesn't go to their taste, in every newspaper article "slanted in the wrong direction". Every time they use the Great Idea to interpret another event, the Great Idea is confirmed all the more. It feels better - positive reinforcement - and of course, when something feels good, that, alas, makes us want to believe it all the more.

When the Great Thingy feels good enough to make you seek out new opportunities to feel even better about the Great Thingy, applying it to interpret new events every day, the resonance of positive affect is like a chamber full of mousetraps loaded with ping-pong balls.

You could call it a "happy attractor", "overly positive feedback", a "praise locked loop", or "funpaper". Personally I prefer the term "affective death spiral".

Coming tomorrow: How to resist an affective death spiral. (Hint: It's not by refusing to ever admire anything again, nor by keeping the things you admire in safe little restricted magisteria.)

Sequence: Death Spirals and the Cult Attractor

Mere Messiahs

Resist the Happy Death Spiral

Referenced by: Resist the Happy Death Spiral • Uncritical Supercriticality • Fake Fake Utility Functions • Fake Utility Functions • When None Dare Urge Restraint • Every Cause Wants To Be A Cult • Guardians of Ayn Rand • Politics and Awful Art • Cultish Countercultishness • Expecting Beauty • Science Doesn't Trust Your Rationality • Do Scientists Already Know This Stuff? • Einstein's Superpowers • A Premature Word on AI • Heading Toward Morality • The Moral Void • Changing Your Metaethics • My Childhood Death Spiral • A Prodigy of Refutation • Crisis of Faith • Failure By Affective Analogy • The Uses of Fun (Theory) • Seduced by Imagination • Getting Nearer • Raising the Sanity Waterline • Why Our Kind Can't Cooperate • Can Humanism Match Religion's Output? • That Crisis thing seems pretty useful • Bayesians vs. Barbarians • Go Forth and Create the Art!

Original with comments: Affective Death Spirals

Resist the Happy Death Spiral

Eliezer Yudkowsky, 04 December 2007 01:15AM

Followup to: Affective Death Spirals

Once upon a time, there was a man who was convinced that he possessed a Great Idea. Indeed, as the man thought upon the Great Idea more and more, he realized that it was not just a great idea, but the most wonderful idea ever. The Great Idea would unravel the mysteries of the universe, supersede the authority of the corrupt and error-ridden Establishment, confer nigh-magical powers upon its wielders, feed the hungry, heal the sick, make the whole world a better place, etc. etc. etc.

The man was Francis Bacon, his Great Idea was the scientific method, and he was the only crackpot in all history to claim that level of benefit to humanity and turn out to be completely right.

(Bacon didn't singlehandedly invent science, of course, but he did contribute, and may have been the first to realize the power.)

That's the problem with deciding that you'll never admire anything that much: Some ideas really are that good. Though no one has fulfilled claims more audacious than Bacon's; at least, not yet.

But then how can we resist the happy death spiral with respect to Science itself? The happy death spiral starts when you believe something is so wonderful that the halo effect leads you to find more and more nice things to say about it, making you see it as even more wonderful, and so on, spiraling up into the abyss. What if Science is in fact so beneficial that we cannot acknowledge its true glory and retain our sanity? Sounds like a nice thing to say, doesn't it? Oh no it's starting ruuunnnnn...

If you retrieve the standard cached deep wisdom for don't go overboard on admiring science, you will find thoughts like "Science gave us air conditioning, but it also made the hydrogen bomb" or "Science can tell us about stars and biology, but it can never prove or disprove the dragon in my garage." But the people who originated such thoughts were not trying to resist a happy death spiral. They weren't worrying about their own admiration of science spinning out of control. Probably they didn't like something science had to say about their pet beliefs, and sought ways to undermine its authority.

The standard negative things to say about science, aren't likely to appeal to someone who genuinely feels the exultation of science - that's not the intended audience. So we'll have to search for other negative things to say instead.

But if you look selectively for something negative to say about science - even in an attempt to resist a happy death spiral - do you not automatically convict yourself of rationalization? Why would you pay attention to your own thoughts, if you knew you were trying to manipulate yourself?

I am generally skeptical of people who claim that one bias can be used to counteract another. It sounds to me like an automobile mechanic who says that the motor is broken on your right windshield wiper, but instead of fixing it, they'll just break your left windshield wiper to balance things out. This is the sort of cleverness that leads to shooting yourself in the foot. Whatever the solution, it ought to involve believing true things, rather than believing you believe things that you believe are false.

Can you prevent the happy death spiral by restricting your admiration of Science to a narrow domain? Part of the happy death spiral is seeing the Great Idea everywhere - thinking about how Communism could cure cancer if it was only given a chance. Probably the single most reliable sign of a cult guru is that the guru claims expertise, not in one area, not even in a cluster of related areas, but in everything. The guru knows what cult members should eat, wear, do for a living; who they should have sex with; which art they should look at; which music they should listen to...

Unfortunately for this plan, most people fail miserably when they try to describe the neat little box that science has to stay inside. The usual trick, "Hey, science won't cure cancer" isn't going to fly. "Science has nothing to say about a parent's love for their child" - sorry, that's simply false. If you try to sever science from e.g. parental love, you aren't just denying cognitive science and evolutionary psychology. You're also denying Martine Rothblatt's founding of United Therapeutics to seek a cure for her daughter's pulmonary hypertension. (Successfully, I might add.) Science is legitimately related, one way or another, to just about every important facet of human existence.

All right, so what's an example of a false nice claim you could make about science?

In my humble opinion, one false claim is that science is so wonderful that scientists shouldn't even try to take ethical responsibility for their work, it will automatically end well. This claim, to me, seems to misunderstand the nature of the process whereby science benefits humanity. Scientists are human, they have prosocial concerns just like most other other people, and this is at least part of why science ends up doing more good than evil.

But that point is, evidently, not beyond dispute. So here's a simpler false nice claim: "A cancer patient can be cured just by publishing enough journal papers." Or, "Sociopaths could become fully normal, if they just committed themselves to never believing anything without replicated experimental evidence with p<0.05."

The way to avoid believing such statements isn't an affective cap, deciding that science is only slightly nice. Nor searching for reasons to believe that publishing journal papers causes cancer. Nor believing that science has nothing to say about cancer one way or the other.

Rather, if you know with enough specificity how science works, then you know that, while it may be possible for "science to cure cancer", a cancer patient writing journal papers isn't going to experience a miraculous remission. That specific proposed chain of cause and effect is not going to work out.

The happy death spiral is only an emotional problem because of a perceptual problem, the halo effect, which makes us more likely to accept future positive claims once we've accepted an initial positive claim. We can't get rid of this effect just by wishing; it will probably always influence us a little. But we can manage to slow down, stop, consider each additional nice claim as an additional burdensome detail, and focus on the specific points of the claim apart from its positiveness.

What if a specific nice claim "can't be disproven" but there are arguments "both for and against" it? Actually these are words to be wary of in general, because often this is what people say when they're rehearsing the evidence or avoiding the real weak points. Given the danger of the happy death spiral, it makes sense to try to avoid being happy about unsettled claims - to avoid making them into a source of yet more positive affect about something you liked already.

The happy death spiral is only a big emotional problem because of the overly positive feedback, the ability for the process to go critical. You may not be able to eliminate the halo effect entirely, but you can apply enough critical reasoning to keep the halos subcritical - make sure that the resonance dies out rather than exploding.

You might even say that the whole problem starts with people not bothering to critically examine every additional burdensome detail - demanding sufficient evidence to compensate for complexity, searching for flaws as well as support, invoking curiosity - once they've accepted some core premise. Without the conjunction fallacy, there might still be a halo effect, but there wouldn't be a happy death spiral.

Even on the nicest Nice Thingies in the known universe, a perfect rationalist who demanded exactly the necessary evidence for every additional (positive) claim, would experience no affective resonance. You can't do this, but you can stay close enough to rational to keep your happiness from spiraling out of control.

The really dangerous cases are the ones where any criticism of any positive claim about the Great Thingy feels bad or is socially unacceptable. Arguments are soldiers, any positive claim is a soldier on our side, stabbing your soldiers in the back is treason. Then the chain reaction goes supercritical. More on this tomorrow.

Addendum: Stuart Armstrong gives closely related advice:

Cut up your Great Thingy into smaller independent ideas, and treat them as independent.

For instance a marxist would cut up Marx's Great Thingy into a theory of value of labour, a theory of the political relations between classes, a theory of wages, a theory on the ultimate political state of mankind. Then each of them should be assessed independently, and the truth or falsity of one should not halo on the others. If we can do that, we should be safe from the spiral, as each theory is too narrow to start a spiral on its own.

This, metaphorically, is like keeping subcritical masses of plutonium from coming together. Three Great Ideas are far less likely to drive you mad than one Great Idea. Armstrong's advice also helps promote specificity: As soon as someone says, "Publishing enough papers can cure your cancer," you ask, "Is that a benefit of the experimental method, and if so, at which stage of the experimental process is the cancer cured? Or is it a benefit of science as a social process, and if so, does it rely on individual scientists wanting to cure cancer, or can they be self-interested?" Hopefully this leads you away from the good or bad feeling, and toward noticing the confusion and lack of support.

Addendum 2: To summarize, you do avoid a Happy Death Spiral by (1) splitting the Great Idea into parts (2) treating every additional detail as burdensome (3) thinking about the specifics of the causal chain instead of the good or bad feelings (4) not rehearsing evidence (5) not adding happiness from claims that "you can't prove are wrong"; but not by (6) refusing to admire anything too much (7) conducting a biased search for negative points until you feel unhappy again (8) forcibly shoving an idea into a safe box.

Sequence: Death Spirals and the Cult Attractor

Affective Death Spirals

Uncritical Supercriticality

Referenced by: Uncritical Supercriticality • Fake Fake Utility Functions • Every Cause Wants To Be A Cult • Guardians of Ayn Rand • Cultish Countercultishness • Heading Toward Morality • Crisis of Faith • The Thing That I Protect • That Crisis thing seems pretty useful

Original with comments: Resist the Happy Death Spiral

Uncritical Supercriticality

Eliezer Yudkowsky, 04 December 2007 04:40PM

Followup to: Resist the Happy Death Spiral

Every now and then, you see people arguing over whether atheism is a "religion". As I touched on in Purpose and Pragmatism, arguing over the meaning of a word nearly always means that you've lost track of the original question. How might this argument arise to begin with?

An atheist is holding forth, blaming "religion" for the Inquisition, the Crusades, and various conflicts with or within Islam. The religious one may reply, "But atheism is also a religion, because you also have beliefs about God; you believe God doesn't exist." Then the atheist answers, "If atheism is a religion, then not collecting stamps is a hobby," and the argument begins.

Or the one may reply, "But horrors just as great were inflicted by Stalin, who was an atheist, and who suppressed churches in the name of atheism; therefore you are wrong to blame the violence on religion." Now the atheist may be tempted to reply "No true Scotsman", saying, "Stalin's religion was Communism." The religious one answers "If Communism is a religion, then Star Wars fandom is a government," and the argument begins.

Should a "religious" person be defined as someone who has a definite opinion about the existence of at least one God, e.g., assigning a probability lower than 10% or higher than 90% to the existence of Zeus? Or should a "religious" person be defined as someone who has a positive opinion, say a probability higher than 90%, for the existence of at least one God? In the former case, Stalin was "religious"; in the latter case, Stalin was "not religious".

But this is exactly the wrong way to look at the problem. What you really want to know - what the argument was originally about - is why, at certain points in human history, large groups of people were slaughtered and tortured, ostensibly in the name of an idea. Redefining a word won't change the facts of history one way or the other.

Communism was a complex catastrophe, and there may be no single why, no single critical link in the chain of causality. But if I had to suggest an ur-mistake, it would be... well, I'll let God say it for me:

"If your brother, the son of your father or of your mother, or your son or daughter, or the spouse whom you embrace, or your most intimate friend, tries to secretly seduce you, saying, 'Let us go and serve other gods,' unknown to you or your ancestors before you, gods of the peoples surrounding you, whether near you or far away, anywhere throughout the world, you must not consent, you must not listen to him; you must show him no pity, you must not spare him or conceal his guilt. No, you must kill him, your hand must strike the first blow in putting him to death and the hands of the rest of the people following. You must stone him to death, since he has tried to divert you from Yahweh your God." (Deuteronomy 13:7-11, emphasis added)

This was likewise the rule which Stalin set for Communism, and Hitler for Nazism: if your brother tries to tell you why Marx is wrong, if your son tries to tell you the Jews are not planning world conquest, then do not debate him or set forth your own evidence; do not perform replicable experiments or examine history; but turn him in at once to the secret police.

Yesterday, I suggested that one key to resisting an affective death spiral is the principle of "burdensome details" - just remembering to question the specific details of each additional nice claim about the Great Idea. (It's not trivial advice. People often don't remember to do this when they're listening to a futurist sketching amazingly detailed projections about the wonders of tomorrow, let alone when they're thinking about their favorite idea ever.) This wouldn't get rid of the halo effect, but it would hopefully reduce the resonance to below criticality, so that one nice-sounding claim triggers less than 1.0 additional nice-sounding claims, on average.

The diametric opposite of this advice, which sends the halo effect supercritical, is when it feels wrong to argue against any positive claim about the Great Idea. Politics is the mind-killer. Arguments are soldiers. Once you know which side you're on, you must support all favorable claims, and argue against all unfavorable claims. Otherwise it's like giving aid and comfort to the enemy, or stabbing your friends in the back.

If...

- ...you feel that contradicting someone else who makes a flawed nice claim in favor of evolution, would be giving aid and comfort to the creationists;

- ...you feel like you get spiritual credit for each nice thing you say about God, and arguing about it would interfere with your relationship with God;

- ...you have the distinct sense that the other people in the room will dislike you for "not supporting our troops" if you argue against the latest war;

- ...saying anything against Communism gets you stoned to death shot;

...then the affective death spiral has gone supercritical. It is now a Super Happy Death Spiral.

It's not religion, as such, that is the key categorization, relative to our original question: "What makes the slaughter?" The best distinction I've heard between "supernatural" and "naturalistic" worldviews is that a supernatural worldview asserts the existence of ontologically basic mental substances, like spirits, while a naturalistic worldview reduces mental phenomena to nonmental parts. (Can't find original source thanks, g!) Focusing on this as the source of the problem buys into religious exceptionalism. Supernaturalist claims are worth distinguishing, because they always turn out to be wrong for fairly fundamental reasons. But it's still just one kind of mistake.

An affective death spiral can nucleate around supernatural beliefs; especially monotheisms whose pinnacle is a Super Happy Agent, defined primarily by agreeing with any nice statement about it; especially meme complexes grown sophisticated enough to assert supernatural punishments for disbelief. But the death spiral can also start around a political innovation, a charismatic leader, belief in racial destiny, or an economic hypothesis. The lesson of history is that affective death spirals are dangerous whether or not they happen to involve supernaturalism. Religion isn't special enough, as a class of mistake, to be the key problem.

Sam Harris came closer when he put the accusing finger on faith. If you don't place an appropriate burden of proof on each and every additional nice claim, the affective resonance gets started very easily. Look at the poor New Agers. Christianity developed defenses against criticism, arguing for the wonders of faith; New Agers culturally inherit the cached thought that faith is positive, but lack Christianity's exclusionary scripture to keep out competing memes. New Agers end up in happy death spirals around stars, trees, magnets, diets, spells, unicorns...

But the affective death spiral turns much deadlier after criticism becomes a sin, or a gaffe, or a crime. There are things in this world that are worth praising greatly, and you can't flatly say that praise beyond a certain point is forbidden. But there is never an Idea so true that it's wrong to criticize any argument that supports it. Never. Never ever never for ever. That is flat. The vast majority of possible beliefs in a nontrivial answer space are false, and likewise, the vast majority of possible supporting arguments for a true belief are also false, and not even the happiest idea can change that.

And it is triple ultra forbidden to respond to criticism with violence. There are a very few injunctions in the human art of rationality that have no ifs, ands, buts, or escape clauses. This is one of them. Bad argument gets counterargument. Does not get bullet. Never. Never ever never for ever.

Sequence: Death Spirals and the Cult Attractor

Resist the Happy Death Spiral

Evaporative Cooling of Group Beliefs

Referenced by: Fake Fake Utility Functions • Evaporative Cooling of Group Beliefs • When None Dare Urge Restraint • Politics and Awful Art • Cultish Countercultishness • Disguised Queries • The Argument from Common Usage • Is Humanism A Religion-Substitute? • Heading Toward Morality • ...And Say No More Of It • Why Our Kind Can't Cooperate

Original with comments: Uncritical Supercriticality

Fake Fake Utility Functions

Eliezer Yudkowsky, 06 December 2007 06:30AM

Followup to: Most of my posts over the last month...

Every now and then, you run across someone who has discovered the One Great Moral Principle, of which all other values are a mere derivative consequence.

I run across more of these people than you do. Only in my case, it's people who know the amazingly simple utility function that is all you need to program into an artificial superintelligence and then everything will turn out fine...

It's incredible how one little issue can require so much prerequisite material. My original schedule called for "Fake Utility Functions" to follow "Fake Justification" on Oct 31.

Talk about your planning fallacy. I've been planning to post on this topic in "just a few days" for the past month. A fun little demonstration of underestimated inferential distances.

You see, before I wrote this post, it occurred to me that if I wanted to properly explain the problem of fake utility functions, it would be helpful to illustrate a mistake about what a simple optimization criterion implied. The strongest real-world example I knew was the Tragedy of Group Selectionism. At first I thought I'd mention it in passing, within "Fake Utility Functions", but I decided the Tragedy of Group Selectionism was a long enough story that it needed its own blog post...

So I started to write "The Tragedy of Group Selectionism". A few hours later, I noticed that I hadn't said anything about group selectionism yet. I'd been too busy introducing basic evolutionary concepts. Select all the introductory stuff, cut, Compose New Post, paste, title... "An Alien God". Then keep writing until the "Alien God" post gets too long, and start taking separate subjects out into their own posts: "The Wonder of Evolution", "Evolutions Are Stupid", and at this point it became clear that, since I was planning to say a few words on evolution anyway, that was the time. Besides, a basic familiarity with evolution would help to shake people loose of their human assumptions when it came to visualizing nonhuman optimization processes.

So, finally I posted "The Tragedy of Group Selectionism". Now I was ready to write "Fake Utility Functions", right? The post that was supposed to come immediately afterward? So I thought, but each time I tried to write the post, I ended up recursing on a prerequisite post instead. Such as "Fake Selfishness", "Fake Morality", and "Fake Optimization Criteria".

When I got to "Fake Optimization Criteria", I really thought I could do "Fake Utility Functions" the next day. But then it occurred to me that I'd never explained why a simple utility function wouldn't be enough. We are a thousand shards of desire, as I said in "Thou Art Godshatter". Only that first required discussing "Evolutionary Psychology", which required explaining that human minds are "Adaptation-Executers, not Fitness-Maximizers", plus the difference between "Protein Reinforcement and DNA Consequentialism".

Furthermore, I'd never really explained the difference between "Terminal Values and Instrumental Values", without which I could hardly talk about utility functions.

Surely now I was ready? Yet I thought about conversations I'd had over the years, and how people seem to think a simple instruction like "Get my mother out of that burning building!" contains all the motivations that shape a human plan to rescue her, so I thought that first I'd do "The Hidden Complexity of Wishes". But, really, the hidden complexity of planning, and all the special cases needed to patch the genie's wish, was part of the general problem of recording outputs without absorbing the process that generates the outputs - as I explained in "Artificial Addition" and "Truly Part Of You". You don't want to keep the local goal description and discard the nonlocal utility function: "Leaky Generalizations" and "Lost Purposes".

Plus it occurred to me that evolution itself made an interesting genie, so before all that, came "Conjuring An Evolution To Serve You".

One kind of lost purpose is artificial pleasure, and "happiness" is one of the Fake Utility Functions I run into more often: "Not for the Sake of Happiness (Alone)". Similarly, it was worth taking the time to establish that fitness is not always your friend ("Evolving to Extinction") and that not everything in the universe is subject to significant selection pressures ("No Evolutions for Corporations or Nanodevices"), to avoid the Fake Utility Function of "genetic fitness".

Right after "Lost Purposes" seemed like a good time to point out the deep link between keeping track of your original goal and keeping track of your original question: "Purpose and Pragmatism".

Into the home stretch! No, wait, this would be a good time to discuss "Affective Death Spirals", since that's one of the main things that goes wrong when someone discovers The One True Valuable Thingy - they keep finding nicer and nicer things to say about it. Well, you can't discuss affective death spirals unless you first discuss "The Affect Heuristic", but I'd been meaning to do that for a while anyway. "Evaluability" illustrates the affect heuristic and leads to an important point about "Unbounded Scales and Futurism". The second key to affective death spirals is "The Halo Effect", which we can see illustrated in "Superhero Bias" and "Mere Messiahs". Then it's on to affective death spirals and how to "Resist the Happy Death Spiral" and "Uncritical Supercriticality".

A bonus irony is that "Fake Utility Functions" isn't a grand climax. It's just one of many Overcoming Bias posts relevant to my AI work, with plenty more scheduled. This particular post just turned out to require just a little more prerequisite material which - I thought on each occasion - I would have to write anyway, sooner or later.

And that's why blogging is difficult, and why it is necessary, at least for me. I would have been doomed, yea, utterly doomed, if I'd tried to write all this as one publication rather than as a series of blog posts. One month is nothing for this much material.

But now, it's done! Now, after only slightly more than an extra month of prerequisite material, I can do the blog post originally scheduled for November 1st!

Except...

Now that I think about it...

This post is pretty long already, right?

So I'll do the real "Fake Utility Functions" tomorrow.

Referenced by: Fake Utility Functions • GAZP vs. GLUT • Heading Toward Morality • The Psychological Unity of Humankind • Setting Up Metaethics • What I Think, If Not Why • Value is Fragile • ...And Say No More Of It

Original with comments: Fake Fake Utility Functions

Fake Utility Functions

Eliezer Yudkowsky, 06 December 2007 04:55PM

Continuation of: Fake Fake Utility Functions

Every now and then, you run across someone who has discovered the One Great Moral Principle, of which all other values are a mere derivative consequence.

I run across more of these people than you do. Only in my case, it's people who know the amazingly simple utility function that is all you need to program into an artificial superintelligence and then everything will turn out fine.

(This post should come as an anticlimax, since you already know virtually all the concepts involved, I bloody well hope. See yesterday's post, and all my posts since October 31st, actually...)

Some people, when they encounter the how-to-program-a-superintelligence problem, try to solve the problem immediately. Norman R. F. Maier: "Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any." Robyn Dawes: "I have often used this edict with groups I have led - particularly when they face a very tough problem, which is when group members are most apt to propose solutions immediately." Friendly AI is an extremely tough problem so people solve it extremely fast.

There's several major classes of fast wrong solutions I've observed; and one of these is the Incredibly Simple Utility Function That Is All A Superintelligence Needs For Everything To Work Out Just Fine.

I may have contributed to this problem with a really poor choice of phrasing, years ago when I first started talking about "Friendly AI". I referred to the optimization criterion of an optimization process - the region into which an agent tries to steer the future - as the "supergoal". I'd meant "super" in the sense of "parent", the source of a directed link in an acyclic graph. But it seems the effect of my phrasing was to send some people into happy death spirals as they tried to imagine the Superest Goal Ever, the Goal That Overrides All Over Goals, the Single Ultimate Rule From Which All Ethics Can Be Derived.

But a utility function doesn't have to be simple. It can contain an arbitrary number of terms. We have every reason to believe that insofar as humans can said to be have values, there are lots of them - high Kolmogorov complexity. A human brain implements a thousand shards of desire, though this fact may not be appreciated by one who has not studied evolutionary psychology. (Try to explain this without a full, long introduction, and the one hears "humans are trying to maximize fitness", which is exactly the opposite of what evolutionary psychology says.)

So far as descriptive theories of morality are concerned, the complicatedness of human morality is a known fact. It is a descriptive fact about human beings, that the love of a parent for a child, and the love of a child for a parent, and the love of a man for a woman, and the love of a woman for a man, have not been cognitively derived from each other or from any other value. A mother doesn't have to do complicated moral philosophy to love her daughter, nor extrapolate the consequences to some other desideratum. There are many such shards of desire, all different values.

Leave out just one of these values from a superintelligence, and even if you successfully include every other value, you could end up with a hyperexistential catastrophe, a fate worse than death. If there's a superintelligence that wants everything for us that we want for ourselves, except the human values relating to controlling your own life and achieving your own goals, that's one of the oldest dystopias in the book. (Jack Williamson's "With Folded Hands", in this case.)

So how does the one constructing the Amazingly Simple Utility Function deal with this objection?

Objection? Objection? Why would they be searching for possible objections to their lovely theory? (Note that the process of searching for real, fatal objections isn't the same as performing a dutiful search that amazingly hits on only questions to which they have a snappy answer.) They don't know any of this stuff. They aren't thinking about burdens of proof. They don't know the problem is difficult. They heard the word "supergoal" and went off in a happy death spiral around "complexity" or whatever.

Press them on some particular point, like the love a mother has for her children, and they reply "But if the superintelligence wants 'complexity', it will see how complicated the parent-child relationship is, and therefore encourage mothers to love their children." Goodness, where do I start?

Begin with the motivated stopping: A superintelligence actually searching for ways to maximize complexity wouldn't conveniently stop if it noticed that a parent-child relation was complex. It would ask if anything else was more complex. This is a fake justification; the one trying to argue the imaginary superintelligence into a policy selection, didn't really arrive at that policy proposal by carrying out a pure search for ways to maximize complexity.

The whole argument is a fake morality. If what you really valued was complexity, then you would be justifying the parental-love drive by pointing to how it increases complexity. If you justify a complexity drive by alleging that it increases parental love, it means that what you really value is the parental love. It's like giving a prosocial argument in favor of selfishness.

But if you consider the affective death spiral, then it doesn't increase the perceived niceness of "complexity" to say "A mother's relationship to her daughter is only important because it increases complexity; consider that if the relationship became simpler, we would not value it." What does increase the perceived niceness of "complexity" is saying, "If you set out to increase complexity, mothers will love their daughters - look at the positive consequence this has!"

This point applies whenever you run across a moralist who tries to convince you that their One Great Idea is all that anyone needs for moral judgment, and proves this by saying, "Look at all these positive consequences of this Great Thingy", rather than saying, "Look at how all these things we think of as 'positive' are only positive when their consequence is to increase the Great Thingy." The latter being what you'd actually need to carry such an argument.

But if you're trying to persuade others (or yourself) of your theory that the One Great Idea is "bananas", you'll sell a lot more bananas by arguing how bananas lead to better sex, rather than claiming that you should only want sex when it leads to bananas.

Unless you're so far gone into the Happy Death Spiral that you really do start saying "Sex is only good when it leads to bananas." Then you're in trouble. But at least you won't convince anyone else.

In the end, the only process that reliably regenerates all the local decisions you would make given your morality, is your morality. Anything else - any attempt to substitute instrumental means for terminal ends - ends up losing purpose and requiring an infinite number of patches because the system doesn't contain the source of the instructions you're giving it. You shouldn't expect to be able to compress a human morality down to a simple utility function, any more than you should expect to compress a large computer file down to 10 bits.

Addendum: Please note that we're not yet ready to discuss Friendly AI, as such, on Overcoming Bias. That will require a lot more prerequisite material. This post is only about why simple utility functions fail to compress our values.

Referenced by: Fake Fake Utility Functions • Against Discount Rates • Heading Toward Morality • No Universally Compelling Arguments • Setting Up Metaethics • Morality as Fixed Computation • Three Fallacies of Teleology • That Tiny Note of Discord • Ethical Injunctions

Original with comments: Fake Utility Functions

Evaporative Cooling of Group Beliefs

Eliezer Yudkowsky, 07 December 2007 11:08PM

Followup to: Uncritical Supercriticality

Early studiers of cults were surprised to discover than when cults receive a major shock - a prophecy fails to come true, a moral flaw of the founder is revealed - they often come back stronger than before, with increased belief and fanaticism. The Jehovah's Witnesses placed Armageddon in 1975, based on Biblical calculations; 1975 has come and passed. The Unarian cult, still going strong today, survived the nonappearance of an intergalactic spacefleet on September 27, 1975. (The Wikipedia article on Unarianism mentions a failed prophecy in 2001, but makes no mention of the earlier failure in 1975, interestingly enough.)

Why would a group belief become stronger after encountering crushing counterevidence?

The conventional interpretation of this phenomenon is based on cognitive dissonance. When people have taken "irrevocable" actions in the service of a belief - given away all their property in anticipation of the saucers landing - they cannot possibly admit they were mistaken. The challenge to their belief presents an immense cognitive dissonance; they must find reinforcing thoughts to counter the shock, and so become more fanatical. In this interpretation, the increased group fanaticism is the result of increased individual fanaticism.

I was looking at a Java applet which demonstrates the use of evaporative cooling to form a Bose-Einstein condensate, when it occurred to me that another force entirely might operate to increase fanaticism. Evaporative cooling sets up a potential energy barrier around a collection of hot atoms. Thermal energy is essentially statistical in nature - not all atoms are moving at the exact same speed. The kinetic energy of any given atom varies as the atoms collide with each other. If you set up a potential energy barrier that's just a little higher than the average thermal energy, the workings of chance will give an occasional atom a kinetic energy high enough to escape the trap. When an unusually fast atom escapes, it takes with an unusually large amount of kinetic energy, and the average energy decreases. The group becomes substantially cooler than the potential energy barrier around it. Playing with the Java applet may make this clearer.

In Festinger's classic "When Prophecy Fails", one of the cult members walked out the door immediately after the flying saucer failed to land. Who gets fed up and leaves first? An average cult member? Or a relatively more skeptical member, who previously might have been acting as a voice of moderation, a brake on the more fanatic members?

After the members with the highest kinetic energy escape, the remaining discussions will be between the extreme fanatics on one end and the slightly less extreme fanatics on the other end, with the group consensus somewhere in the "middle".

And what would be the analogy to collapsing to form a Bose-Einstein condensate? Well, there's no real need to stretch the analogy that far. But you may recall that I used a fission chain reaction analogy for the affective death spiral; when a group ejects all its voices of moderation, then all the people encouraging each other, and suppressing dissents, may internally increase in average fanaticism. (No thermodynamic analogy here, unless someone develops a nuclear weapon that explodes when it gets cold.)

When Ayn Rand's long-running affair with Nathaniel Branden was revealed to the Objectivist membership, a substantial fraction of the Objectivist membership broke off and followed Branden into espousing an "open system" of Objectivism not bound so tightly to Ayn Rand. Who stayed with Ayn Rand even after the scandal broke? The ones who really, really believed in her - and perhaps some of the undecideds, who, after the voices of moderation left, heard arguments from only one side. This may account for how the Ayn Rand Institute is (reportedly) more fanatic after the breakup, than the original core group of Objectivists under Branden and Rand.

A few years back, I was on a transhumanist mailing list where a small group espousing "social democratic transhumanism" vitriolically insulted every libertarian on the list. Most libertarians left the mailing list, most of the others gave up on posting. As a result, the remaining group shifted substantially to the left. Was this deliberate? Probably not, because I don't think the perpetrators knew that much psychology. (For that matter, I can't recall seeing the evaporative cooling analogy elsewhere, though that doesn't mean it hasn't been noted before.) At most, they might have thought to make themselves "bigger fish in a smaller pond".

This is one reason why it's important to be prejudiced in favor of tolerating dissent. Wait until substantially after it seems to you justified in ejecting a member from the group, before actually ejecting. If you get rid of the old outliers, the group position will shift, and someone else will become the oddball. If you eject them too, you're well on the way to becoming a Bose-Einstein condensate and, er, exploding.

The flip side: Thomas Kuhn believed that a science has to become a "paradigm", with a shared technical language that excludes outsiders, before it can get any real work done. In the formative stages of a science, according to Kuhn, the adherents go to great pains to make their work comprehensible to outside academics. But (according to Kuhn) a science can only make real progress as a technical discipline once it abandons the requirement of outside accessibility, and scientists working in the paradigm assume familiarity with large cores of technical material in their communications. This sounds cynical, relative to what is usually said about public understanding of science, but I can definitely see a core of truth here.

My own theory of Internet moderation is that you have to be willing to exclude trolls and spam to get a conversation going. You must even be willing to exclude kindly but technically uninformed folks from technical mailing lists if you want to get any work done. A genuinely open conversation on the Internet degenerates fast. It's the articulate trolls that you should be wary of ejecting, on this theory - they serve the hidden function of legitimizing less extreme disagreements. But you should not have so many articulate trolls that they begin arguing with each other, or begin to dominate conversations. If you have one person around who is the famous Guy Who Disagrees With Everything, anyone with a more reasonable, more moderate disagreement won't look like the sole nail sticking out. This theory of Internet moderation may not have served me too well in practice, so take it with a grain of salt.

Sequence: Death Spirals and the Cult Attractor

Uncritical Supercriticality

When None Dare Urge Restraint

Referenced by: Guardians of Ayn Rand • Cultish Countercultishness • Why Our Kind Can't Cooperate • Why I'm Staying On Bloggingheads.tv

Original with comments: Evaporative Cooling of Group Beliefs

When None Dare Urge Restraint

Eliezer Yudkowsky, 08 December 2007 11:09PM

Followup to: Uncritical Supercriticality

One morning, I got out of bed, turned on my computer, and my Netscape email client automatically downloaded that day's news pane. On that particular day, the news was that two hijacked planes had been flown into the World Trade Center.

These were my first three thoughts, in order:

I guess I really am living in the Future.
Thank goodness it wasn't nuclear.
and then
The overreaction to this will be ten times worse than the original event.

A mere factor of "ten times worse" turned out to be a vast understatement. Even I didn't guess how badly things would go. That's the challenge of pessimism; it's really hard to aim low enough that you're pleasantly surprised around as often and as much as you're unpleasantly surprised.

Nonetheless, I did realize immediately that everyone everywhere would be saying how awful, how terrible this event was; and that no one would dare to be the voice of restraint, of proportionate response. Initially, on 9/11, it was thought that six thousand people had died. Any politician who'd said "6000 deaths is 1/8 the annual US casualties from automobile accidents," would have been asked to resign the same hour.

No, 9/11 wasn't a good day. But if everyone gets brownie points for emphasizing how much it hurts, and no one dares urge restraint in how hard to hit back, then the reaction will be greater than the appropriate level, whatever the appropriate level may be.

This is the even darker mirror of the happy death spiral - the spiral of hate. Anyone who attacks the Enemy is a patriot; and whoever tries to dissect even a single negative claim about the Enemy is a traitor. But just as the vast majority of all complex statements are untrue, the vast majority of negative things you can say about anyone, even the worst person in the world, are untrue.

I think the best illustration was "the suicide hijackers were cowards". Some common sense, please? It takes a little courage to voluntarily fly your plane into a building. Of all their sins, cowardice was not on the list. But I guess anything bad you say about a terrorist, no matter how silly, must be true. Would I get even more brownie points if I accused al Qaeda of having assassinated John F. Kennedy? Maybe if I accused them of being Stalinists? Really, cowardice?

Yes, it matters that the 9/11 hijackers weren't cowards. Not just for understanding the enemy's realistic psychology. There is simply too much damage done by spirals of hate. It is just too dangerous for there to be any target in the world, whether it be the Jews or Adolf Hitler, about whom saying negative things trumps saying accurate things.

When the defense force contains thousands of aircraft and hundreds of thousands of heavily armed soldiers, one ought to consider that the immune system itself is capable of wreaking more damage than 19 guys and four nonmilitary airplanes. The US spent billions of dollars and thousands of soldiers' lives shooting off its own foot more effectively than any terrorist group could dream.

If the USA had completely ignored the 9/11 attack - just shrugged and rebuilt the building - it would have been better than the real course of history. But that wasn't a political option. Even if anyone privately guessed that the immune response would be more damaging than the disease, American politicians had no career-preserving choice but to walk straight into al Qaeda's trap. Whoever argues for a greater response is a patriot. Whoever dissects a patriotic claim is a traitor.

Initially, there were smarter responses to 9/11 than I had guessed. I saw a Congressperson - I forget who - say in front of the cameras, "We have forgotten that the first purpose of government is not the economy, it is not health care, it is defending the country from attack." That widened my eyes, that a politician could say something that wasn't an applause light. The emotional shock must have been very great for a Congressperson to say something that... real.

But within two days, the genuine shock faded, and concern-for-image regained total control of the political discourse. Then the spiral of escalation took over completely. Once restraint becomes unspeakable, no matter where the discourse starts out, the level of fury and folly can only rise with time.

Addendum: Welcome redditors! You may also enjoy A Fable of Science and Politics and Policy Debates Should Not Appear One-Sided.

Sequence: Death Spirals and the Cult Attractor

Evaporative Cooling of Group Beliefs

The Robbers Cave Experiment

Referenced by: Misc Meta • Every Cause Wants To Be A Cult

Original with comments: When None Dare Urge Restraint

The Robbers Cave Experiment

Eliezer Yudkowsky, 10 December 2007 06:18AM

Did you ever wonder, when you were a kid, whether your inane "summer camp" actually had some kind of elaborate hidden purpose - say, it was all a science experiment and the "camp counselors" were really researchers observing your behavior?

Me neither.

But we'd have been more paranoid if we'd read Intergroup Conflict and Cooperation: The Robbers Cave Experiment by Sherif, Harvey, White, Hood, and Sherif (1954/1961). In this study, the experimental subjects - excuse me, "campers" - were 22 boys between 5th and 6th grade, selected from 22 different schools in Oklahoma City, of stable middle-class Protestant families, doing well in school, median IQ 112. They were as well-adjusted and as similar to each other as the researchers could manage.

The experiment, conducted in the bewildered aftermath of World War II, was meant to investigate the causes - and possible remedies - of intergroup conflict. How would they spark an intergroup conflict to investigate? Well, the 22 boys were divided into two groups of 11 campers, and -

- and that turned out to be quite sufficient.

The researchers' original plans called for the experiment to be conducted in three stages. In Stage 1, each group of campers would settle in, unaware of the other group's existence. Toward the end of Stage 1, the groups would gradually be made aware of each other. In Stage 2, a set of contests and prize competitions would set the two groups at odds.

They needn't have bothered with Stage 2. There was hostility almost from the moment each group became aware of the other group's existence: They were using our campground, our baseball diamond. On their first meeting, the two groups began hurling insults. They named themselves the Rattlers and the Eagles (they hadn't needed names when they were the only group on the campground).

When the contests and prizes were announced, in accordance with pre-established experimental procedure, the intergroup rivalry rose to a fever pitch. Good sportsmanship in the contests was evident for the first two days but rapidly disintegrated.

The Eagles stole the Rattlers' flag and burned it. Rattlers raided the Eagles' cabin and stole the blue jeans of the group leader, which they painted orange and carried as a flag the next day, inscribed with the legend "The Last of the Eagles". The Eagles launched a retaliatory raid on the Rattlers, turning over beds, scattering dirt. Then they returned to their cabin where they entrenched and prepared weapons (socks filled with rocks) in case of a return raid. After the Eagles won the last contest planned for Stage 2, the Rattlers raided their cabin and stole the prizes. This developed into a fistfight that the staff had to shut down for fear of injury. The Eagles, retelling the tale among themselves, turned the whole affair into a magnificent victory - they'd chased the Rattlers "over halfway back to their cabin" (they hadn't).

Each group developed a negative stereotype of Them and a contrasting positive stereotype of Us. The Rattlers swore heavily. The Eagles, after winning one game, concluded that the Eagles had won because of their prayers and the Rattlers had lost because they used cuss-words all the time. The Eagles decided to stop using cuss-words themselves. They also concluded that since the Rattlers swore all the time, it would be wiser not to talk to them. The Eagles developed an image of themselves as proper-and-moral; the Rattlers developed an image of themselves as rough-and-tough.

Group members held their noses when members of the other group passed.

In Stage 3, the researchers tried to reduce friction between the two groups.

Mere contact (being present without contesting) did not reduce friction between the two groups. Attending pleasant events together - for example, shooting off Fourth of July fireworks - did not reduce friction; instead it developed into a food fight.

Would you care to guess what did work?

(Spoiler space...)

The boys were informed that there might be a water shortage in the whole camp, due to mysterious trouble with the water system - possibly due to vandals. (The Outside Enemy, one of the oldest tricks in the book.)

The area between the camp and the reservoir would have to be inspected by four search details. (Initially, these search details were composed uniformly of members from each group.) All details would meet up at the water tank if nothing was found. As nothing was found, the groups met at the water tank and observed for themselves that no water was coming from the faucet. The two groups of boys discussed where the problem might lie, pounded the sides of the water tank, discovered a ladder to the top, verified that the water tank was full, and finally found the sack stuffed in the water faucet. All the boys gathered around the faucet to clear it. Suggestions from members of both groups were thrown at the problem and boys from both sides tried to implement them.

When the faucet was finally cleared, the Rattlers, who had canteens, did not object to the Eagles taking a first turn at the faucets (the Eagles didn't have canteens with them). No insults were hurled, not even the customary "Ladies first".

It wasn't the end of the rivalry. There was another food fight, with insults, the next morning. But a few more common tasks, requiring cooperation from both groups - e.g. restarting a stalled truck - did the job. At the end of the trip, the Rattlers used $5 won in a bean-toss contest to buy malts for all the boys in both groups.

The Robbers Cave Experiment illustrates the psychology of hunter-gatherer bands, echoed through time, as perfectly as any experiment ever devised by social science.

Any resemblance to modern politics is just your imagination.

(Sometimes I think humanity's second-greatest need is a supervillain. Maybe I'll go into that line of work after I finish my current job.)

Sherif, M., Harvey, O. J., White, B. J., Hood, W. R., & Sherif, C. W. 1954/1961. Study of positive and negative intergroup attitudes between experimentally produced groups: Robbers Cave study. University of Oklahoma.

(This post is part of the sequence Politics is the Mind-Killer.)

Sequence: Politics is the Mind-Killer

Are Your Enemies Innately Evil?

Reversed Stupidity Is Not Intelligence

Sequence: Death Spirals and the Cult Attractor

When None Dare Urge Restraint

Every Cause Wants To Be A Cult

Referenced by: Every Cause Wants To Be A Cult • Cultish Countercultishness • The Two-Party Swindle • Of Gender and Rationality

Original with comments: The Robbers Cave Experiment

Misc Meta

Eliezer Yudkowsky, 10 December 2007 10:57PM

Overcoming Bias now has a new Welcome page, as I'm sure you've noticed on the sidebar. A completely ad-hoc eyeballing "statistical" test during our recent Redditing showed that a less prominent placement didn't increase pageviews per visit. Hopefully it won't get in the way too much.

Handy social bookmarking thingy is just below "Recent Posts".

The "Contributors" section now contains only individuals who have made 3 or more Overcoming Bias posts. For the curious, the following is the complete list of individuals who've made 10 or more contributions: Stuart Armstrong, David Balan, Nick Bostrom, Hal Finney, Robin Hanson, Andrew Gelman, James Miller, Eliezer Yudkowsky.

Many of us, including me, have been having trouble with a odd Typekey bug that shows us as logged in, but marks our contributions as having come from nowhere. If you "Sign out", manually enter your name and email address (and optionally URL), hit "Remember personal info", and then post, you shouldn't have this problem. At least it's worked for me, so far.

I've located what looks to be an acceptable restaurant for the Bay Area Overcoming Bias meetup, in Millbrae right next to the Millbrae BART/Caltrain station. This seems like a fairly central location and very well accessible by public transport. It's even centrally located for anyone who wants to quickly fly in to SFO. However, we're currently approaching the holiday crunch, so my thought is to schedule the first meetup for mid-January. Will post on this soon, I hope.

During November I generated 40,965 words of posts (not including comments). And here I was wondering why I've been feeling tired lately. Blooking feels like trying to run up a mountain, through concrete, at top speed - but it gets things said. 41,000 words/month, even if only a third of them end up being used, would be nearly in the range of a professional author if I could sustain it.

The recent post "When None Dare Urge Restraint" rose to #1 on Reddit, which raises interesting issues about how often that sort of thing should be allowed to happen on Overcoming Bias. Political posts are less interesting, and generate lower-quality discussion; they violate both Hanson's injunction to "Tug the rope sideways" and my own principle of "Learn rationality first on nondistracting problems." So we definitely don't want to do this too often.

However, I also recall an occasion where a Congressperson visited a transhumanist gathering, and asked "How many people here are signed up for cryonics?" Many hands went up. "How many of you know the name of your representative in the House?" Fewer hands went up. "And you wonder why you don't have much political influence." Point taken.

There is something to be said for being a little relevant every now and then. I didn't write "When None Dare Urge Restraint" with the intent that it would rise on Reddit, but I'm glad it did, and I'm currently considering whether to write another political post. It has obvious pros and obvious cons.

Original with comments: Misc Meta

Every Cause Wants To Be A Cult

Eliezer Yudkowsky, 12 December 2007 03:04AM

Followup to: Correspondence Bias, Affective Death Spirals, The Robbers Cave Experiment

Cade Metz at The Register recently alleged that a secret mailing list of Wikipedia's top administrators has become obsessed with banning all critics and possible critics of Wikipedia. Including banning a productive user when one administrator - solely because of the productivity - became convinced that the user was a spy sent by Wikipedia Review. And that the top people at Wikipedia closed ranks to defend their own. (I have not investigated these allegations myself, as yet. Hat tip to Eugen Leitl.)

Is there some deep moral flaw in seeking to systematize the world's knowledge, which would lead pursuers of that Cause into madness? Perhaps only people with innately totalitarian tendencies would try to become the world's authority on everything -

Correspondence bias alert! (Correspondence bias: making inferences about someone's unique disposition from behavior that can be entirely explained by the situation in which it occurs. When we see someone else kick a vending machine, we think they are "an angry person", but when we kick the vending machine, it's because the bus was late, the train was early and the machine ate our money.) If the allegations about Wikipedia are true, they're explained by ordinary human nature, not by extraordinary human nature.

The ingroup-outgroup dichotomy is part of ordinary human nature. So are happy death spirals and spirals of hate. A Noble Cause doesn't need a deep hidden flaw for its adherents to form a cultish in-group. It is sufficient that the adherents be human. Everything else follows naturally, decay by default, like food spoiling in a refrigerator after the electricity goes off.

In the same sense that every thermal differential wants to equalize itself, and every computer program wants to become a collection of ad-hoc patches, every Cause wants to be a cult. It's a high-entropy state into which the system trends, an attractor in human psychology. It may have nothing to do with whether the Cause is truly Noble. You might think that a Good Cause would rub off its goodness on every aspect of the people associated with it - that the Cause's followers would also be less susceptible to status games, ingroup-outgroup bias, affective spirals, leader-gods. But believing one true idea won't switch off the halo effect. A noble cause won't make its adherents something other than human. There are plenty of bad ideas that can do plenty of damage - but that's not necessarily what's going on.

Every group of people with an unusual goal - good, bad, or silly - will trend toward the cult attractor unless they make a constant effort to resist it. You can keep your house cooler than the outdoors, but you have to run the air conditioner constantly, and as soon as you turn off the electricity - give up the fight against entropy - things will go back to "normal".

On one notable occasion there was a group that went semicultish whose rallying cry was "Rationality! Reason! Objective reality!" (More on this in future posts.) Labeling the Great Idea "rationality" won't protect you any more than putting up a sign over your house that says "Cold!" You still have to run the air conditioner - expend the required energy per unit time to reverse the natural slide into cultishness. Worshipping rationality won't make you sane any more than worshipping gravity enables you to fly. You can't talk to thermodynamics and you can't pray to probability theory. You can use it, but not join it as an in-group.

Cultishness is quantitative, not qualitative. The question is not "Cultish, yes or no?" but "How much cultishness and where?" Even in Science, which is the archetypal Genuinely Truly Noble Cause, we can readily point to the current frontiers of the war against cult-entropy, where the current battle line creeps forward and back. Are journals more likely to accept articles with a well-known authorial byline, or from an unknown author from a well-known institution, compared to an unknown author from an unknown institution? How much belief is due to authority and how much is from the experiment? Which journals are using blinded reviewers, and how effective is blinded reviewing?

I cite this example, rather than the standard vague accusations of "Scientists aren't open to new ideas", because it shows a battle line - a place where human psychology is being actively driven back, where accumulated cult-entropy is being pumped out. (Of course this requires emitting some waste heat.)

This post is not a catalog of techniques for actively pumping against cultishness. Some such techniques I have said before, and some I will say later. Today I just want to point out that the worthiness of the Cause does not mean you can spend any less effort in resisting the cult attractor. And that if you can point to current battle lines, it does not mean you confess your Noble Cause unworthy. You might think that if the question were "Cultish, yes or no?" that you were obliged to answer "No", or else betray your beloved Cause. But that is like thinking that you should divide engines into "perfectly efficient" and "inefficient", instead of measuring waste.

Contrariwise, if you believe that it was the Inherent Impurity of those Foolish Other Causes that made them go wrong, if you laugh at the folly of "cult victims", if you think that cults are led and populated by mutants, then you will not expend the necessary effort to pump against entropy - to resist being human.

Sequence: Death Spirals and the Cult Attractor

The Robbers Cave Experiment

Guardians of the Truth

Referenced by: Reversed Stupidity Is Not Intelligence • Guardians of the Truth • Guardians of Ayn Rand • Two Cult Koans • Cultish Countercultishness • Something to Protect • Causality and Moral Responsibility • Changing Your Metaethics • The Thing That I Protect • ...And Say No More Of It • You're Calling *Who* A Cult Leader?

Original with comments: Every Cause Wants To Be A Cult

Reversed Stupidity Is Not Intelligence

Eliezer Yudkowsky, 12 December 2007 10:14PM

"...then our people on that time-line went to work with corrective action. Here."
He wiped the screen and then began punching combinations. Page after page appeared, bearing accounts of people who had claimed to have seen the mysterious disks, and each report was more fantastic than the last.
"The standard smother-out technique," Verkan Vall grinned. "I only heard a little talk about the 'flying saucers,' and all of that was in joke. In that order of culture, you can always discredit one true story by setting up ten others, palpably false, parallel to it."
-- H. Beam Piper, Police Operation

Piper had a point. Pers'nally, I don't believe there are any poorly hidden aliens infesting these parts. But my disbelief has nothing to do with the awful embarrassing irrationality of flying saucer cults - at least, I hope not.

You and I believe that flying saucer cults arose in the total absence of any flying saucers. Cults can arise around almost any idea, thanks to human silliness. This silliness operates orthogonally to alien intervention: We would expect to see flying saucer cults whether or not there were flying saucers. Even if there were poorly hidden aliens, it would not be any less likely for flying saucer cults to arise. p(cults|aliens) isn't less than p(cults|~aliens), unless you suppose that poorly hidden aliens would deliberately suppress flying saucer cults. By the Bayesian definition of evidence, the observation "flying saucer cults exist" is not evidence against the existence of flying saucers. It's not much evidence one way or the other.

This is an application of the general principle that, as Robert Pirsig puts it, "The world's greatest fool may say the Sun is shining, but that doesn't make it dark out."

If you knew someone who was wrong 99.99% of the time on yes-or-no questions, you could obtain 99.99% accuracy just by reversing their answers. They would need to do all the work of obtaining good evidence entangled with reality, and processing that evidence coherently, just to anticorrelate that reliably. They would have to be superintelligent to be that stupid.

A car with a broken engine cannot drive backward at 200 mph, even if the engine is really really broken.

If stupidity does not reliably anticorrelate with truth, how much less should human evil anticorrelate with truth? The converse of the halo effect is the horns effect: All perceived negative qualities correlate. If Stalin is evil, then everything he says should be false. You wouldn't want to agree with Stalin, would you?

Stalin also believed that 2 + 2 = 4. Yet if you defend any statement made by Stalin, even "2 + 2 = 4", people will see only that you are "agreeing with Stalin"; you must be on his side.

Corollaries of this principle:

- To argue against an idea honestly, you should argue against the best arguments of the strongest advocates. Arguing against weaker advocates proves nothing, because even the strongest idea will attract weak advocates. If you want to argue against transhumanism or the intelligence explosion, you have to directly challenge the arguments of Nick Bostrom or Eliezer Yudkowsky post-2003. The least convenient path is the only valid one.

- Exhibiting sad, pathetic lunatics, driven to madness by their apprehension of an Idea, is no evidence against that Idea. Many New Agers have been made crazier by their personal apprehension of quantum mechanics.

- Someone once said, "Not all conservatives are stupid, but most stupid people are conservatives." If you cannot place yourself in a state of mind where this statement, true or false, seems completely irrelevant as a critique of conservatism, you are not ready to think rationally about politics.

- Ad hominem argument is not valid.

- You need to be able to argue against genocide without saying "Hitler wanted to exterminate the Jews." If Hitler hadn't advocated genocide, would it thereby become okay?

- In Hansonian terms: Your instinctive willingness to believe something will change along with your willingness to affiliate with people who are known for believing it - quite apart from whether the belief is actually true. Some people may be reluctant to believe that God does not exist, not because there is evidence that God does exist, but rather because they are reluctant to affiliate with Richard Dawkins or those darned "strident" atheists who go around publicly saying "God does not exist".

- If your current computer stops working, you can't conclude that everything about the current system is wrong and that you need a new system without an AMD processor, an ATI video card, a Maxtor hard drive, or case fans - even though your current system has all these things and it doesn't work. Maybe you just need a new power cord.

- If a hundred inventors fail to build flying machines using metal and wood and canvas, it doesn't imply that what you really need is a flying machine of bone and flesh. If a thousand projects fail to build Artificial Intelligence using electricity-based computing, this doesn't mean that electricity is the source of the problem. Until you understand the problem, hopeful reversals are exceedingly unlikely to hit the solution.

(This post belongs to the sequence Politics is the Mind-Killer.)

Sequence: Politics is the Mind-Killer

The Robbers Cave Experiment

Argument Screens Off Authority

Referenced by: Argument Screens Off Authority • Hug the Query • Guardians of the Truth • Cultish Countercultishness • But There's Still A Chance, Right? • Absolute Authority • Dissolving the Question • Is Humanism A Religion-Substitute? • No Safe Defense, Not Even Science • Against Devil's Advocacy • Surface Analogies and Deep Causes • Where Recursive Justification Hits Bottom • The Genetic Fallacy • The Comedy of Behaviorism • My Best and Worst Mistake • Logical or Connectionist AI? • Failure By Affective Analogy • Prolegomena to a Theory of Fun • Amputation of Destiny • Serious Stories • Justified Expectation of Pleasant Surprises • Why Our Kind Can't Cooperate • Undiscriminating Skepticism

Original with comments: Reversed Stupidity Is Not Intelligence

Argument Screens Off Authority

Eliezer Yudkowsky, 14 December 2007 12:05AM

Black Belt Bayesian (aka "steven") tries to explain the asymmetry between good arguments and good authority, but it doesn't seem to be resolving the comments on Reversed Stupidity Is Not Intelligence, so let me take my own stab at it:

Scenario 1: Barry is a famous geologist. Charles is a fourteen-year-old juvenile delinquent with a long arrest record and occasional psychotic episodes. Barry flatly asserts to Arthur some counterintuitive statement about rocks, and Arthur judges it 90% probable. Then Charles makes an equally counterintuitive flat assertion about rocks, and Arthur judges it 10% probable. Clearly, Arthur is taking the speaker's authority into account in deciding whether to believe the speaker's assertions.

Scenario 2: David makes a counterintuitive statement about physics and gives Arthur a detailed explanation of the arguments, including references. Ernie makes an equally counterintuitive statement, but gives an unconvincing argument involving several leaps of faith. Both David and Ernie assert that this is the best explanation they can possibly give (to anyone, not just Arthur). Arthur assigns 90% probability to David's statement after hearing his explanation, but assigns a 10% probability to Ernie's statement.

It might seem like these two scenarios are roughly symmetrical: both involve taking into account useful evidence, whether strong versus weak authority, or strong versus weak argument.

But now suppose that Arthur asks Barry and Charles to make full technical cases, with references; and that Barry and Charles present equally good cases, and Arthur looks up the references and they check out. Then Arthur asks David and Ernie for their credentials, and it turns out that David and Ernie have roughly the same credentials - maybe they're both clowns, maybe they're both physicists.

Assuming that Arthur is knowledgeable enough to understand all the technical arguments - otherwise they're just impressive noises - it seems that Arthur should view David as having a great advantage in plausibility over Ernie, while Barry has at best a minor advantage over Charles.

Indeed, if the technical arguments are good enough, Barry's advantage over Charles may not be worth tracking. A good technical argument is one that eliminates reliance on the personal authority of the speaker.

Similarly, if we really believe Ernie that the argument he gave is the best argument he could give, which includes all of the inferential steps that Ernie executed, and all of the support that Ernie took into account - citing any authorities that Ernie may have listened to himself - then we can pretty much ignore any information about Ernie's credentials. Ernie can be a physicist or a clown, it shouldn't matter. (Again, this assumes we have enough technical ability to process the argument. Otherwise, Ernie is simply uttering mystical syllables, and whether we "believe" these syllables depends a great deal on his authority.)

So it seems there's an asymmetry between argument and authority. If we know authority we are still interested in hearing the arguments; but if we know the arguments fully, we have very little left to learn from authority.

Clearly (says the novice) authority and argument are fundamentally different kinds of evidence, a difference unaccountable in the boringly clean methods of Bayesian probability theory. For while the strength of the evidences - 90% versus 10% - is just the same in both cases, they do not behave similarly when combined. How, oh how, will we account for this?

Here's half a technical demonstration of how to represent this difference in probability theory. (The rest you can take on my personal authority, or look up in the references.)

If p(H|E1) = 90% and p(H|E2) = 9%, what is the probability p(H|E1,E2)? If learning E1 is true leads us to assign 90% probability to H, and learning E2 is true leads us to assign 9% probability to H, then what probability should we assign to H if we learn both E1 and E2? This is simply not something you can calculate in probability theory from the information given. No, the missing information is not the prior probability of H. E1 and E2 may not be independent of each other.

Suppose that H is "My sidewalk is slippery", E1 is "My sprinkler is running", and E2 is "It's night." The sidewalk is slippery starting from 1 minute after the sprinkler starts, until just after the sprinkler finishes, and the sprinkler runs for 10 minutes. So if we know the sprinkler is on, the probability is 90% that the sidewalk is slippery. The sprinkler is on during 10% of the nighttime, so if we know that it's night, the probability of the sidewalk being slippery is 9%. If we know that it's night and the sprinkler is on - that is, if we know both facts - the probability of the sidewalk being slippery is 90%.

We can represent this in a graphical model as follows:

Night -> Sprinkler -> Slippery

Whether or not it's Night causes the Sprinkler to be on or off, and whether the Sprinkler is on causes the Sidewalk to be slippery or unslippery.

The direction of the arrows is meaningful. If I wrote:

Night -> Sprinkler <- Slippery

This would mean that, if I didn't know anything about the Sprinkler, the probability of Nighttime and Slipperiness would be independent of each other. For example, suppose that I roll Die One and Die Two, and add up the showing numbers to get the Sum:

Die 1 -> Sum <- Die 2.

If you don't tell me the sum of the two numbers, and you tell me the first die showed 6, this doesn't tell me anything about the result of the second die, yet. But if you now also tell me the sum is 7, I know the second die showed 1.

Figuring out when various pieces of information are dependent or independent of each other, given various background knowledge, actually turns into a quite technical topic. The books to read are Judea Pearl's Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference and Causality. (If you only have time to read one book, read the first one.)

If you know how to read causal graphs, then you look at the dice-roll graph and immediately see:

p(die1,die2) = p(die1)*p(die2)
p(die1,die2|sum) ≠ p(die1|sum)*p(die2|sum)

If you look at the correct sidewalk diagram, you see facts like:

p(slippery|night) ≠ p(slippery)
p(slippery|sprinkler) ≠ p(slippery)
p(slippery|night, sprinkler) = p(slippery|sprinkler)

That is, the probability of the sidewalk being Slippery, given knowledge about the Sprinkler and the Night, is the same probability we would assign if we knew only about the Sprinkler. Knowledge of the Sprinkler has made knowledge of the Night irrelevant to inferences about Slipperiness.

This is known as screening off, and the criterion that lets us read such conditional independences off causal graphs is known as D-separation.

For the case of argument and authority, the causal diagram looks like this:

Truth -> Argument Goodness -> Expert Belief

If something is true, then it therefore tends to have arguments in favor of it, and the experts therefore observe these evidences and change their opinions. (In theory!)

If we see that an expert believes something, we infer back to the existence of evidence-in-the-abstract (even though we don't know what that evidence is exactly), and from the existence of this abstract evidence, we infer back to the truth of the proposition.

But if we know the value of the Argument node, this D-separates the node "Truth" from the node "Expert Belief" by blocking all paths between them, according to certain technical criteria for "path blocking" that seem pretty obvious in this case. So even without checking the exact probability distribution, we can read off from the graph that:

p(truth|argument,expert) = p(truth|argument)

This does not represent a contradiction of ordinary probability theory. It's just a more compact way of expressing certain probabilistic facts. You could read the same equalities and inequalities off an unadorned probability distribution - but it would be harder to see it by eyeballing. Authority and argument don't need two different kinds of probability, any more than sprinklers are made out of ontologically different stuff than sunlight.

In practice you can never completely eliminate reliance on authority. Good authorities are more likely to know about any counterevidence that exists and should be taken into account; a lesser authority is less likely to know this, which makes their arguments less reliable. This is not a factor you can eliminate merely by hearing the evidence they did take into account.

It's also very hard to reduce arguments to pure math; and otherwise, judging the strength of an inferential step may rely on intuitions you can't duplicate without the same thirty years of experience.

There is an ineradicable legitimacy to assigning slightly higher probability to what E. T. Jaynes tells you about Bayesian probability, than you assign to Eliezer Yudkowsky making the exact same statement. Fifty additional years of experience should not count for literally zero influence.

But this slight strength of authority is only ceteris paribus, and can easily be overwhelmed by stronger arguments. I have a minor erratum in one of Jaynes's books - because algebra trumps authority.

Sequence: Politics is the Mind-Killer

Reversed Stupidity Is Not Intelligence

Hug the Query

Referenced by: Hug the Query • Conditional Independence, and Naive Bayes • The Ultimate Source • Three Fallacies of Teleology • You're Entitled to Arguments, But Not (That Particular) Proof

Original with comments: Argument Screens Off Authority

Hug the Query

Eliezer Yudkowsky, 14 December 2007 07:51PM

Continuation of: Argument Screens Off Authority

In the art of rationality there is a discipline of closeness-to-the-issue - trying to observe evidence that is as near to the original question as possible, so that it screens off as many other arguments as possible.

The Wright Brothers say, "My plane will fly." If you look at their authority (bicycle mechanics who happen to be excellent amateur physicists) then you will compare their authority to, say, Lord Kelvin, and you will find that Lord Kelvin is the greater authority.

If you demand to see the Wright Brothers' calculations, and you can follow them, and you demand to see Lord Kelvin's calculations (he probably doesn't have any apart from his own incredulity), then authority becomes much less relevant.

If you actually watch the plane fly, the calculations themselves become moot for many purposes, and Kelvin's authority not even worth considering.

The more directly your arguments bear on a question, without intermediate inferences - the closer the observed nodes are to the queried node, in the Great Web of Causality - the more powerful the evidence. It's a theorem of these causal graphs that you can never get more information from distant nodes, than from strictly closer nodes that screen off the distant ones.

Jerry Cleaver said: "What does you in is not failure to apply some high-level, intricate, complicated technique. It's overlooking the basics. Not keeping your eye on the ball."

Just as it is superior to argue physics than credentials, it is also superior to argue physics than rationality. Who was more rational, the Wright Brothers or Lord Kelvin? If we can check their calculations, we don't have to care! The virtue of a rationalist cannot directly cause a plane to fly.

If you forget this principle, learning about more biases will hurt you, because it will distract you from more direct arguments. It's all too easy to argue that someone is exhibiting Bias #182 in your repertoire of fully generic accusations, but you can't settle a factual issue without closer evidence. If there are biased reasons to say the sun is shining, that doesn't make it dark out.

Just as you can't always experiment today, you can't always check the calculations today. Sometimes you don't know enough background material, sometimes there's private information, sometimes there just isn't time. There's a sadly large number of times when it's worthwhile to judge the speaker's rationality. You should always do it with a hollow feeling in your heart, though, a sense that something's missing.

Whenever you can, dance as near to the original question as possible - press yourself up against it - get close enough to hug the query!

Sequence: Politics is the Mind-Killer

Argument Screens Off Authority

Politics and Awful Art

Referenced by: Cultish Countercultishness • The Argument from Common Usage • Replace the Symbol with the Substance • Sneaking in Connotations

Original with comments: Hug the Query

Guardians of the Truth

Eliezer Yudkowsky, 15 December 2007 06:44PM

Followup to: Tsuyoku Naritai, Reversed Stupidity is not Intelligence

The criticism is sometimes leveled against rationalists: "The Inquisition thought they had the truth! Clearly this 'truth' business is dangerous."

There are many obvious responses, such as "If you think that possessing the truth would license you to torture and kill, you're making a mistake that has nothing to do with epistemology." Or, "So that historical statement you just made about the Inquisition - is it true?"

Reversed stupidity is not intelligence: "If your current computer stops working, you can't conclude that everything about the current system is wrong and that you need a new system without an AMD processor, an ATI video card... even though your current system has all these things and it doesn't work. Maybe you just need a new power cord." To arrive at a poor conclusion requires only one wrong step, not every step wrong. The Inquisitors believed that 2 + 2 = 4, but that wasn't the source of their madness. Maybe epistemological realism wasn't the problem either?

It does seem plausible that if the Inquisition had been made up of relativists, professing that nothing was true and nothing mattered, they would have mustered less enthusiasm for their torture. They would also have had been less enthusiastic if lobotomized. I think that's a fair analogy.

And yet... I think the Inquisition's attitude toward truth played a role. The Inquisition believed that there was such a thing as truth, and that it was important; well, likewise Richard Feynman. But the Inquisitors were not Truth-Seekers. They were Truth-Guardians.

I once read an argument (can't find source) that a key component of a zeitgeist is whether it locates its ideals in its future or its past. Nearly all cultures before the Enlightenment believed in a Fall from Grace - that things had once been perfect in the distant past, but then catastrophe had struck, and everything had slowly run downhill since then:

"In the age when life on Earth was full... They loved each other and did not know that this was 'love of neighbor'. They deceived no one yet they did not know that they were 'men to be trusted'. They were reliable and did not know that this was 'good faith'. They lived freely together giving and taking, and did not know that they were generous. For this reason their deeds have not been narrated. They made no history."
-- The Way of Chuang Tzu, trans. Thomas Merton

The perfect age of the past, according to our best anthropological evidence, never existed. But a culture that sees life running inexorably downward is very different from a culture in which you can reach unprecedented heights.

(I say "culture", and not "society", because you can have more than one subculture in a society.)

You could say that the difference between e.g. Richard Feynman and the Inquisition was that the Inquisition believed they had truth, while Richard Feynman sought truth. This isn't quite defensible, though, because there were undoubtedly some truths that Richard Feynman thought he had as well. "The sky is blue," for example, or "2 + 2 = 4".

Yes, there are effectively certain truths of science. General Relativity may be overturned by some future physics - albeit not in any way that predicts the Sun will orbit Jupiter; the new theory must steal the successful predictions of the old theory, not contradict them. But evolutionary theory takes place on a higher level of organization than atoms, and nothing we discover about quarks is going to throw out Darwinism, or the cell theory of biology, or the atomic theory of chemistry, or a hundred other brilliant innovations whose truth is now established beyond reasonable doubt.

Are these "absolute truths"? Not in the sense of possessing a probability of literally 1.0. But they are cases where science basically thinks it's got the truth.

And yet scientists don't torture people who question the atomic theory of chemistry. Why not? Because they don't believe that certainty licenses torture? Well, yes, that's the surface difference; but why don't scientists believe this?

Because chemistry asserts no supernatural penalty of eternal torture for disbelieving in the atomic theory of chemistry? But again we recurse and ask the question, "Why?" Why don't chemists believe that you go to hell if you disbelieve in the atomic theory?

Because journals won't publish your paper until you get a solid experimental observation of Hell? But all too many scientists can suppress their skeptical reflex at will. Why don't chemists have a private cult which argues that nonchemists go to hell, given that many are Christians anyway?

Questions like that don't have neat single-factor answers. But I would argue that one of the factors has to do with assuming a defensive posture toward the truth, versus a productive posture toward the truth.

When you are the Guardian of the Truth, you've got nothing useful to contribute to the Truth but your guardianship of it. When you're trying to win the Nobel Prize in chemistry by discovering the next benzene or buckyball, someone who challenges the atomic theory isn't so much a threat to your worldview as a waste of your time.

When you are a Guardian of the Truth, all you can do is try to stave off the inevitable slide into entropy by zapping anything that departs from the Truth. If there's some way to pump against entropy, generate new true beliefs along with a little waste heat, that same pump can keep the truth alive without secret police. In chemistry you can replicate experiments and see for yourself - and that keeps the precious truth alive without need of violence.

And it's not such a terrible threat if we make one mistake somewhere - end up believing a little untruth for a little while - because tomorrow we can recover the lost ground.

But this whole trick only works because the experimental method is a "criterion of goodness" which is not a mere "criterion of comparison". Because experiments can recover the truth without need of authority, they can also override authority and create new true beliefs where none existed before.

Where there are criteria of goodness that are not criteria of comparison, there can exist changes which are improvements, rather than threats. Where there are only criteria of comparison, where there's no way to move past authority, there's also no way to resolve a disagreement between authorities. Except extermination. The bigger guns win.

I don't mean to provide a grand overarching single-factor view of history. I do mean to point out a deep psychological difference between seeing your grand cause in life as protecting, guarding, preserving, versus discovering, creating, improving. Does the "up" direction of time point to the past or the future? It's a distinction that shades everything, casts tendrils everywhere.

This is why I've always insisted, for example, that if you're going to start talking about "AI ethics", you had better be talking about how you are going to improve on the current situation using AI, rather than just keeping various things from going wrong. Once you adopt criteria of mere comparison, you start losing track of your ideals - lose sight of wrong and right, and start seeing simply "different" and "same".

I would also argue that this basic psychological difference is one of the reasons why an academic field that stops making active progress tends to turn mean. (At least by the refined standards of science. Reputational assassination is tame by historical standards; most defensive-posture belief systems went for the real thing.) If major shakeups don't arrive often enough to regularly promote young scientists based on merit rather than conformity, the field stops resisting the standard degeneration into authority. When there's not many discoveries being made, there's nothing left to do all day but witch-hunt the heretics.

To get the best mental health benefits of the discover/create/improve posture, you've got to actually be making progress, not just hoping for it.

Sequence: Death Spirals and the Cult Attractor

Every Cause Wants To Be A Cult

Guardians of the Gene Pool

Sequence: Challenging the Difficult

Tsuyoku vs. the Egalitarian Instinct

Guardians of Ayn Rand

Referenced by: Guardians of the Gene Pool • Guardians of Ayn Rand • The Argument from Common Usage • Categorizing Has Consequences • Reductionism • My Childhood Role Model • This Failing Earth

Original with comments: Guardians of the Truth

Guardians of the Gene Pool

Eliezer Yudkowsky, 16 December 2007 08:08PM

Followup to: Guardians of the Truth

Like any educated denizen of the 21st century, you may have heard of World War II. You may remember that Hitler and the Nazis planned to carry forward a romanticized process of evolution, to breed a new master race, supermen, stronger and smarter than anything that had existed before.

Actually this is a common misconception. Hitler believed that the Aryan superman had previously existed - the Nordic stereotype, the blond blue-eyed beast of prey - but had been polluted by mingling with impure races. There had been a racial Fall from Grace.

It says something about the degree to which the concept of progress permeates Western civilization, that the one is told about Nazi eugenics and hears "They tried to breed a superhuman." You, dear reader - if you failed hard enough to endorse coercive eugenics, you would try to create a superhuman. Because you locate your ideals in your future, not in your past. Because you are creative. The thought of breeding back to some Nordic archetype from a thousand years earlier would not even occur to you as a possibility - what, just the Vikings? That's all? If you failed hard enough to kill, you would damn well try to reach heights never before reached, or what a waste it would all be, eh? Well, that's one reason you're not a Nazi, dear reader.

It says something about how difficult it is for the relatively healthy to envision themselves in the shoes of the relatively sick, that we are told of the Nazis, and distort the tale to make them defective transhumanists.

It's the Communists who were the defective transhumanists. "New Soviet Man" and all that. The Nazis were quite definitely the bioconservatives of the tale.

Sequence: Death Spirals and the Cult Attractor

Guardians of the Truth

Guardians of Ayn Rand

Original with comments: Guardians of the Gene Pool

Guardians of Ayn Rand

Eliezer Yudkowsky, 18 December 2007 06:24AM

Followup to: Every Cause Wants To Be A Cult, Guardians of the Truth

"For skeptics, the idea that reason can lead to a cult is absurd. The characteristics of a cult are 180 degrees out of phase with reason. But as I will demonstrate, not only can it happen, it has happened, and to a group that would have to be considered the unlikeliest cult in history. It is a lesson in what happens when the truth becomes more important than the search for truth..."
-- Michael Shermer, "The Unlikeliest Cult in History"

I think Michael Shermer is over-explaining Objectivism. I'll get around to amplifying on that.

Ayn Rand's novels glorify technology, capitalism, individual defiance of the System, limited government, private property, selfishness. Her ultimate fictional hero, John Galt, was <SPOILER>a scientist who invented a new form of cheap renewable energy; but then refuses to give it to the world since the profits will only be stolen to prop up corrupt governments.</SPOILER>

And then - somehow - it all turned into a moral and philosophical "closed system" with Ayn Rand at the center. The term "closed system" is not my own accusation; it's the term the Ayn Rand Institute uses to describe Objectivism. Objectivism is defined by the works of Ayn Rand. Now that Rand is dead, Objectivism is closed. If you disagree with Rand's works in any respect, you cannot be an Objectivist.

Max Gluckman once said: "A science is any discipline in which the fool of this generation can go beyond the point reached by the genius of the last generation." Science moves forward by slaying its heroes, as Newton fell to Einstein. Every young physicist dreams of being the new champion that future physicists will dream of dethroning.

Ayn Rand's philosophical idol was Aristotle. Now maybe Aristotle was a hot young math talent 2350 years ago, but math has made noticeable progress since his day. Bayesian probability theory is the quantitative logic of which Aristotle's qualitative logic is a special case; but there's no sign that Ayn Rand knew about Bayesian probability theory when she wrote her magnum opus, Atlas Shrugged. Rand wrote about "rationality", yet failed to familiarize herself with the modern research in heuristics and biases. How can anyone claim to be a master rationalist, yet know nothing of such elementary subjects?

"Wait a minute," objects the reader, "that's not quite fair! Atlas Shrugged was published in 1957! Practically nobody knew about Bayes back then." Bah. Next you'll tell me that Ayn Rand died in 1982, and had no chance to read Judgment Under Uncertainty: Heuristics and Biases, which was published that same year.

Science isn't fair. That's sorta the point. An aspiring rationalist in 2007 starts with a huge advantage over an aspiring rationalist in 1957. It's how we know that progress has occurred.

To me the thought of voluntarily embracing a system explicitly tied to the beliefs of one human being, who's dead, falls somewhere between the silly and the suicidal. A computer isn't five years old before it's obsolete.

The vibrance that Rand admired in science, in commerce, in every railroad that replaced a horse-and-buggy route, in every skyscraper built with new architecture - it all comes from the principle of surpassing the ancient masters. How can there be science, if the most knowledgeable scientist there will ever be, has already lived? Who would raise the New York skyline that Rand admired so, if the tallest building that would ever exist, had already been built?

And yet Ayn Rand acknowledged no superior, in the past, or in the future yet to come. Rand, who began in admiring reason and individuality, ended by ostracizing anyone who dared contradict her. Shermer: "[Barbara] Branden recalled an evening when a friend of Rand's remarked that he enjoyed the music of Richard Strauss. 'When he left at the end of the evening, Ayn said, in a reaction becoming increasingly typical, 'Now I understand why he and I can never be real soulmates. The distance in our sense of life is too great.' Often she did not wait until a friend had left to make such remarks."

Ayn Rand changed over time, one suspects.

Rand grew up in Russia, and witnessed the Bolshevik revolution firsthand. She was granted a visa to visit American relatives at the age of 21, and she never returned. It's easy to hate authoritarianism when you're the victim. It's easy to champion the freedom of the individual, when you are yourself the oppressed.

It takes a much stronger constitution to fear authority when you have the power. When people are looking to you for answers, it's harder to say "What the hell do I know about music? I'm a writer, not a composer," or "It's hard to see how liking a piece of music can be untrue."

When you're the one crushing those who dare offend you, the exercise of power somehow seems much more justifiable than when you're the one being crushed. All sorts of excellent justifications somehow leap to mind.

Michael Shermer goes into detail on how he thinks that Rand's philosophy ended up descending into cultishness. In particular, Shermer says (it seems) that Objectivism failed because Rand thought that certainty was possible, while science is never certain. I can't back Shermer on that one. The atomic theory of chemistry is pretty damned certain. But chemists haven't become a cult.

Actually, I think Shermer's falling prey to correspondence bias by supposing that there's any particular correlation between Rand's philosophy and the way her followers formed a cult. Every cause wants to be a cult.

Ayn Rand fled the Soviet Union, wrote a book about individualism that a lot of people liked, got plenty of compliments, and formed a coterie of admirers. Her admirers found nicer and nicer things to say about her (happy death spiral), and she enjoyed it too much to tell them to shut up. She found herself with the power to crush those of whom she disapproved, and she didn't resist the temptation of power.

Ayn Rand and Nathaniel Branden carried on a secret extramarital affair. (With permission from both their spouses, which counts for a lot in my view. If you want to turn that into a "problem", you have to specify that the spouses were unhappy - and then it's still not a matter for outsiders.) When Branden was revealed to have "cheated" on Rand with yet another woman, Rand flew into a fury and excommunicated him. Many Objectivists broke away when news of the affair became public.

Who stayed with Rand, rather than following Branden, or leaving Objectivism altogether? Her strongest supporters. Who departed? The previous voices of moderation. (Evaporative cooling of group beliefs.) Ever after, Rand's grip over her remaining coterie was absolute, and no questioning was allowed.

The only extraordinary thing about the whole business, is how ordinary it was.

You might think that a belief system which praised "reason" and "rationality" and "individualism" would have gained some kind of special immunity, somehow...?

Well, it didn't.

It worked around as well as putting a sign saying "Cold" on a refrigerator that wasn't plugged in.

The active effort required to resist the slide into entropy wasn't there, and decay inevitably followed.

And if you call that the "unlikeliest cult in history", you're just calling reality nasty names.

Let that be a lesson to all of us: Praising "rationality" counts for nothing. Even saying "You must justify your beliefs through Reason, not by agreeing with the Great Leader" just runs a little automatic program that takes whatever the Great Leader says and generates a justification that your fellow followers will view as Reason-able.

So where is the true art of rationality to be found? Studying up on the math of probability theory and decision theory. Absorbing the cognitive sciences like evolutionary psychology, or heuristics and biases. Reading history books...

"Study science, not just me!" is probably the most important piece of advice Ayn Rand should've given her followers and didn't. There's no one human being who ever lived, whose shoulders were broad enough to bear all the weight of a true science with many contributors.

It's noteworthy, I think, that Ayn Rand's fictional heroes were architects and engineers; John Galt, her ultimate, was a physicist; and yet Ayn Rand herself wasn't a great scientist. As far as I know, she wasn't particularly good at math. She could not aspire to rival her own heroes. Maybe that's why she began to lose track of Tsuyoku Naritai.

Now me, y'know, I admire Francis Bacon's audacity, but I retain my ability to bashfully confess, "If I could go back in time, and somehow make Francis Bacon understand the problem I'm currently working on, his eyeballs would pop out of their sockets like champagne corks and explode."

I admire Newton's accomplishments. But my attitude toward a woman's right to vote, bars me from accepting Newton as a moral paragon. Just as my knowledge of Bayesian probability bars me from viewing Newton as the ultimate unbeatable source of mathematical knowledge. And my knowledge of Special Relativity, paltry and little-used though it may be, bars me from viewing Newton as the ultimate authority on physics.

Newton couldn't realistically have discovered any of the ideas I'm lording over him - but progress isn't fair! That's the point!

Science has heroes, but no gods. The great Names are not our superiors, or even our rivals, they are passed milestones on our road; and the most important milestone is the hero yet to come.

To be one more milestone in humanity's road is the best that can be said of anyone; but this seemed too lowly to please Ayn Rand. And that is how she became a mere Ultimate Prophet.

Sequence: Death Spirals and the Cult Attractor

Guardians of the Gene Pool

The Litany Against Gurus

Sequence: Challenging the Difficult

Guardians of the Truth

Lotteries: A Waste of Hope

Referenced by: False Laughter • Cultish Countercultishness • The Comedy of Behaviorism • Raising the Sanity Waterline • A Sense That More Is Possible • Whining-Based Communities • Beware of Other-Optimizing

Original with comments: Guardians of Ayn Rand

The Litany Against Gurus

Eliezer Yudkowsky, 18 December 2007 08:11PM

I am your hero!
I am your master!
Learn my arts,
Seek my way.

Learn as I learned,
Seek as I sought.

Envy me!
Aim at me!
Rival me!
Transcend me!

Look back,
Smile,
And then -
Eyes front!

I was never your city,
Just a stretch of your road.

Sequence: Death Spirals and the Cult Attractor

Guardians of Ayn Rand

Two Cult Koans

Referenced by: Politics and Awful Art • Cultish Countercultishness • To Spread Science, Keep It Secret

Original with comments: The Litany Against Gurus

Politics and Awful Art

Eliezer Yudkowsky, 20 December 2007 03:46AM

Followup to: Rationality and the English Language

One of my less treasured memories is of a State of the Union address, or possibly a presidential inauguration, at which a Nobel Laureate got up and read, in a terribly solemn voice, some politically correct screed about what a wonderfully inclusive nation we all were - "The African-Americans, the Ethiopians, the Etruscans", or something like that. The "poem", if you can call it that, was absolutely awful. As far as my ears could tell, it had no redeeming artistic merit whatsoever.

Every now and then, yet another atheist is struck by the amazing idea that atheists should have hymns, just like religious people have hymns, and they take some existing religious song and turn out an atheistic version. And then this "atheistic hymn" is, almost without exception, absolutely awful. But the author can't see how dreadful the verse is as verse. They're too busy congratulating themselves on having said "Religion sure sucks, amen." Landing a punch on the Hated Enemy feels so good that they overlook the hymn's lack of any other merit. Verse of the same quality about something unpolitical, like mountain streams, would be seen as something a kindergartener's mother would post on her refrigerator.

In yesterday's Litany Against Gurus, there are only two lines that might be classifiable as "poetry", not just "verse". When I was composing the litany's end, the lines that first popped into my head were:

I was not your destination
Only a step on your path

Which didn't sound right at all. Substitute "pathway" for "road", so the syllable counts would match? But that sounded even worse. The prosody - the pattern of stressed syllables - was all wrong.

The real problem was the word des-ti-NA-tion - a huge awkward lump four syllables long. So get rid of it! "I was not your goal" was the first alternative that came to mind. Nicely short. But now that I was thinking about it, "goal" sounded very airy and abstract. Then the word "city" came into my mind - and it echoed.

"I was never your city" came to me, not by thinking about rationality, but by thinking about prosody. The constraints of art force us to toss out the first, old, tired phrasing that comes to mind; and in searching for a less obvious phrasing, often lead us to less obvious thoughts.

If I'd said, "Well, this is such a wonderful thought about rationality, that I don't have to worry about the prosodic problem", then I would have not received the benefit of being constrained.

The other poetic line began as "Laugh once, and never look back," which had problems as rationality, not just as prosody. "Laugh once" is the wrong kind of laughter; too derisive. "Never look back" is even less correct, because the memory of past mistakes can be useful years later. So... "Look back, laugh once smile, and then," um, "look forward"? Now if I'd been enthralled by the wonders of rationality, I would have said, "Ooh, 'look forward'! What a progressive sentiment!" and forgiven the extra syllable.

"Eyes front!" It was two syllables. It had the crisp click of a drill sergeant telling you to stop woolgathering, snap out of that daze, and get to work! Nothing like the soft cliche of "look forward, look upward, look to the future in a vaguely admiring sort of way..."

Eyes front! It's a better thought as rationality, which I would never have found, if I'd been so impressed with daring to write about rationality, that I had forgiven myself the prosodic transgression of an extra syllable.

If you allow affirmation of My-Favorite-Idea to compensate for lack of rhythm in a song, lack of beauty in a painting, lack of poignancy in fiction, then your art will, inevitably, suck. When you do art about My-Favorite-Idea, you have to hold yourself to the same standard as if you were doing art about a butterfly.

There is powerful politicized art, just as there are great religious paintings. But merit in politicized art is more the exception than the rule. Most of it ends up as New Soviet Man Heroically Crushing Capitalist Snakes. It's an easy living. If anyone criticizes your art on grounds of general suckiness, they'll be executed for siding with the capitalist snakes.

Tolerance of awful art, just because it lands a delicious punch on the Enemy, or just because it affirms the Great Truth, is a dangerous sign: It indicates an affective death spiral entering the supercritical phase where you can no longer criticize any argument whose conclusion is the "right" one.

And then the next thing you know, you're composing dreadful hymns, or inserting giant philosophical lectures into the climax of your fictional novel...

Sequence: Politics is the Mind-Killer

Hug the Query

False Laughter

Referenced by: False Laughter

Original with comments: Politics and Awful Art

Two Cult Koans

Eliezer Yudkowsky, 21 December 2007 05:45AM

Followup to: Every Cause Wants To Be A Cult

A novice rationalist studying under the master Ougi was rebuked by a friend who said, "You spend all this time listening to your master, and talking of 'rational' this and 'rational' that - you have fallen into a cult!"
The novice was deeply disturbed; he heard the words, "You have fallen into a cult!" resounding in his ears as he lay in bed that night, and even in his dreams.
The next day, the novice approached Ougi and related the events, and said, "Master, I am constantly consumed by worry that this is all really a cult, and that your teachings are only dogma."
Ougi replied, "If you find a hammer lying in the road and sell it, you may ask a low price or a high one. But if you keep the hammer and use it to drive nails, who can doubt its worth?"
The novice said, "See, now that's just the sort of thing I worry about - your mysterious Zen replies."
Ougi said, "Fine, then, I will speak more plainly, and lay out perfectly reasonable arguments which demonstrate that you have not fallen into a cult. But first you have to wear this silly hat."

Ougi gave the novice a huge brown ten-gallon cowboy hat.
"Er, master..." said the novice.
"When I have explained everything to you," said Ougi, "you will see why this was necessary. Or otherwise, you can continue to lie awake nights, wondering whether this is a cult."
The novice put on the cowboy hat.
Ougi said, "How long will you repeat my words and ignore the meaning? Disordered thoughts begin as feelings of attachment to preferred conclusions. You are too anxious about your self-image as a rationalist. You came to me to seek reassurance. If you had been truly curious, not knowing one way or the other, you would have thought of ways to resolve your doubts. Because you needed to resolve your cognitive dissonance, you were willing to put on a silly hat. If I had been an evil man, I could have made you pay a hundred silver coins. When you concentrate on a real-world question, the worth or worthlessness of your understanding will soon become apparent. You are like a swordsman who keeps glancing away to see if anyone might be laughing at him -"
"All right," said the novice.
"You asked for the long version," said Ougi.
This novice later succeeded Ougi and became known as Ni no Tachi. Ever after, he would not allow his students to cite his words in their debates, saying, "Use the techniques and do not mention them."

A novice rationalist approached the master Ougi and said, "Master, I worry that our rationality dojo is... well... a little cultish."
"That is a grave concern," said Ougi.
The novice waited a time, but Ougi said nothing more.
So the novice spoke up again: "I mean, I'm sorry, but having to wear these robes, and the hood - it just seems like we're the bloody Freemasons or something."
"Ah," said Ougi, "the robes and trappings."
"Well, yes the robes and trappings," said the novice. "It just seems terribly irrational."
"I will address all your concerns," said the master, "but first you must put on this silly hat." And Ougi drew out a wizard's hat, embroidered with crescents and stars.
The novice took the hat, looked at it, and then burst out in frustration: "How can this possibly help?"
"Since you are so concerned about the interactions of clothing with probability theory," Ougi said, "it should not surprise you that you must wear a special hat to understand."
When the novice attained the rank of grad student, he took the name Bouzo and would only discuss rationality while wearing a clown suit.

Sequence: Death Spirals and the Cult Attractor

The Litany Against Gurus

Asch's Conformity Experiment

Referenced by: Zen and the Art of Rationality • Cultish Countercultishness • Replace the Symbol with the Substance • To Spread Science, Keep It Secret

Original with comments: Two Cult Koans

False Laughter

Eliezer Yudkowsky, 22 December 2007 06:03AM

Followup to: Politics and Awful Art

There's this thing called "derisive laughter" or "mean-spirited laughter", which follows from seeing the Hated Enemy get a kick in the pants. It doesn't have to be an unexpected kick in the pants, or a kick followed up with a custard pie. It suffices that the Hated Enemy gets hurt. It's like humor, only without the humor.

If you know what your audience hates, it doesn't take much effort to get a laugh like that - which marks this as a subspecies of awful political art.

There are deliciously biting satires, yes; not all political art is bad art. But satire is a much more demanding art than just punching the Enemy in the nose. In fact, never mind satire - just an atom of ordinary genuine humor takes effort.

Imagine this political cartoon: A building labeled "science", and a standard Godzilla-ish monster labeled "Bush" stomping on the "science" building. Now there are people who will laugh at this - hur hur, scored a point off Bush, hur hur - but this political cartoon didn't take much effort to imagine. In fact, it was the very first example that popped into my mind when I thought "political cartoon about Bush and science". This degree of obviousness is a bad sign.

If I want to make a funny political cartoon, I have to put in some effort. Go beyond the cached thought. Use my creativity. Depict Bush as a tentacle monster and Science as a Japanese schoolgirl.

There are many art forms that suffer from obviousness. But humor more than most, because humor relies on surprise - the ridiculous, the unexpected, the absurd.

(Satire achieves surprise by saying, out loud, the thoughts you didn't dare think. Fake satires repeat thoughts you were already thinking.)

You might say that a predictable punchline is too high-entropy to be funny, by that same logic which says you should be enormously less surprised to find your thermostat reading 30 degrees than 29 degrees.

The general test against awful political art is to ask whether the art would seem worthwhile if it were not political. If someone writes a song about space travel, and the song is good enough that I would enjoy listening to it even if it were about butterflies, then and only then does it qualify to pick up bonus points for praising a Worthy Cause.

So one test for derisive laughter is to ask if the joke would still be funny, if it weren't the Hated Enemy getting the kick in the pants. Bill Gates once got hit by an unexpected pie in the face. Would it still have been funny (albeit less funny) if Linus Torvalds had gotten hit by the pie?

Of course I'm not suggesting that you sit around all day asking which jokes are "really" funny, or which jokes you're "allowed" to laugh at. As the saying goes, analyzing a joke is like dissecting a frog - it kills the frog and it's not much fun for you, either.

So why this blog post, then? Don't you and I already know which jokes are funny?

One application: If you find yourself in a group of people who tell consistently unfunny jokes about the Hated Enemy, it may be a good idea to head for the hills, before you start to laugh as well...

Another application: You and I should be allowed not to laugh at certain jokes - even jokes that target our own favorite causes - on the grounds that the joke is too predictable to be funny. We should be able to do this without being accused of being humorless, "unable to take a joke", or protecting sacred cows. If labeled-Godzilla-stomps-a-labeled-building isn't funny about "Bush" and "Science", then it also isn't funny about "libertarian economists" and "American national competitiveness", etc.

The most scathing accusation I ever heard against Objectivism is that hardcore Objectivists have no sense of humor; but no one could prove this by showing an Objectivist a cartoon of Godzilla-"Rand" stomping on building-"humor" and demanding that he laugh.

Requiring someone to laugh in order to prove their non-cultishness - well, like most kinds of obligatory laughter, it doesn't quite work. Laughter, of all things, has to come naturally. The most you can do is get fear and insecurity out of its way.

If an Objectivist, innocently browsing the Internet, came across a depiction of Ayn Rand as a Japanese schoolgirl lecturing a tentacle monster, and still didn't laugh, then that would be a problem. But they couldn't fix this problem by deliberately trying to laugh.

Obstacles to humor are a sign of dreadful things. But making humor obligatory, or constantly wondering whether you're laughing enough, just throws up another obstacle. In that way it's rather Zen. There are things you can accomplish by deliberately composing a joke, but very few things you can accomplish by deliberately believing a joke is funny.

Sequence: Politics is the Mind-Killer

Politics and Awful Art

Rationality and the English Language

Original with comments: False Laughter

Effortless Technique

Eliezer Yudkowsky, 23 December 2007 04:22AM

"All my life I have been intensely repelled by the idea of 'making an effort'. I hate this idea today as much as I did as a child. I don't know why I hate it so much; I just do."
-- Raymond Smullyan, The Tao Is Silent

In the Hollywood version of rationality - or even the Traditional rationality that was passed down from supervisor to grad student in ancient days before Bayesianism - rationality is a great strain, a great effort, a continuous battle to coerce your mind into a desired shape. Spock, the archetype of Hollywood's concept of rationality, represses all his emotions.

And this great effort, they conceive, is virtue unto a rationalist. The more effort you expend on forcing yourself into the mold, the better the rationalist you must be. It's like working extra hard at your job, as demanded by the Protestant work-ethic. If the one works long hours - sweating, getting ulcers - surely the one must be worthy of praise?

This, I think, is an instance of a Lost Purpose. People see that successful folk must sometimes make an effort, and so they conclude that effort of itself is virtuous whether or not it succeeds.

I am reminded of an AI koan from AGI '06, where the discussion turned (as it often does) to defining "intelligence". A medium-prominent AI researcher suggested that an agent's "intelligence" could be measured in the agent's processing cycles per second, bits of memory, and bits of sensory bandwidth. To which I replied with a quote from Dijkstra:

"If we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger."

Surely (I said), an agent is less intelligent if it uses more memory, processing power, and sensory bandwidth to accomplish the same task?

This reply was due, in no small part, to my having read Raymond Smullyan's The Tao Is Silent at the age of sixteen. Raymond Smullyan is a mathematical logician, a great composer of logic puzzles, and sometime Westernized Taoist. Though I disagree with much of The Tao Is Silent, I would count "just the parts of the book I liked" as one of my most important formative influences as a rationalist.

In particular, it was in The Tao Is Silent that I first encountered the Taoistic principles of spontaneity, working with rather than against the nature of things, human goodness as a distinct phenomenon from the heavy weight of dutiful moral obligation, and above all, wei wu wei, "acting through not acting".

Smullyan's Taoism was more inspiration than instruction, but it was important inspiration. I matured as a rationalist while keeping firmly in mind that my "rationality" was not measured by how much effort I expended on proper thinking, but rather how little.

You can see this same view manifested in these lines from The Simple Truth:

"You have to throw in a pebble every time a sheep leaves through the gate?" says Mark. "Take out a pebble every time a sheep returns?"
Autrey nods. "Yeah."
"That must be really hard," Mark says sympathetically.
Autrey brightens, soaking up Mark's sympathy like rain. "Exactly!" says Autrey. "It's extremely hard on your emotions. When the bucket has held its level for a while, you... tend to get attached to that level."
A sheep passes then, leaving through the gate. Autrey sees; he stoops, picks up a pebble, holds it aloft in the air. "Behold!" Autrey proclaims. "A sheep has passed! I must now toss a pebble into this bucket, my dear bucket, and destroy that fond level which has held for so long -" Another sheep passes. Autrey, caught up in his drama, misses it; so I plunk a pebble into the bucket. Autrey is still speaking: "- for that is the supreme test of the shepherd, to throw in the pebble, be it ever so agonizing, be the old level ever so precious. Indeed, only the best of shepherds can meet a requirement so stern -"
"Autrey," I say, "if you want to be a great shepherd someday, learn to shut up and throw in the pebble. No fuss. No drama. Just do it."

Long ago - I think I must have been pretty young - I decided to move my limbs with maximum "efficiency", to save effort. I might even have been thinking of Vulcans.

So I tried what my youthful mind wordlessly conceived of as "efficiency": I tried to move my limbs in perfectly straight lines as quickly as possible, with corresponding sudden stops and sudden starts.

"Efficiency" didn't feel very efficient. The sudden starts and sudden stops took effort. Moving my hand in a straight line forced my elbow and shoulder to move in strange curves.

You can buy books that teach this same life lesson, but they use a lot more pages.

Now this is scarcely Taoism, at least so far as philosophical premises are concerned. According to authentic Taoism, you can exert no effort at all while accomplishing all worthwhile things. This seems to me around as plausible as an agent that achieves its utility function using zero computing power and is therefore maximally intelligent. The only way you could do it is if the agent assigns constant utility to all outcomes, or if the utility function's maximum is set by sleight of hand to wherever the universe goes anyway. This may be why I am not a Taoist: "A maximally intelligent agent with zero computing power and no utility function" sounds like a good metaphor for the Tao. I object to a metric of intelligence that makes me dumber than a rock.

According to Taoism, everyone ought to act in accordance with their natures. One can scarcely see how it could be otherwise. I think this religion only appears nontrivial because of selective failure to consider all its consequences.

In any case, my own nature is to make certain efforts even if they seem unpleasant. Therefore I have no objection to making an effort now and then.

But one should not think that force, effort, and control are virtuous unto a rationalist - that would book them on the wrong side of the ledger.

Addendum: Before anyone else points it out: Yes, I know that my critique of Taoism is appallingly simplistic and that the Taoists are well aware of it. That doesn't make it wrong.

Referenced by: Zen and the Art of Rationality • Inner Goodness

Original with comments: Effortless Technique

Zen and the Art of Rationality

Eliezer Yudkowsky, 24 December 2007 04:36AM

Followup to: Effortless Technique

No one would mistake my writings for ancient Eastern wisdom. Successfully or not, I aspire to clearly set forth the reasoning, antecedent assumptions, and pragmatic use of my conclusions. Successfully or not, I aspire to cut my proposals into modular pieces, so that a user can reject one mistake without destroying the whole. This standard of writing is inherited from the ancient traditions of technical thinking, not the ancient traditions of Zen.

No one would mistake my writings for ancient Eastern wisdom. My goals are not the goals of Buddha or Lao Tse. Feeling Rational suggested that emotions should follow from beliefs but not beliefs follow from emotions: the ideal is to free yourself of all attachment to preferred conclusions about reality, arrive at your beliefs of fact by weighing the evidence without prejudice, and then feel fully whatever emotions follow from these beliefs-of-fact. In stereotypical Eastern philosophy, you are supposed to free yourself of all attachments, not just attachment to beliefs-of-fact apart from evidence; you are supposed to relinquish all desire. Yes, I know it's more complicated than that - but still, their goals are not mine.

And yet it oftimes seems to me that my thoughts are expressed in conceptual language that owes a great deal to the inspiration of Eastern philosophy. "Free yourself of attachments to thinking that the universe is one way or another: Arrive at your picture of the world without prejudice, and then feel fully whichever feelings arise from this picture. Let your emotions flow from your beliefs, not the other way around." It's not a Buddhist conclusion, but the language owes a nod in the direction of Buddhism. Even if a Buddhist teacher would vehemently disagree, they might still grasp immediately what was being proposed. Grasp it more clearly, perhaps, than an old-school (i.e. pre-Bayesian) Western rationalist.

No one would mistake my writings for ancient Eastern wisdom. And this is well, because I can't stand people who try to pass off their ideas as ancient wisdom. As if that were a recommendation! The fifth-century Chinese philosopher Xiaoguang Li observed that ancient civilizations are revered, and yet ancient civilizations are not wise like venerable human elders are wise. A civilization further back in time is younger, not older. The current civilization is always the senior, because the present enjoys a longer history than the past. Incidentally, does it change your opinion if I tell you that Xiaoguang "Mike" Li is actually a friend of mine who lives in the Bay Area?

So be it far from me to spray-paint my work with a patina of venerability. And yet in too many ways to list here, my work owes a nod in the direction of Buddhism, Taoism, Zen - and even Bushido. Yes, Bushido! See e.g. the Musashi quotes in the Twelve Virtues of Rationality. Whatever their other flaws, samurai had a deep grasp of the virtue of perfectionism as a life-principle. To Westerners, "perfectionism" refers to something that seems like work, makes people unhappy, and causes software to ship late.

Of the virtue of curiosity, I said: "A burning itch to know is higher than a solemn vow to pursue truth." Here is the conceptual language - but not the propositional statements - of Lao Tse admonishing, "Stop talking about morality and righteousness, and people will regain the love of their fellows." People are not naturally rational - but you sure can trip over your own feet by thinking too much about "rationality" instead of paying attention to the obvious evidence. Learned virtues are powerful but dangerous; they have many degrees of freedom for error.

Western religions demand submission to God, bended knee and bowed neck. Many Christian saints achieved their canonization by going to great lengths of voluntary suffering. You obey God's precepts out of dutiful morality and reverence, on penalty of judgment and damnation. Such concepts have contaminated Eastern street religions as well, of course. But so far as Eastern religious philosophy is concerned, one speaks of harmony with the Tao, rather than submitting to the Tao.

When I ask myself whether rationality seems more like submitting to the commands of Bayes, or moving in harmony with the Bayes, the latter seems far closer to the mark. By placing yourself in correspondence with the Bayes, you wield the power of the Bayes. If you misstep in the dance (accidentally or deliberately), there is no judge who damns you, or any divine watcher disappointed in you: You have failed yourself. The laws of probability theory still govern you, entirely indifferent to your submission or defiance. The consequences of your disharmony will occur to you according to the natural order of things: the Bayes does not condemn you for your disobedience, but reality will not go according to your hopeful plans. Neither guilt nor repentance will save you, since the Bayes cares nothing for your allegiance. Worshipping the Bayes will not gain its favor, for the Bayes has no ego-desire to demand your praise. Probability theory is there to be used, not believed-in. There is no ancient Taoist manuscript that agrees with such Bayesianity, but the language...

The axioms of Bayesian probability theory make no mention of clothing, and therefore a valid derivation is valid whether you wear a lab coat or a clown suit. The Bayes makes no mention of solemnity or silliness, and therefore lecture on rationality is just the same whether spoken in deep portentous tones or a high squeaky voice from inhaling helium. Understanding what probability theory constrains and does not constrain, we are free to be spontaneous in all other respects. This purity and freedom is preached in no Buddhist tract, but there is something of an Eastern aesthetic about it - and a mathematical aesthetic also, but math knows no East or West, and is simply math.

Miyamoto Musashi said:

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him."

Likewise in rationality. Every step cuts through to the truth in the same movement. Every step carries the map through to reflect the territory. If you fail to achieve a correct answer, it is futile to protest that you acted with propriety. Whether you wear a lab coat or a clown suit, however much it might naively seem to associate with science, does not affect whether you cut through to the correct answer. (This is why I'm not afraid to borrow the language of Taoism, or verse-form when the whim takes me; mere style makes no difference to probability theory.) You might think that such focus, such purposefulness, is more Western than Eastern - but where is the equivalent declaration of Musashi's by a Western philosopher?

Lest I seem to give the East too much praise, I note a well-known truism to the effect that Westerners overestimate the average quality of Eastern philosophy because only the good stuff gets imported. Buddhism seems "atheistic" because you don't read about the ten thousand minor deities unabashedly worshipped on the street. Such selectivity is right and proper, and I make no apology for it. I am not trying for authenticity, that is not my purpose.

Likewise, I don't spend much time pondering my "Western influences" because they are as natural to me as breathing, as unseen to me as air. If I had grown up in Taiwan, my writing would probably sound far more Buddhist and Taoistic; and perhaps I would talk of the inspiration (though not advice) I had received from reading some Taiwanese book about Greek philosophers, and how I often felt closer to Judaism than my forgotten childhood Buddhism.

Nonetheless, I think it a wise thing for an aspiring rationalist to read at least one book of Buddhist or Taoist or Zen philosophy - preferably a book in its original English, recommended to you by some mathematician or programmer or scientist.

Referenced by: Something to Protect • Bind Yourself to Reality • A Broken Koan • Principles of Disagreement • Rebelling Within Nature • Make an Extraordinary Effort • Which Parts Are "Me"? • Inner Goodness • Higher Purpose

Original with comments: Zen and the Art of Rationality

The Amazing Virgin Pregnancy

Eliezer Yudkowsky, 24 December 2007 02:00PM

People who grow up believing certain things,
even if they later stop believing them,
may not quite realize how the beliefs sound to outsiders...

(SCENE: A small cottage in Nazareth.)

Joseph: Mary, my dearest fiancée, there's something I've been meaning to talk to you about.

(Mary's shoulders slump. Slowly, as if under a heavy burden, she turns around to face Joseph.)

Joseph: You seem to be getting fat around the waistline, and throwing up in the morning, and, er, not getting any periods. Which is odd, because it's sort of like -

Mary: Yes! I'm pregnant! All right? I'm PREGNANT!

Joseph: How is that possible?

(Mary's shoulders slump further.) Mary: How do you think?

Joseph: I don't know, that's why I'm asking you. I mean, you're still a virgin, right?

(Mary looks up cautiously, and sees Joseph's face looking blankly puzzled.)

Joseph: Well?

Mary: God did it.

Joseph: You had sex with -

Mary: No! Haha. Of course not. I mean, God just snapped his fingers and did one of those miracle things and made me pregnant.

Joseph: God made you pregnant.

Mary: (Starts to sweat.) Yes.

Joseph: Mary, that is just so... completely...

(Mary's eyes squeeze shut.)

Joseph: ...COOL!

(Mary opens her eyes again, cautiously.)

Mary: You think so?

Joseph: Of course! Who wouldn't think so? Come on, we've got to tell everyone the news!

Mary: Maybe we should keep this between just the two of us -

Joseph: No, no, silly girl, this is way too important! Come on!

(Joseph grabs Mary's wrist and drags her out of the house. SCENE: The gathering square of Nazareth. A dozen well-dressed men, and the town's head rabbi, look on Joseph and Mary impatiently.)

Rabbi: What's this all about, Joseph? I trust there's a good reason for the fuss?

Joseph: Go ahead, Mary! Tell them what you told me.

Mary: Um... (She swallows.) God made me pregnant.

Rabbi, looking stern, yet understanding: Now, Joseph, you know you're not supposed to do that before -

Joseph: No, no, you don't get it! She's still a virgin! God made her pregnant directly!

(There's a long pause.)

Man #1: So, what you're saying here, basically, is that Mary tells you she's a virgin.

Joseph: Uh huh!

Man #2: And you haven't had sex with her.

Joseph: Uh huh!

Man #3: And now she's pregnant.

Joseph: Precisely!

Man #4: So you think that God did it.

Joseph: What other explanation could there be?

Rabbi: Joseph, that is just so... unbelievably...

(Mary holds her breath.)

Rabbi: NEAT!

(Mary exhales.)

Man #5: A miracle! A miracle right here in Nazareth!

Man #6: Wow! I thought that miracles only happened in Jerusalem!

Man #7: Come on! Let's spread the good news!

(They depart. SCENE: Mary is alone with her friend, Betty, in Betty's house.)

Betty: "God did it."

Mary: I panicked! It was all I could think of!

Betty: So who's the real -

(Mary lifts an eyebrow significantly. There's a brief pause.)

Betty: Ah. So that's why the rabbi went along with it.

Mary: Well, he thinks he's the father, anyway. Why, does it matter?

Betty: It puts some things in a different light.

Mary: Like what?

Betty: The rabbi has been telling all the pretty young girls that you, Mary, are the ultimate embodiment of feminine virtue, and when they grow up, they should be just like you -

Mary: I just feel so awful about the whole mess. What kind of thing is this to have hanging over my child's life?

Betty: You've got to put things in perspective, dearie. You told one little white lie. It's not as if you caused the fall of the Roman Empire.

Mary: But what if the Romans hear about it? I don't want my baby to end up being crucified!

Betty: No one's going to obsess about it that long. In a couple of months this whole thing will blow over.

Mary: I hope you're right...

(Exeunt Omnes.)

Referenced by: Entangled Truths, Contagious Lies

Original with comments: The Amazing Virgin Pregnancy

Asch's Conformity Experiment

Eliezer Yudkowsky, 26 December 2007 07:03AM

Solomon Asch, with experiments originally carried out in the 1950s and well-replicated since, highlighted a phenomenon now known as "conformity". In the classic experiment, a subject sees a puzzle like the one in the nearby diagram: Which of the lines A, B, and C is the same size as the line X? Take a moment to determine your own answer...

The gotcha is that the subject is seated alongside a number of other people looking at the diagram - seemingly other subjects, actually confederates of the experimenter. The other "subjects" in the experiment, one after the other, say that line C seems to be the same size as X. The real subject is seated next-to-last. How many people, placed in this situation, would say "C" - giving an obviously incorrect answer that agrees with the unanimous answer of the other subjects? What do you think the percentage would be?

Three-quarters of the subjects in Asch's experiment gave a "conforming" answer at least once. A third of the subjects conformed more than half the time.

Interviews after the experiment showed that while most subjects claimed to have not really believed their conforming answers, some said they'd really thought that the conforming option was the correct one.

Asch was disturbed by these results:

"That we have found the tendency to conformity in our society so strong... is a matter of concern. It raises questions about our ways of education and about the values that guide our conduct."

It is not a trivial question whether the subjects of Asch's experiments behaved irrationally. Robert Aumann's Agreement Theorem shows that honest Bayesians cannot agree to disagree - if they have common knowledge of their probability estimates, they have the same probability estimate. Aumann's Agreement Theorem was proved more than twenty years after Asch's experiments, but it only formalizes and strengthens an intuitively obvious point - other people's beliefs are often legitimate evidence.

If you were looking at a diagram like the one above, but you knew for a fact that the other people in the experiment were honest and seeing the same diagram as you, and three other people said that C was the same size as X, then what are the odds that only you are the one who's right? I lay claim to no advantage of visual reasoning - I don't think I'm better than an average human at judging whether two lines are the same size. In terms of individual rationality, I hope I would notice my own severe confusion and then assign >50% probability to the majority vote.

In terms of group rationality, seems to me that the proper thing for an honest rationalist to say is, "How surprising, it looks to me like B is the same size as X. But if we're all looking at the same diagram and reporting honestly, I have no reason to believe that my assessment is better than yours." The last sentence is important - it's a much weaker claim of disagreement than, "Oh, I see the optical illusion - I understand why you think it's C, of course, but the real answer is B."

So the conforming subjects in these experiments are not automatically convicted of irrationality, based on what I've described so far. But as you might expect, the devil is in the details of the experimental results. According to a meta-analysis of over a hundred replications by Smith and Bond (1996):

Conformity increases strongly up to 3 confederates, but doesn't increase further up to 10-15 confederates. If people are conforming rationally, then the opinion of 15 other subjects should be substantially stronger evidence than the opinion of 3 other subjects.

Adding a single dissenter - just one other person who gives the correct answer, or even an incorrect answer that's different from the group's incorrect answer - reduces conformity very sharply, down to 5-10%. If you're applying some intuitive version of Aumann's Agreement to think that when 1 person disagrees with 3 people, the 3 are probably right, then in most cases you should be equally willing to think that 2 people will disagree with 6 people. (Not automatically true, but true ceteris paribus.) On the other hand, if you've got people who are emotionally nervous about being the odd one out, then it's easy to see how a single other person who agrees with you, or even a single other person who disagrees with the group, would make you much less nervous.

Unsurprisingly, subjects in the one-dissenter condition did not think their nonconformity had been influenced or enabled by the dissenter. Like the 90% of drivers who think they're above-average in the top 50%, some of them may be right about this, but not all. People are not self-aware of the causes of their conformity or dissent, which weighs against trying to argue them as manifestations of rationality. For example, in the hypothesis that people are socially-rationally choosing to lie in order to not stick out, it appears that (at least some) subjects in the one-dissenter condition do not consciously anticipate the "conscious strategy" they would employ when faced with unanimous opposition.

When the single dissenter suddenly switched to conforming to the group, subjects' conformity rates went back up to just as high as in the no-dissenter condition. Being the first dissenter is a valuable (and costly!) social service, but you've got to keep it up.

Consistently within and across experiments, all-female groups (a female subject alongside female confederates) conform significantly more often than all-male groups. Around one-half the women conform more than half the time, versus a third of the men. If you argue that the average subject is rational, then apparently women are too agreeable and men are too disagreeable, so neither group is actually rational...

Ingroup-outgroup manipulations (e.g., a handicapped subject alongside other handicapped subjects) similarly show that conformity is significantly higher among members of an ingroup.

Conformity is lower in the case of blatant diagrams, like the one at the top of this page, versus diagrams where the errors are more subtle. This is hard to explain if (all) the subjects are making a socially rational decision to avoid sticking out.

Added: Paul Crowley reminds me to note that when subjects can respond in a way that will not be seen by the group, conformity also drops, which also argues against an Aumann interpretation.

YouTube video of a conformity experiment:

Asch, S. E. (1956). Studies of independence and conformity: A minority of one against a unanimous majority. Psychological Monographs, 70.

Bond, R. and Smith, P. B. (1996.) Culture and Conformity: A Meta-Analysis of Studies Using Asch's ( 1952b, 1956) Line Judgment Task. Psychological Bulletin, 119, 111-137.

Sequence: Death Spirals and the Cult Attractor

Two Cult Koans

Lonely Dissent

Sequence: Seeing with Fresh Eyes

Hold Off On Proposing Solutions

On Expressing Your Concerns

Referenced by: On Expressing Your Concerns • Lonely Dissent • Cultish Countercultishness • No Safe Defense, Not Even Science • Fundamental Doubts • Raising the Sanity Waterline • Can Humanism Match Religion's Output? • That Magical Click

Original with comments: Asch's Conformity Experiment

On Expressing Your Concerns

Eliezer Yudkowsky, 27 December 2007 04:04AM

Followup to: Asch's Conformity Experiment

The scary thing about Asch's conformity experiments is that you can get many people to say black is white, if you put them in a room full of other people saying the same thing. The hopeful thing about Asch's conformity experiments is that a single dissenter tremendously drove down the rate of conformity, even if the dissenter was only giving a different wrong answer. And the wearisome thing is that dissent was not learned over the course of the experiment - when the single dissenter started siding with the group, rates of conformity rose back up.

Being a voice of dissent can bring real benefits to the group. But it also (famously) has a cost. And then you have to keep it up. Plus you could be wrong.

I recently had an interesting experience wherein I began discussing a project with two people who had previously done some planning on their own. I thought they were being too optimistic and made a number of safety-margin-type suggestions for the project. Soon a fourth guy wandered by, who was providing one of the other two with a ride home, and began making suggestions. At this point I had a sudden insight about how groups become overconfident, because whenever I raised a possible problem, the fourth guy would say, "Don't worry, I'm sure we can handle it!" or something similarly reassuring.

An individual, working alone, will have natural doubts. They will think to themselves, "Can I really do XYZ?", because there's nothing impolite about doubting your own competence. But when two unconfident people form a group, it is polite to say nice and reassuring things, and impolite to question the other person's competence. Together they become more optimistic than either would be on their own, each one's doubts quelled by the other's seemingly confident reassurance, not realizing that the other person initially had the same inner doubts.

The most fearsome possibility raised by Asch's experiments on conformity is the specter of everyone agreeing with the group, swayed by the confident voices of others, careful not to let their own doubts show - not realizing that others are suppressing similar worries. This is known as "pluralistic ignorance".

Robin Hanson and I have a long-running debate over when, exactly, aspiring rationalists should dare to disagree. I tend toward the widely held position that you have no real choice but to form your own opinions. Robin Hanson advocates a more iconoclastic position, that you - not just other people - should consider that others may be wiser. Regardless of our various disputes, we both agree that Aumann's Agreement Theorem extends to imply that common knowledge of a factual disagreement shows someone must be irrational. Despite the funny looks we've gotten, we're sticking to our guns about modesty: Forget what everyone tells you about individualism, you should pay attention to what other people think.

Ahem. The point is that, for rationalists, disagreeing with the group is serious business. You can't wave it off with "Everyone is entitled to their own opinion."

I think the most important lesson to take away from Asch's experiments is to distinguish "expressing concern" from "disagreement". Raising a point that others haven't voiced is not a promise to disagree with the group at the end of its discussion.

The ideal Bayesian's process of convergence involves sharing evidence that is unpredictable to the listener. The Aumann agreement result holds only for common knowledge, where you know, I know, you know I know, etc. Hanson's post or paper on "We Can't Foresee to Disagree" provides a picture of how strange it would look to watch ideal rationalists converging on a probability estimate; it doesn't look anything like two bargainers in a marketplace converging on a price.

Unfortunately, there's not much difference socially between "expressing concerns" and "disagreement". A group of rationalists might agree to pretend there's a difference, but it's not how human beings are really wired. Once you speak out, you've committed a socially irrevocable act; you've become the nail sticking up, the discord in the comfortable group harmony, and you can't undo that. Anyone insulted by a concern you expressed about their competence to successfully complete task XYZ, will probably hold just as much of a grudge afterward if you say "No problem, I'll go along with the group" at the end.

Asch's experiment shows that the power of dissent to inspire others is real. Asch's experiment shows that the power of conformity is real. If everyone refrains from voicing their private doubts, that will indeed lead groups into madness. But history abounds with lessons on the price of being the first, or even the second, to say that the Emperor has no clothes. Nor are people hardwired to distinguish "expressing a concern" from "disagreement even with common knowledge"; this distinction is a rationalist's artifice. If you read the more cynical brand of self-help books (e.g. Machiavelli's The Prince) they will advise you to mask your nonconformity entirely, not voice your concerns first and then agree at the end. If you perform the group service of being the one who gives voice to the obvious problems, don't expect the group to thank you for it.

These are the costs and the benefits of dissenting - whether you "disagree" or just "express concern" - and the decision is up to you.

Sequence: Seeing with Fresh Eyes

Asch's Conformity Experiment

Lonely Dissent

Referenced by: Why Our Kind Can't Cooperate

Original with comments: On Expressing Your Concerns

Lonely Dissent

Eliezer Yudkowsky, 28 December 2007 04:23AM

Followup to: The Modesty Argument, The "Outside the Box" Box, Asch's Conformity Experiment

Asch's conformity experiment showed that the presence of a single dissenter tremendously reduced the incidence of "conforming" wrong answers. Individualism is easy, experiment shows, when you have company in your defiance. Every other subject in the room, except one, says that black is white. You become the second person to say that black is black. And it feels glorious: the two of you, lonely and defiant rebels, against the world! (Followup interviews showed that subjects in the one-dissenter condition expressed strong feelings of camaraderie with the dissenter - though, of course, they didn't think the presence of the dissenter had influenced their own nonconformity.)

But you can only join the rebellion, after someone, somewhere, becomes the first to rebel. Someone has to say that black is black after hearing everyone else, one after the other, say that black is white. And that - experiment shows - is a lot harder.

Lonely dissent doesn't feel like going to school dressed in black. It feels like going to school wearing a clown suit.

That's the difference between joining the rebellion and leaving the pack.

If there's one thing I can't stand, it's fakeness - you may have noticed this if you've been reading Overcoming Bias for a while. Well, lonely dissent has got to be one of the most commonly, most ostentatiously faked characteristics around. Everyone wants to be an iconoclast.

I don't mean to degrade the act of joining a rebellion. There are rebellions worth joining. It does take courage to brave the disapproval of your peer group, or perhaps even worse, their shrugs. Needless to say, going to a rock concert is not rebellion. But, for example, vegetarianism is. I'm not a vegetarian myself, but I respect people who are, because I expect it takes a noticeable amount of quiet courage to tell people that hamburgers won't work for dinner. (Albeit that in the Bay Area, people ask as a matter of routine.)

Still, if you tell people that you're a vegetarian, they'll think they understand your motives (even if they don't). They may disagree. They may be offended if you manage to announce it proudly enough, or for that matter, they may be offended just because they're easily offended. But they know how to relate to you.

When someone wears black to school, the teachers and the other children understand the role thereby being assumed in their society. It's Outside the System - in a very standard way that everyone recognizes and understands. Not, y'know, actually outside the system. It's a Challenge to Standard Thinking, of a standard sort, so that people indignantly say "I can't understand why you -", but don't have to actually think any thoughts they had not thought before. As the saying goes, "Has any of the 'subversive literature' you've read caused you to modify any of your political views?"

What takes real courage is braving the outright incomprehension of the people around you, when you do something that isn't Standard Rebellion #37, something for which they lack a ready-made script. They don't hate you for a rebel, they just think you're, like, weird, and turn away. This prospect generates a much deeper fear. It's the difference between explaining vegetarianism and explaining cryonics. There are other cryonicists in the world, somewhere, but they aren't there next to you. You have to explain it, alone, to people who just think it's weird. Not forbidden, but outside bounds that people don't even think about. You're going to get your head frozen? You think that's going to stop you from dying? What do you mean, brain information? Huh? What? Are you crazy?

I'm tempted to essay a post facto explanation in evolutionary psychology: You could get together with a small group of friends and walk away from your hunter-gatherer band, but having to go it alone in the forests was probably a death sentence - at least reproductively. We don't reason this out explicitly, but that is not the nature of evolutionary psychology. Joining a rebellion that everyone knows about is scary, but nowhere near as scary as doing something really differently. Something that in ancestral times might have ended up, not with the band splitting, but with you being driven out alone.

As the case of cryonics testifies, the fear of thinking really different is stronger than the fear of death. Hunter-gatherers had to be ready to face death on a routine basis, hunting large mammals, or just walking around in a world that contained predators. They needed that courage in order to live. Courage to defy the tribe's standard ways of thinking, to entertain thoughts that seem truly weird - well, that probably didn't serve its bearers as well. We don't reason this out explicitly; that's not how evolutionary psychology works. We human beings are just built in such fashion that many more of us go skydiving than sign up for cryonics.

And that's not even the highest courage. There's more than one cryonicist in the world. Only Robert Ettinger had to say it first.

To be a scientific revolutionary, you've got to be the first person to contradict what everyone else you know is thinking. This is not the only route to scientific greatness; it is rare even among the great. No one can become a scientific revolutionary by trying to imitate revolutionariness. You can only get there by pursuing the correct answer in all things, whether the correct answer is revolutionary or not. But if, in the due course of time - if, having absorbed all the power and wisdom of the knowledge that has already accumulated - if, after all that and a dose of sheer luck, you find your pursuit of mere correctness taking you into new territory... then you have an opportunity for your courage to fail.

This is the true courage of lonely dissent, which every damn rock band out there tries to fake.

Of course not everything that takes courage is a good idea. It would take courage to walk off a cliff, but then you would just go splat.

The fear of lonely dissent is a hindrance to good ideas, but not every dissenting idea is good. See also Robin Hanson's Against Free Thinkers. Most of the difficulty in having a new true scientific thought is in the "true" part.

It really isn't necessary to be different for the sake of being different. If you do things differently only when you see an overwhelmingly good reason, you will have more than enough trouble to last you the rest of your life.

There are a few genuine packs of iconoclasts around. The Church of the SubGenius, for example, seems to genuinely aim at confusing the mundanes, not merely offending them. And there are islands of genuine tolerance in the world, such as science fiction conventions. There are certain people who have no fear of departing the pack. Many fewer such people really exist, than imagine themselves rebels; but they do exist. And yet scientific revolutionaries are tremendously rarer. Ponder that.

Now me, you know, I really am an iconoclast. Everyone thinks they are, but with me it's true, you see. I would totally have worn a clown suit to school. My serious conversations were with books, not with other children.

But if you think you would totally wear that clown suit, then don't be too proud of that either! It just means that you need to make an effort in the opposite direction to avoid dissenting too easily. That's what I have to do, to correct for my own nature. Other people do have reasons for thinking what they do, and ignoring that completely is as bad as being afraid to contradict them. You wouldn't want to end up as a free thinker. It's not a virtue, you see - just a bias either way.

Sequence: Death Spirals and the Cult Attractor

Asch's Conformity Experiment

Cultish Countercultishness

Sequence: Seeing with Fresh Eyes

On Expressing Your Concerns

The Genetic Fallacy

Referenced by: To Lead, You Must Stand Up • Cultish Countercultishness • Something to Protect • No Safe Defense, Not Even Science • Timeless Identity • Make an Extraordinary Effort • Why Our Kind Can't Cooperate • Selecting Rationalist Groups • The End (of Sequences) • Normal Cryonics

Original with comments: Lonely Dissent

To Lead, You Must Stand Up

Eliezer Yudkowsky, 29 December 2007 06:38AM

Followup to: Lonely Dissent

True story: In July, I attended a certain Silicon Valley event. I was not an organizer, or a speaker, or in any other wise involved on an official level; just an attendee. It was an evening event, and after the main presentations were done, much of the audience hung around talking... and talking... and talking... Finally the event organizer began dimming the lights and turning them back up again. And the crowd still stayed; no one left. So the organizer dimmed the lights and turned them up some more. And lo, the people continued talking.

I walked over to the event organizer, standing by the light switches, and said, "Are you hinting for people to leave?" And he said, "Yes. In fact [the host company] says we've got to get out of here now - the building needs to close down."

I nodded.

I walked over to the exit.

I shouted, "LISTEN UP, EVERYONE! WE'VE GOT TO GO! OUR TIME HERE HAS PASSED! YOU CAN TALK OUTSIDE IF YOU LIKE! NOW FOLLOW ME... TO FREEDOM!"

I turned.

I marched out the door.

And everyone followed.

I expect there were at least two or three CEOs in that Silicon Valley crowd. It didn't lack for potential leaders. Why was it left to me to lead the CEOs to freedom?

Well, what was in it for them to perform that service to the group? It wasn't their problem. I'm in the habit of doing work I see being left undone; but this doesn't appear to be a common habit.

So why didn't some aspiring would-be future-CEO take the opportunity to distinguish themselves by acting the part of the leader? I bet at least five people in that Silicon Valley crowd had recently read a business book on leadership...

But it's terribly embarrassing to stand up in front of a crowd. What if the crowd hadn't followed me? What if I'd turned and marched out the door, and been left looking like a complete fool? Oh nos! Oh horrors!

While I have sometimes pretended to wisdom, I have never pretended to solemnity. I wasn't worried about looking silly, because heck, I am silly. It runs in the Yudkowsky family. There is a difference between being serious and being solemn.

As for daring to stand out in the crowd, to have everyone staring at me - that was a feature of grade school. The first time I gave a presentation - the first time I ever climbed onto a stage in front of a couple of hundred people to talk about the Singularity - I briefly thought to myself: "I bet most people would be experiencing 'stage fright' about now. But that wouldn't be helpful, so I'm not going to go there."

I expect that a majority of my readers like to think of themselves as having strong leadership qualities. Well, maybe you do, and maybe you don't. But you'll never get a chance to express those leadership qualities if you're too embarrassed to call attention to yourself, to stand up in front of the crowd and have all eyes turn to you. To lead the pack, you must be willing to leave the pack.

Referenced by: ESR's comments on some EY:OB/LW posts

Original with comments: To Lead, You Must Stand Up

Cultish Countercultishness

Eliezer Yudkowsky, 30 December 2007 12:53AM

Followup to: Every Cause Wants To Be A Cult, Lonely Dissent

In the modern world, joining a cult is probably one of the worse things that can happen to you. The best-case scenario is that you'll end up in a group of sincere but deluded people, making an honest mistake but otherwise well-behaved, and you'll spend a lot of time and money but end up with nothing to show. Actually, that could describe any failed Silicon Valley startup. Which is supposed to be a hell of a harrowing experience, come to think. So yes, very scary.

Real cults are vastly worse. "Love bombing" as a recruitment technique, targeted at people going through a personal crisis. Sleep deprivation. Induced fatigue from hard labor. Distant communes to isolate the recruit from friends and family. Daily meetings to confess impure thoughts. It's not unusual for cults to take all the recruit's money - life savings plus weekly paycheck - forcing them to depend on the cult for food and clothing. Starvation as a punishment for disobedience. Serious brainwashing and serious harm.

With all that taken into account, I should probably sympathize more with people who are terribly nervous, embarking on some odd-seeming endeavor, that they might be joining a cult. It should not grate on my nerves. Which it does.

Point one: "Cults" and "non-cults" aren't separated natural kinds like dogs and cats. If you look at any list of cult characteristics, you'll see items that could easily describe political parties and corporations - "group members encouraged to distrust outside criticism as having hidden motives", "hierarchical authoritative structure". I've posted on group failure modes like group polarization, happy death spirals, uncriticality, and evaporative cooling, all of which seem to feed on each other. When these failures swirl together and meet, they combine to form a Super-Failure stupider than any of the parts, like Voltron. But this is not a cult essence; it is a cult attractor.

Dogs are born with dog DNA, and cats are born with cat DNA. In the current world, there is no in-between. (Even with genetic manipulation, it wouldn't be as simple as creating an organism with half dog genes and half cat genes.) It's not like there's a mutually reinforcing set of dog-characteristics, which an individual cat can wander halfway into and become a semidog.

The human mind, as it thinks about categories, seems to prefer essences to attractors. The one wishes to say "It is a cult" or "It is not a cult", and then the task of classification is over and done. If you observe that Socrates has ten fingers, wears clothes, and speaks fluent Greek, then you can say "Socrates is human" and from there deduce "Socrates is vulnerable to hemlock" without doing specific blood tests to confirm his mortality. You have decided Socrates's humanness once and for all.

But if you observe that a certain group of people seems to exhibit ingroup-outgroup polarization and see a positive halo effect around their Favorite Thing Ever - which could be Objectivism, or vegetarianism, or neural networks - you cannot, from the evidence gathered so far, deduce whether they have achieved uncriticality. You cannot deduce whether their main idea is true, or false, or genuinely useful but not quite as useful as they think. From the information gathered so far, you cannot deduce whether they are otherwise polite, or if they will lure you into isolation and deprive you of sleep and food. The characteristics of cultness are not all present or all absent.

If you look at online arguments over "X is a cult", "X is not a cult", then one side goes through an online list of cult characteristics and finds one that applies and says "Therefore is a cult!" And the defender finds a characteristic that does not apply and says "Therefore it is not a cult!"

You cannot build up an accurate picture of a group's reasoning dynamic using this kind of essentialism. You've got to pay attention to individual characteristics individually.

Furthermore, reversed stupidity is not intelligence. If you're interested in the central idea, not just the implementation group, then smart ideas can have stupid followers. Lots of New Agers talk about "quantum physics" but this is no strike against quantum physics. Of course stupid ideas can also have stupid followers. Along with binary essentialism goes the idea that if you infer that a group is a "cult", therefore their beliefs must be false, because false beliefs are characteristic of cults, just like cats have fur. If you're interested in the idea, then look at the idea, not the people. Cultishness is a characteristic of groups more than hypotheses.

The second error is that when people nervously ask, "This isn't a cult, is it?" it sounds to me like they're seeking reassurance of rationality. The notion of a rationalist not getting too attached to their self-image as a rationalist deserves its own post (though see this, this and this). But even without going into detail, surely one can see that nervously seeking reassurance is not the best frame of mind in which to evaluate questions of rationality. You will not be genuinely curious or think of ways to fulfill your doubts. Instead, you'll find some online source which says that cults use sleep deprivation to control people, you'll notice that Your-Favorite-Group doesn't use sleep deprivation, and you'll conclude "It's not a cult. Whew!" If it doesn't have fur, it must not be a cat. Very reassuring.

But Every Cause Wants To Be A Cult, whether the cause itself is wise or foolish. The ingroup-outgroup dichotomy etc. are part of human nature, not a special curse of mutants. Rationality is the exception, not the rule. You have to put forth a constant effort to maintain rationality against the natural slide into entropy. If you decide "It's not a cult!" and sigh with relief, then you will not put forth a continuing effort to push back ordinary tendencies toward cultishness. You'll decide the cult-essence is absent, and stop pumping against the entropy of the cult-attractor.

If you are terribly nervous about cultishness, then you will want to deny any hint of any characteristic that resembles a cult. But any group with a goal seen in a positive light, is at risk for the halo effect, and will have to pump against entropy to avoid an affective death spiral. This is true even for ordinary institutions like political parties - people who think that "liberal values" or "conservative values" can cure cancer, etc. It is true for Silicon Valley startups, both failed and successful. It is true of Mac users and of Linux users. The halo effect doesn't become okay just because everyone does it; if everyone walks off a cliff, you wouldn't too. The error in reasoning is to be fought, not tolerated. But if you're too nervous about "Are you sure this isn't a cult?" then you will be reluctant to see any sign of cultishness, because that would imply you're in a cult, and It's not a cult!! So you won't see the current battlefields where the ordinary tendencies toward cultishness are creeping forward, or being pushed back.

The third mistake in nervously asking "This isn't a cult, is it?" is that, I strongly suspect, the nervousness is there for entirely the wrong reasons.

Why is it that groups which praise their Happy Thing to the stars, encourage members to donate all their money and work in voluntary servitude, and run private compounds in which members are kept tightly secluded, are called "religions" rather than "cults" once they've been around for a few hundred years?

Why is it that most of the people who nervously ask of cryonics, "This isn't a cult, is it?" would not be equally nervous about attending a Republican or Democrat political rally? Ingroup-outgroup dichotomies and happy death spirals can happen in political discussion, in mainstream religions, in sports fandom. If the nervousness came from fear of rationality errors, people would ask "This isn't an ingroup-outgroup dichotomy, is it?" about Democrat or Republican political rallies, in just the same fearful tones.

There's a legitimate reason to be less fearful of Libertarianism than of a flying-saucer cult, because Libertarians don't have a reputation for employing sleep deprivation to convert people. But cryonicists don't have a reputation for using sleep deprivation, either. So why be any more worried about having your head frozen after you stop breathing?

I suspect that the nervousness is not the fear of believing falsely, or the fear of physical harm. It is the fear of lonely dissent. The nervous feeling that subjects get in Asch's conformity experiment, when all the other subjects (actually confederates) say one after another that line C is the same size as line X, and it looks to the subject like line B is the same size as line X. The fear of leaving the pack.

That's why groups whose beliefs have been around long enough to seem "normal" don't inspire the same nervousness as "cults", though some mainstream religions may also take all your money and send you to a monastery. It's why groups like political parties, that are strongly liable for rationality errors, don't inspire the same nervousness as "cults". The word "cult" isn't being used to symbolize rationality errors, it's being used as a label for something that seems weird.

Not every change is an improvement, but every improvement is necessarily a change. That which you want to do better, you have no choice but to do differently. Common wisdom does embody a fair amount of, well, actual wisdom; yes, it makes sense to require an extra burden of proof for weirdness. But the nervousness isn't that kind of deliberate, rational consideration. It's the fear of believing something that will make your friends look at you really oddly. And so people ask "This isn't a cult, is it?" in a tone that they would never use for attending a political rally, or for putting up a gigantic Christmas display.

That's the part that bugs me.

It's as if, as soon as you believe anything that your ancestors did not believe, the Cult Fairy comes down from the sky and infuses you with the Essence of Cultness, and the next thing you know, you're all wearing robes and chanting. As if "weird" beliefs are the direct cause of the problems, never mind the sleep deprivation and beatings. The harm done by cults - the Heaven's Gate suicide and so on - just goes to show that everyone with an odd belief is crazy; the first and foremost characteristic of "cult members" is that they are Outsiders with Peculiar Ways.

Yes, socially unusual belief puts a group at risk for ingroup-outgroup thinking and evaporative cooling and other problems. But the unusualness is a risk factor, not a disease in itself. Same thing with having a goal that you think is worth accomplishing. Whether or not the belief is true, having a nice goal always puts you at risk of the happy death spiral. But that makes lofty goals a risk factor, not a disease. Some goals are genuinely worth pursuing.

On the other hand, I see no legitimate reason for sleep deprivation or threatening dissenters with beating, full stop. When a group does this, then whether you call it "cult" or "not-cult", you have directly answered the pragmatic question of whether to join.

Problem four: The fear of lonely dissent is something that cults themselves exploit. Being afraid of your friends looking at you disapprovingly is exactly the effect that real cults use to convert and keep members - surrounding converts with wall-to-wall agreement among cult believers.

The fear of strange ideas, the impulse to conformity, has no doubt warned many potential victims away from flying-saucer cults. When you're out, it keeps you out. But when you're in, it keeps you in. Conformity just glues you to wherever you are, whether that's a good place or a bad place.

The one wishes there was some way they could be sure that they weren't in a "cult". Some definite, crushing rejoinder to people who looked at them funny. Some way they could know once and for all that they were doing the right thing, without these constant doubts. I believe that's called "need for closure". And - of course - cults exploit that, too.

Hence the phrase, "Cultish countercultishness."

Living with doubt is not a virtue - the purpose of every doubt is to annihilate itself in success or failure, and a doubt that just hangs around, accomplishes nothing. But sometimes a doubt does take a while to annihilate itself. Living with a stack of currently unresolved doubts is an unavoidable fact of life for rationalists. Doubt shouldn't be scary. Otherwise you're going to have to choose between living one heck of a hunted life, or one heck of a stupid one.

If you really, genuinely can't figure out whether a group is a "cult", then you'll just have to choose under conditions of uncertainty. That's what decision theory is all about.

Problem five: Lack of strategic thinking.

I know people who are cautious around Singularitarianism, and they're also cautious around political parties and mainstream religions. Cautious, not nervous or defensive. These people can see at a glance that Singularitarianism is obviously not a full-blown cult with sleep deprivation etc. But they worry that Singularitarianism will become a cult, because of risk factors like turning the concept of a powerful AI into a Super Happy Agent (an agent defined primarily by agreeing with any nice thing said about it). Just because something isn't a cult now, doesn't mean it won't become a cult in the future. Cultishness is an attractor, not an essence.

Does this kind of caution annoy me? Hell no. I spend a lot of time worrying about that scenario myself. I try to place my Go stones in advance to block movement in that direction. Hence, for example, the series of posts on cultish failures of reasoning.

People who talk about "rationality" also have an added risk factor. Giving people advice about how to think is an inherently dangerous business. But it is a risk factor, not a disease.

Both of my favorite Causes are at-risk for cultishness. Yet somehow, I get asked "Are you sure this isn't a cult?" a lot more often when I talk about powerful AIs, than when I talk about probability theory and cognitive science. I don't know if one risk factor is higher than the other, but I know which one sounds weirder...

Problem #6 with asking "This isn't a cult, is it?"...

Just the question itself places me in a very annoying sort of Catch-22. An actual Evil Guru would surely use the one's nervousness against them, and design a plausible elaborate argument explaining Why This Is Not A Cult, and the one would be eager to accept it. Sometimes I get the impression that this is what people want me to do! Whenever I try to write about cultishness and how to avoid it, I keep feeling like I'm giving in to that flawed desire - that I am, in the end, providing people with reassurance. Even when I tell people that a constant fight against entropy is required.

It feels like I'm making myself a first dissenter in Asch's conformity experiment, telling people, "Yes, line X really is the same as line B, it's okay for you to say so too." They shouldn't need to ask! Or, even worse, it feels like I'm presenting an elaborate argument for Why This Is Not A Cult. It's a wrong question.

Just look at the group's reasoning processes for yourself, and decide for yourself whether it's something you want to be part of, once you get rid of the fear of weirdness. It is your own responsibility to stop yourself from thinking cultishly, no matter which group you currently happen to be operating in.

Once someone asks "This isn't a cult, is it?" then no matter how I answer, I always feel like I'm defending something. I do not like this feeling. It is not the function of a Bayesian Master to give reassurance, nor of rationalists to defend.

Cults feed on groupthink, nervousness, desire for reassurance. You cannot make nervousness go away by wishing, and false self-confidence is even worse. But so long as someone needs reassurance - even reassurance about being a rationalist - that will always be a flaw in their armor. A skillful swordsman focuses on the target, rather than glancing away to see if anyone might be laughing. When you know what you're trying to do and why, you'll know whether you're getting it done or not, and whether a group is helping you or hindering you.

(PS: If the one comes to you and says, "Are you sure this isn't a cult?", don't try to explain all these concepts in one breath. You're underestimating inferential distances. The one will say, "Aha, so you're admitting you're a cult!" or "Wait, you're saying I shouldn't worry about joining cults?" or "So... the fear of cults is cultish? That sounds awfully cultish to me." So the last annoyance factor - #7 if you're keeping count - is that all of this is such a long story to explain.)

Sequence: Death Spirals and the Cult Attractor

Lonely Dissent

Referenced by: Disguised Queries • Neural Categories • To Spread Science, Keep It Secret • Heat vs. Motion • No Safe Defense, Not Even Science • When Anthropomorphism Became Stupid • Crisis of Faith • Is That Your True Rejection? • Schools Proliferating Without Evidence • Why Our Kind Can't Cooperate • You're Calling *Who* A Cult Leader? • Whining-Based Communities

Original with comments: Cultish Countercultishness

My Strange Beliefs

Eliezer Yudkowsky, 30 December 2007 12:15PM

Yesterday, "Overcoming Cryonics" wrote:

Eliezer, enough with your nonsense about cryonicism, life-extensionism, trans-humanism, and the singularity. These things have nothing to do with overcoming bias... if you're going to enforce the comments policy then you should also self-enforce the overcoming bias posting policy instead of using posts to blithely proselytize your cryonicism / life-extensionism / trans-humanism / singularity religion.

One, there is nothing in the Overcoming Bias posting policy against transhumanism.

Two, as a matter of fact, I do try to avoid proselytizing here. I have other forums in which to vent my thoughts on transhumanism. When I write a blog post proselytizing transhumanism, it looks like this, this, or this.

But it's hard for me to avoid all references to transhumanism. "Overcoming Cryonics" commented to a post in which there was exactly one reference to a transhumanist topic. I had said:

The first time I gave a presentation - the first time I ever climbed onto a stage in front of a couple of hundred people to talk about the Singularity - I briefly thought to myself: "I bet most people would be experiencing 'stage fright' about now. But that wouldn't be helpful, so I'm not going to go there.

What, exactly, am I supposed to do about that? The first time I ever got up on stage, I was in fact talking about the Singularity! That's the actual history! Transhumanism is not a hobby for me, it's my paid day job as a Research Fellow of the Singularity Institute. Asking me to avoid all mentions of transhumanism is like asking Robin Hanson to avoid all mentions of academia.

Occasionally, someone remarks that I seem to take notions like the Singularity on faith, because I mention them but don't defend them.

I don't defend my views here. Because I know that not everyone is interested in the considerable volume of work I have produced on transhumanism. Which you can find on yudkowsky.net.

If, however, you don't like any mention of transhumanism, even as an illustration of some other point about rationality - well, this is a blog. These are blog posts. They are written in the first person. I am occasionally going to use anecdotes from my history, or even, y'know, transcribe my thought processes a little?

Given the amount of time that I spend thinking about transhumanism, I naturally tend to think of transhumanist illustrations for my points about rationality. If I had spent the last eleven years as a geologist, I would find it easy to illustrate my ideas by talking about rocks. If you don't like my illustrations and think you can do better, feel free to invent superior illustrations and post them in the comments. I may even adopt them.

On some transhumanist topics, such as cryonics, I haven't written all that much myself. But there is plenty about cryonics at Alcor or Cryonics Institute. Also, the Transhumanist FAQ has some nice intros. If you don't want it discussed here, then why are you asking?

I will probably post explicitly on cryonics at some point, because I think there are some points about sour grapes for which I would have difficulty finding an equally strong illustration. Meanwhile, yes, I sometimes do mention "cryonics" as the archetype for a socially weird belief which happens to be true. No matter what I use as an example of "socially weird but true", some people are going to disagree with it. Otherwise it wouldn't be an example. And weird-but-true is certainly an important topic in rationality - otherwise there would be a knockdown argument against ever dissenting.

Even after checking the referenced sources, you might find that you - gasp! - still disagree with me. Oh, the horror! The horror! You don't read any other blogs where one of the authors occasionally disagrees with you.

Just because this blog is called Overcoming Bias, it does not mean that any time any author says something you disagree with, you should comment "OMG! How biased! I am sooo disappointed in you I thought you would do better." Part of the art of rationality is having extended discussions with people you disagree with. "OMG U R BIASED!" does not present much basis for continuing discussion.

It is a good rule of thumb that you should never flatly accuse someone of being "biased". Name the specific bias that attaches to the specific problem. Conjunction fallacy? Availability?

If you disagree with someone, you presumably think they're doing something wrong. Saying "You are like so totally biased, dude" is not helpful. If you strike a tragic, sorrowful pose and go, "Oh, alas, oh, woe, I am so disappointed in you," it is still not helpful. If you point to a specific belief that you disagree with, and say, "See, that belief is biased," then that doesn't convey any additional information beyond "I disagree with that belief." Which bias? There's quite a lot of possibilities.

If you think that "rationality" means people will agree with you on their first try, so that anyone who doesn't do this can be dismissed out of hand as a poseur, you have an exaggerated idea of how obvious your beliefs are.

So stop telling me, or Robin Hanson, "Why, you... you... you're not absolutely rational!" We already know that.

Just because I try to be rational doesn't mean I think I'm a god.

Well, sure, I want to be a god when I grow up, but that is like a totally different issue from that first part.

Except that both goals involve Bayesian methods.

(And are intertwined in other ways you won't realize until it's too late to turn back.)

Thank you.

Yours in the darkest abyssal depths of sincerity,
Eliezer Yudkowsky.

Referenced by: The American System and Misleading Labels • Against Discount Rates • Identity Isn't In Specific Atoms

Original with comments: My Strange Beliefs

Posting on Politics

Eliezer Yudkowsky, 01 January 2008 07:25AM

Politics, ah, politics! If human insanities could physically manifest as lumps of putrefaction, and the dripping slimes collected into a giant festering pit, and the mound of corruption formed itself into a monster and shambled forth to eat brains...

Ordinarily I prefer to discuss politics indirectly, rather than directly. Politics varies from government to government. Better to talk about human universals - cognitive biases that can be nailed down and examined in the laboratory - malfunctions of sanity that appear wherever humans go. Then it's up to you to apply the knowledge to your own political situation.

This policy also avoids offending people, so I tend to suspect my clever-sounding rationale for following it.

Over the next two weeks, Iowa and New Hampshire will exercise their Constitutional right to appoint the next President of the United States. I hope you will forgive me if I am, briefly, relevant.

I intend to do a series of three posts directly applying to politics. Don't worry, after that it's back to the safe refuge of cognitive science.

Rest assured that I don't plan on endorsing a party, let alone a candidate.

If I say something that you disagree with, remember that my attempts at rationality are not sourced from a divine scripture, and hence are not a package deal. You read plenty of other blogs, I hope, where an author occasionally says something you dislike, from time to time?

Original with comments: Posting on Politics

The Two-Party Swindle

Eliezer Yudkowsky, 01 January 2008 08:38AM

The Robbers Cave Experiment had as its subject 22 twelve-year-old boys, selected from 22 different schools in Oklahoma City, all doing well in school, all from stable middle-class Protestant families. In short, the boys were as similar to each other as the experimenters could arrange, though none started out knowing any of the others. The experiment, conducted in the aftermath of WWII, was meant to investigate conflicts between groups. How would the scientists spark an intergroup conflict to investigate? Well, the first step was to divide the 22 boys into two groups of 11 campers -

- and that was quite sufficient. There was hostility almost from the moment each group became aware of the other group's existence. Though they had not needed any name for themselves before, they named themselves the Eagles and the Rattlers. After the researchers (disguised as camp counselors) instigated contests for prizes, rivalry reached a fever pitch and all traces of good sportsmanship disintegrated. The Eagles stole the Rattlers' flag and burned it; the Rattlers raided the Eagles' cabin and stole the blue jeans of the group leader and painted it orange and carried it as a flag the next day.

Each group developed a stereotype of itself and a contrasting stereotype of the opposing group (though the boys had been initially selected to be as similar as possible). The Rattlers swore heavily and regarded themselves as rough-and-tough. The Eagles swore off swearing, and developed an image of themselves as proper-and-moral.

Consider, in this light, the episode of the Blues and the Greens in the days of Rome. Since the time of the ancient Romans, and continuing into the era of Byzantium and the Roman Empire, the Roman populace had been divided into the warring Blue and Green factions. Blues murdered Greens and Greens murdered Blues, despite all attempts at policing. They died in single combats, in ambushes, in group battles, in riots.

From Procopius, History of the Wars, I:

In every city the population has been divided for a long time past into the Blue and the Green factions [...] And they fight against their opponents knowing not for what end they imperil themselves [...] So there grows up in them against their fellow men a hostility which has no cause, and at no time does it cease or disappear, for it gives place neither to the ties of marriage nor of relationship nor of friendship, and the case is the same even though those who differ with respect to these colours be brothers or any other kin.

Edward Gibbon, The Decline and Fall of the Roman Empire:

The support of a faction became necessary to every candidate for civil or ecclesiastical honors.

Who were the Blues and the Greens?

They were sports fans - the partisans of the blue and green chariot-racing teams.

It's less surprising if you think of the Robbers Cave experiment. Favorite-Team is us; Rival-Team is them. Nothing more is ever necessary to produce fanatic enthusiasms for Us and great hatreds of Them. People pursue their sports allegiances with all the desperate energy of two hunter-gatherer bands lined up for battle - cheering as if their very life depended on it, because fifty thousand years ago, it did.

Evolutionary psychology produces strange echoes in time, as adaptations continue to execute long after they cease to maximize fitness. Sex with condoms. Taste buds still chasing sugar and fat. Rioting basketball fans.

And so the fans of Favorite-Football-Team all praise their favorite players to the stars, and derogate the players on the Hated-Rival-Team. We are the fans and players on the Favorite-Football-Team. They are the fans and players from Hated-Rival-Team. Those are the two opposing tribes, right?

And yet the professional football players from Favorite-Team have a lot more in common with the professional football players from Rival-Team, than either has in common with the truck driver screaming cheers at the top of his lungs. The professional football players live similar lives, undergo similar training regimens, move from one team to another. They're much more likely to hang out at the expensive hotel rooms of fellow football players, than share a drink with a truck driver in his rented trailer home. Whether Favorite-Team or Rival-Team wins, it's professional football players, not truck drivers, who get the girls, the spotlights, and above all the money: professional football players are paid a hell of a lot more than truck drivers.

Why are professional football players better paid than truck drivers? Because the truck driver divides the world into Favorite-Team and Rival-Team. That's what motivates him to buy the tickets and wear the T-Shirts. The whole money-making system would fall apart if people started seeing the world in terms of Professional Football Players versus Spectators.

And I'm not even objecting to professional football. Group identification is pretty much the service provided by football players, and since that service can be provided to many people simultaneously, salaries are naturally competitive. Fans pay for tickets voluntarily, and everyone knows the score.

It would be a very different matter if your beloved professional football players held over you the power of taxation and war, prison and death.

Then it might not be a good idea to lose yourself in the delicious rush of group identification.

Back in the good ol' days, when knights were brave and peasants starved, there was little doubt that the government and the governed were distinct classes. Everyone simply took for granted that this was the Natural Order of Things.

This era did not vanish in an instantaneous flash. The Magna Carta did not challenge the obvious natural distinction between nobles and peasants - but it suggested the existence of a contract, a bargain, two sides at the table rather than one:

No Freeman shall be taken or imprisoned, or be disseised of his Freehold, or Liberties, or free Customs, or be outlawed, or exiled, or any other wise destroyed; nor will We not pass upon him, nor condemn him, but by lawful judgment of his Peers, or by the Law of the Land. We will sell to no man, we will not deny or defer to any man either Justice or Right.

England did not replace the House of Lords with the House of Commons, when the notion of an elected legislature was first being floated. They both exist, side-by-side, to this day.

The American War of Independence did not begin as a revolt against the idea of kings, but rather a revolt against one king who had overstepped his authority and violated the compact.

And then someone suggested a really wild idea...

From Decision in Philadelphia: The Constitutional Convention of 1787:

[The delegates to the Constitutional Convention] had grown up believing in a somewhat different principle of government, the idea of the social contract, which said that government was a bargain between the rulers and the ruled. The people, in essence, agreed to accept the overlordship of their kings and governors; in return, the rulers agreed to respect certain rights of the people.

But as the debate progressed, a new concept of government began more and more to be tossed around. It abandoned the whole idea of the contract between rulers and the ruled as the philosophic basis for the government. It said instead that the power resided solely in the people, they could delegate as much as they wanted to, and withdraw it as they saw fit. All members of the government, not just legislators, would represent the people. The Constitution, then, was not a bargain between the people and whoever ran the new government, but a delegation of certain powers to the new government, which the people could revise whenever they wanted.

That was the theory. But did it work in practice?

In some ways, obviously it did work. I mean, the Presidency of the United States doesn't work like the monarchies of olden times, when the crown passed from father to son, or when a queen would succeed the king her husband.

But that's not even the important question. Forget that Congresspeople on both sides of the "divide" are more likely to be lawyers than truck drivers. Forget that in training and in daily life, they have far more in common with each other than they do with a randomly selected US citizen from their own party. Forget that they are more likely to hang out at each other's expensive hotel rooms than drop by your own house. Is there a political divide - a divide of policies and interests - between Professional Politicians on the one hand, and Voters on the other?

Well, let me put it this way. Suppose that you happen to be socially liberal, fiscally conservative. Who would you vote for?

Or simplify it further: Suppose that you're a voter who prefers a smaller, less expensive government - should you vote Republican or Democratic? Or, lest I be accused of color favoritism, suppose that your voter preference is to get US troops out of Iraq. Should you vote Democratic or Republican?

One needs to be careful, at this point, to keep track of the distinction between marketing materials and historical records. I'm not asking which political party stands for the idea of smaller government - which football team has "Go go smaller government! Go go go!" as one of its cheers. (Or "Troops out of Iraq! Yay!") Rather, over the last several decades, among Republican politicians and Democratic politicians, which group of Professional Politicians shrunk the government while it was in power?

And by "shrunk" I mean "shrunk". If you're suckered into an angry, shouting fight over whether Your Politicians or Their Politicians grew the government slightly less slowly, it means you're not seeing the divide between Politicians and Voters. There isn't a grand conspiracy to expand the government, but there's an incentive for each individual politician to send pork to campaign contributors, or borrow today against tomorrow's income. And that creates a divide between the Politicians and the Voters, as a class, for reasons that have nothing to do with colors and slogans.

Imagine two football teams. The Green team's professional players shout the battle cry, "Cheaper tickets! Cheaper tickets!" as they rush into the game. The Blue team's professional players shout, "Better seating! Better seating!" as they move forward. The Green Spectators likewise cry "Cheaper tickets!" and the Blue Spectators of course cheer "Better seating!"

And yet every year the price of tickets goes up, and the seats get harder and less comfortable. The Blues win a football game, and a great explosion of "Better seating! Better seating!" rises to the heavens with great shouts of excitement and glory, and then the next year the cushions have been replaced by cold steel. The Greens kick a long-range field goal, and the Green Spectators leap up and down and hug each other screaming "Cheaper tickets! Hooray! Cheaper tickets!" and then tomorrow there's a $5 cost increase.

It's not that there's a conspiracy. No conspiracy is required. Even dishonesty is not required - it's so painful to have to lie consciously. But somehow, after the Blue Professional Football Players have won the latest game, and they're just about to install some new cushions, it occurs to them that they'd rather be at home drinking a nice cold beer. So they exchange a few furtive guilty looks, scurry home, and apologize to the Blue Spectators the next day.

As for the Blue Spectators catching on, that's not very likely. See, one of the cheers of the Green side is "Even if the Blues win, they won't install new seat cushions!" So if a Blue Spectator says, "Hey, Blue Players, we cheered real hard and you won the last game! What's up with the cold steel seats?" all the other Blue Spectators will stare aghast and say, "Why are you calling a Green cheer?" And the lonely dissenter says, "No, you don't understand, I'm not cheering for the Greens. I'm pointing out, as a fellow Spectator with an interest in better seating, that the Professional Football Players who are allegedly on the Blue Spectators' side haven't actually -"

"What do you mean?" cry the Blue Spectators. "Listen! You can hear the Players calling it now! 'Better seating!' It resounds from the rafters - how can you say our Players aren't true Blue? Do you want the Green Players to win? You - you're betraying Our Team by criticizing Our Players!"

This is what I mean by the "two-party swindle". Once a politician gets you to identify with them, they pretty much own you.

There doesn't have to be a conscious, collaborative effort by Your Politicians and Their Politicians to keep the Voters screaming at each other, so that they don't notice the increasing gap between the Voters and the Politicians. There doesn't have to be a conspiracy. It emerges from the interests of the individual politicians in getting you to identify with them instead of judging them.

The problem dates back to olden times. Commoners identifying with kings was one of the great supports of the monarchy. The commoners in France and England alike might be cold and starving. And the kings of France and England alike might be living in a palace, drinking from golden cups. But hey, the King of England is our king, right? His glory is our glory? Long live King Henry the Whatever!

But as soon as you managed to take an emotional step back, started to think of your king as a contractor - rather than cheering for him because of the country he symbolized - you started to notice that the king wasn't a very good employee.

And I dare say the Big Mess is not likely to be cleaned up, until the Republifans and Demofans realize that in many ways they have more in common with other Voters than with "their" Politicians; or, at the very least, stop enthusiastically cheering for rich lawyers because they wear certain colors, and begin judging them as employees severely derelict in their duties.

Until then, the wheel will turn, one sector rising and one sector falling, with a great tumult of lamentation and cheers - and turn again, with uninhibited cries of joy or apprehension - turn again and again, and not go anywhere.

Getting emotional over politics as though it were a sports game - identifying with one color and screaming cheers for them, while heaping abuse on the other color's fans - is a very good thing for the Professional Players' Team; not so much for Team Voters.

(This post is related to the sequence Politics is the Mind-Killer.)

Referenced by: The American System and Misleading Labels • Stop Voting For Nincompoops • Words as Hidden Inferences • Dissolving the Question • The Beauty of Settled Science • My Naturalistic Awakening • Logical or Connectionist AI?

Original with comments: The Two-Party Swindle

The American System and Misleading Labels

Eliezer Yudkowsky, 02 January 2008 07:38AM

"How many legs does a dog have, if you call a tail a leg? Four. Calling a tail a leg doesn't make it a leg."
-- Abraham Lincoln

So I was at this conference. Where one of the topics was legal rights for human-level Artificial Intelligence. And personally, I don't think this is going to be the major problem, but these are the kinds of conferences I go to. Anyway. Brad Templeton, chairman of the Electronic Frontier Foundation, was present; and he said:

"The legal status of AIs is ultimately a legislative question, and in the American system of democracy, legislative questions are decided by the Supreme Court."

Much laughter followed. We all knew it was true. (And Brad has taken a case or two to the Supreme Court, so he was speaking from experience.)

I'm not criticizing the Supreme Court. They don't always agree with me on every issue - that is not a politician's job - but reasoned cooperative discourse, compact outputs, and sheer professionalism all make the Supreme Court a far more competent legislative body than Congress.

Try to say aloud the color - not the meaning, the color - of the following letters:

RED

Now say aloud the meaning of the letters, not their color:

PURPLE

Which task felt more difficult? It's actually easier to pay attention to the labels of things than their substances.

But if you're going to be faced with several repetitions of the first task, there's a way to make it easier - just blur your eyes a little, so that you can see the color a moment before you're distracted by the meaning. Try it - defocus your eyes slightly, and then say the following colors aloud:

BLUE       ORANGE       YELLOW

If you want to know what the Supreme Court really does, you should blur your eyes so that you can't see the words "Supreme" or "Court", or all the giant signs reading "judge", "Honorable", or "judicial branch of government". Then how can you tell what they do? Well, you could follow these nine people around for a few days and observe them at work. You'll see that they dress in funny outfits and go to a building where they hear some people arguing with each other. Then they'll talk it over for a while, and issue one or two short written documents, some of which tell other people what they are or aren't allowed to do. If you were a Martian anthropologist and you had absolutely no idea that these people were supposed to be doing this or that or something else, you would probably conclude they were (among other things) making laws.

Do Representatives and Senators make laws? Well, I've met one or two Congresspeople, and I didn't see them writing any documents that tell people what to do. Maybe they do it when no one is looking? I've heard their days are pretty heavily scheduled, though.

Some laws are made by Congressional staff, but the vast majority of legislation is written by professional bureaucrats, who - if you refocus your eyes and read their labels - are part of the "executive" branch.

What if you've got a problem with a bureaucrat getting in your hair? You won't have much luck talking to the White House. But if you contact your Representative or Senator's constituent service office, they'll be happy to help you if they can. If you didn't know how the system works apart from a high school civics class, you might end up very frustrated - the people who help you deal with the executive branch of government have signs reading "legislative branch".

Your Congressperson would much rather help a little old lady deal with a lost Social Security check, than make potentially controversial laws about immigration or intellectual property. That sort of thing may please some of your constituents, but it gets others very angry at you, and voters are faster to forget a favor than a disservice. Keep making laws and you might just get yourself unelected. If you know everyone is going to cheer you for a law, go ahead and pass it; but otherwise it's safer to leave legislation like e.g. school desegregation to the Supreme Court.

What Congresspeople prefer to do is write elaborate budgets that exert detailed control over how the government spends money, so they can send more of it to their districts. That, and help their constituents with the bureaucracy, which makes friends without getting anyone mad at them. The executive branch has no time for such matters, it's busy declaring wars.

So, bearing in mind that nothing works the way it's written down on paper, let's defocus our eyes and ask about the role of the voters.

If we blur out the label over your head and look at what you do, then you go to a certain building and touch one of several names written on a computer screen. You don't choose which names will be on the screen - no, you don't. Forget the labels, remember your actual experiences. You walked into the building, you did choose which rectangle to touch, and you did not choose which little names you would see on the computer screen.

When Stephen Colbert wanted to register for a presidential run in South Carolina, the executive council of the South Carolina Democratic Party voted 13-3 to keep him off the ballot: "He clearly doesn't meet the requirements. It's a distraction and takes away from the seriousness of our primary here and takes attention from the serious candidates: Clinton, Edwards, Barack Obama and the rest."

Hey, South Carolina Democratic Party executive council, you know who ELSE might be interested in determining whether someone is, or isn't, a "serious candidate"? HOW ABOUT THE %#<!ing VOTERS?

Ahem. But the psychology of this response is quite revealing. "They want to prevent wasted votes" would be a polite way of putting it. It doesn't even seem to have occurred to them that a voter might attach their own meaning to a vote for Stephen Colbert - that a voter might have their own thoughts about whether a vote for Stephen Colbert was "wasted" or not. Nor that the voters might have a sense of ownership of their own votes, a wish to determine their use. In the psychology of politicians, politicians own voters, voters do not own politicians. South Carolina Democratic voters are a resource of the South Carolina Democratic Party, not the other way around. They don't want you to waste their votes.

(Am I the only one on the planet to whom it occurred that the South Carolina voters could decide the meaning of a vote for Stephen Colbert? Because I seriously don't remember anyone else pointing that out, at the time.)

How much power does touching a name in a rectangle give you? Well, let me put it this way:

When I blur my eyes and look at the American system of democracy, I see that the three branches of government are the executive, the legislative, the judicial, the bureaucracy, the party structure, and the media. In the next tier down are second-ranked powers, such as "the rich" so often demonized by the foolish - the upper-upper class can exert influence, but they have little in the way of direct political control. Similarly with NGOs (non-governmental organizations) such as the Electronic Frontier Foundation, think tanks, traditional special interest groups, "big corporations", lobbyists, the voters, foreign powers with a carrot or stick to offer the US, and so on.

Which is to say that political powers do make an attempt to court the voters, but not to a noticeably greater degree than they court, say, the agricultural industry. The voters' position on the issues is not without influence, but it is a degree of influence readily comparable to the collective influence of think tanks, and probably less than the collective influence of K Street lobbyists. In practice, that's how it seems to work out.

The voters do have two special powers, but both of them only manifest in cases of great emotional arousal, like a comic-book superhero who can't power up until angry. The first power is the ability to swap control of Congress (and in years divisible by 4, the Presidency) between political parties. To the extent that the two allowed political parties are similar, this will not accomplish anything. Also it's a rather crude threat, not like the fine-grained advice offered by think tanks or lobbyists. There's a difference between the power to write detailed instructions on a sheet of paper, and the power to grunt and throw a big rock.

Possibly due to a coordination problem among individual politicians, the party currently in power rarely acts scared of the voters' threat. Maybe individual politicians have an incentive to pursue their goals while the pursuing is good, since a shift in parties will not necessarily deprive them of their own seats in secure districts? Thus, we regularly see the party in power acting arrogantly until the voters get angry and the vote swings back. Then the minority-turned-majority party stops trying so hard to please the voters, and begins drinking the wine of power. That's my best guess for why the balance of voters tends to be actively restored around a 50/50 ratio.

The voters' second hidden superpower can only be used if the voters get really, really, really angry, like a comic-book hero who's just seen one of their friends killed. This is the power to overthrow the existing political structure entirely by placing a new party in charge. There are barriers to entry that keep out third parties, which prevents the ordinary turnover visible in the history of US politics before the 1850s. But these barriers wouldn't stop the voters if they got really, really mad. Even tampering with electronic voting machines won't work if the vote is 90% lopsided.

And this never-used power of the voters, strangely enough, may be the most beneficial factor in democracy - because it means that although the voters are ordinarily small potatoes in the power structure, no one dares make the voters really, really, really angry.

How much of the benefit of living in a democracy is in the small influences that voters occasionally manage to exert on the political process? And how much of that benefit is from power-wielders being too scared to act like historical kings and slaughter you on a whim?

Arguably, the chief historical improvements in living conditions have not been from voters having the influence to pass legislation which (they think) will benefit them, but, rather, from power-wielders becoming scared of doing anything too horrible to voters. Maybe one retrodiction (I haven't checked) would be that if you looked at the history of England, you would find a smooth improvement in living conditions corresponding to a gradually more plausible threat of revolution, rather than a sharp jump following the introduction of an elected legislature.

You'll notice that my first post, on the Two-Party Swindle, worried about the tendency of voters to lose themselves in emotion and identify with "their" professional politicians. If you think the chief benefit of living in a democracy is the hope of getting Your-Favorite-Legislation passed, then you might abandon yourself in adulation of a professional politician who wears the colors of Your-Favorite-Legislation. Isn't that the only way you can get Your-Favorite-Legislation, the most important thing in the world, passed?

But what if the real benefit of living in a democracy, for voters, comes from their first and second superpowers? Then by identifying with politicians, the voters become less likely to remove the politician from office. By identifying with parties, voters become less likely to swap the party out of power, or cast it from government entirely. Both identifications interfere with the plausible threat. The power-wielders can get away with doing more and worse things before you turn on them.

The feature of democracies, of allowing heated color wars between voters on particular policy issues, is not likely to account, on average, for the benefits of living in a democracy. Even if one option is genuinely better than the other, the existence of a color war implies that large efforts are being spent on tugging in either direction.

So if voters get wrapped up in color wars, identifying emotionally with "their" professional politicians and issues that put them at odds with other voters - at the expense of being less likely to get upset with "their" politician and "their" party - then shouldn't we expect the end result, on average, to be harmful to voters in general?

Coming tomorrow: My shocking, counterintuitive suggestion for a pragmatic voting policy! (Well... maybe not so counterintuitive as all that...)

Referenced by: Stop Voting For Nincompoops

Original with comments: The American System and Misleading Labels

Stop Voting For Nincompoops

Eliezer Yudkowsky, 02 January 2008 06:00PM

Followup to: The Two-Party Swindle, The American System and Misleading Labels

If evolutionary psychology could be simplified down to one sentence (which it can't), it would be: "Our instincts are adaptations that increased fitness in the ancestral environment, and we go on feeling that way regardless of whether it increases fitness today." Sex with condoms, tastes for sugar and fat, etc.

In the ancestral environment, there was no such thing as voter confidentiality. If you backed a power faction in your hunter-gatherer band, everyone knew which side you'd picked. The penalty for choosing the losing side could easily be death.

Our emotions are shaped to steer us through hunter-gatherer life, not life in the modern world. It should be no surprise, then, that when people choose political sides, they feel drawn to the faction that seems stronger, better positioned to win. Even when voting is confidential. Just as people enjoy sex, even when using contraception.

(George Orwell had a few words to say in "Raffles and Miss Blandish" about where the admiration of power can go. The danger, not of lusting for power, but just of feeling drawn to it.)

In a recent special election for California governor, the usual lock of the party structure broke down - they neglected to block that special case, and so you could get in with 65 signatures and $3500. As a result there were 135 candidates.

With 135 candidates, one might have thought there would be an opportunity for some genuine voter choice - a lower barrier to entry, which would create a chance to elect an exceptionally competent governor. However, the media immediately swung into action and decided that only a tiny fraction of these candidates would be allowed to get any publicity. Which ones? Why, the ones who already had name recognition! Those, after all, were the candidates who were likely to win, so those were the ones which the media reported on.

Amazingly, the media collectively exerted such tremendous power, in nearly perfect coordination, without deliberate intention (conspiracies are generally much less necessary than believed). They genuinely thought, I think, that they were reporting the news rather than making it. Did it even occur to them that the entire business was self-referential? Did anyone write about that aspect? With a coordinated action, the media could have chosen any not-completely-pathetic candidate to report as the "front-runner", and their reporting would thereby have been correct.

The technical term for this is Keynesian beauty contest, wherein everyone tries to vote for whoever they think most people will vote for.

If Arnold Schwarzenegger (4,206,284 votes) had been as unable to get publicity as Logan Clements (274 votes), perhaps because the media believed (in uncoordinated unison) that no action-movie hero could be taken seriously as a candidate, then Arnold Schwarzenegger would not have been a "serious candidate".

In effect, Arnold Schwarzenegger was appointed Governor of California by the media. The case is notable because usually it's the party structure that excludes candidates, and the party structure's power has a formal basis that does not require voter complicity. The power of the media to appoint Arnold Schwarzenegger governor derived strictly from voters following what someone told them was the trend. If the voters had ignored the media telling them who the front-runner was, and decided their initial pick of "serious candidates" based on, say, the answers to a questionnaire, then the media would have had no power.

Yes, this is presently around as likely as the Sun rising in the west and illuminating a Moon of green cheese. But there's this thing called the Internet now, which humanity is still figuring out how to use, and there may be another change or two on the way. Twenty years ago, if the media had decided not to report on Ron Paul, that would have been that.

Someone is bound to say, at this point, "But if you vote for a candidate with no chance of winning, you're throwing your vote away!"

"The leaders are lizards. The people hate the lizards and the lizards rule the people."
"Odd," said Arthur, "I thought you said it was a democracy."
"I did," said Ford, "It is."
"So," said Arthur, hoping he wasn't sounding ridiculously obtuse, "why don't the people get rid of the lizards?"
"It honestly doesn't occur to them," said Ford. "They've all got the vote, so they all pretty much assume that the government they've voted in more or less approximates to the government they want."
"You mean they actually vote for the lizards?"
"Oh yes," said Ford with a shrug, "of course."
"But," said Arthur, going for the big one again, "why?"
"Because if they didn't vote for a lizard," said Ford, "the wrong lizard might get in. Got any gin?"

To which the economist replies, "But you can't always jump from a Nash equilibrium to a Pareto optimum," meaning roughly, "Unless everyone else has that same idea at the same time, you'll still be throwing your vote away," or in other words, "You can make fun all you like, but if you don't vote for a lizard, the wrong lizard really might get in."

In which case, the lizards know they can rely on your vote going to one of them, and they have no incentive to treat you kindly. Most of the benefits of democracy may be from the lizards being scared enough of voters to not misbehave really really badly, rather than from the "right lizard" winning a voter-fight.

Besides, picking the better lizard is harder than it looks. In 2000, the comic Melonpool showed a character pondering, "Bush or Gore... Bush or Gore... it's like flipping a two-headed coin." Well, how were they supposed to know? In 2000, based on history, it seemed to me that the Republicans were generally less interventionist and therefore less harmful than the Democrats, so I pondered whether to vote for Bush to prevent Gore from getting in. Yet it seemed to me that the barriers to keep out third parties were a raw power grab, and that I was therefore obliged to vote for third parties wherever possible, to penalize the Republicrats for getting grabby. And so I voted Libertarian, though I don't consider myself one (at least not with a big "L"). I'm glad I didn't do the "sensible" thing. Less blood on my hands.

If we could go back in time and change our votes, and see alternate histories laid out side-by-side, it might make sense to vote for the less evil of two lizards. But in a state of ignorance - voting for candidates that abandon their stated principles like they discard used toilet paper - then it is harder to compare lizards than those enthusiastically cheering for team colors might think.

Are people who vote for Ron Paul in the Republican primary wasting their votes? I'm not asking, mind you, whether you approve of Ron Paul as a candidate. I'm asking you whether the Ron Paul voters are taking an effectless action if Ron Paul doesn't win. Ron Paul is showing what an candidate can do with the Internet, despite the party structure and the media. A competent outsider considering a presidential run in 2012 is much more likely to take a shot at it now. What exactly does a vote for Hilliani accomplish, besides telling the lizards to keep doing whatever it is they're doing?

Make them work for your vote. Vote for more of the same this year, for whatever clever-sounding reason, and next election, they'll give you more of the same. Refuse to vote for nincompoops and maybe they'll try offering you a less nincompoopy candidate, or non-nincompoops will be more likely to run for office when they see they have a chance.

Besides, if you're going to apply game theory to the situation in a shortsighted local fashion - not taking into account others thinking similarly, and not taking into account the incentives you create for later elections based on what potential future candidates see you doing today - if, I say, you think in such a strictly local fashion and call it "rational", then why vote at all, when your single vote is exceedingly unlikely to determine the winner?

Consider these two clever-sounding game-theoretical arguments side by side:

1. You should vote for the less evil of the top mainstream candidates, because your vote is unlikely to make a critical difference if you vote for a candidate that most people don't vote for.

2. You should stay home, because your vote is unlikely to make a critical difference.

It's hard to see who should accept argument #1 but refuse to accept argument #2.

I'm not going to go into the notion of collective action, Prisoner's Dilemma, Newcomblike problems, etcetera, because the last time I tried to write about this, I accidentally started to write a book. But whatever meaning you attach to voting - especially any notions of good citizenship - it's hard to see why you should vote for a lizard if you bother to vote at all.

There is an interaction here, a confluence of folly, between the evolutionary psychology of politics as a football game, and the evolutionary psychology of trying to side with the winner. The media - I am not the first to observe this - report on politics as though it is a horse race. Good feelings about a candidate are generated, not by looking over voting records, but by the media reporting excitedly: "He's pulling ahead in the third stretch!" What the media thinks we should know about candidates is that such-and-such candidate appeals to such-and-such voting faction. Since this is practically all the media report on, it feeds nothing but the instinct to get yourself on the winning side.

And then there's the lovely concept of "electability": Trying to vote for a candidate that you think other people will vote for, because you want your own color to win at any cost. You have to admire the spectacle of the media breathlessly reporting on which voting factions think that candidate X is the most "electable". Is anyone even counting the levels of recursion here?

Or consider it from yet another perspective:

There are roughly 300 million people in the United States, of whom only one can be President at any given time.

With 300 million available candidates, many of whom are not nincompoops, why does America keep electing nincompoops to political office?

Sending a message to select 1 out of 300 million possibilities requires 29 bits. So if you vote in only the general election for the Presidency, then some mysterious force narrows the election down to 2 out of 300 million possibilities - exerting 28 bits of decision power - and then you, or rather the entire voting population, exert 1 more bit of decision power. If you vote in a primary election, you may send another 2 or 3 bits worth of message.

Where do the other 25 bits of decision power come from?

You may object: "Wait a minute, not everyone in the United States is 35 years old and a natural-born citizen, so it's not 300 million possibilities."

I reply, "How do you know that a 34-year-old cannot be President?"

And you cry, "What? It's in the Constitution!"

Well, there you go: Since around half the population is under the age of 35, at least one bit of the missing decision power is exerted by 55 delegates in Philadelphia in 1787. Though the "natural-born citizen" clause comes from a letter sent by John Jay to George Washington, a suggestion that was adopted without debate by the Philadelphia Convention.

I am not necessarily advising that you go outside the box on this one. Sometimes the box is there for a good reason. But you should at least be conscious of the box's existence and origin.

Likewise, not everyone would want to be President. (But see the hidden box: In principle the option exists of enforcing Presidential service, like jury duty.) How many people would run for President if they had a serious chance at winning? Let's pretend the number is only 150,000. That accounts for another 10 bits.

Then some combination of the party structure, and the media telling complicit voters who voters are likely to vote for, is exerting on the order of 14-15 bits of power over the Presidency; while the voters only exert 3-4 bits. And actually the situation is worse than this, because the media and party structure get to move first. They can eliminate nearly all the variance along any particular dimension. So that by the time you get to choose one of four "serious" "front-running" candidates, that is, the ones approved by both the party structure and the media, you're choosing between 90.8% nincompoop and 90.6% nincompoop.

I seriously think the best thing you can do about the situation, as a voter, is stop trying to be clever. Don't try to vote for someone you don't really like, because you think your vote is more likely to make a difference that way. Don't fret about "electability". Don't try to predict and outwit other voters. Don't treat it as a horse race. Don't worry about "wasting your vote" - it always sends a message, you may as well make it a true message.

Remember that this is not the ancestral environment, and that you won't die if you aren't on the winning side. Remember that the threat that voters as a class hold against politicians as a class is more important to democracy than your fights with other voters. Forget all the "game theory" that doesn't take future incentives into account; real game theory is further-sighted, and besides, if you're going to look at it that way, you might as well stay home. When you try to be clever, you usually end up playing the Politicians' game.

Clear your mind of distractions...

And stop voting for nincompoops.

If you vote for nincompoops, for whatever clever-sounding reason, don't be surprised that out of 300 million people you get nincompoops in office.

The arguments are long, but the voting strategy they imply is simple: Stop trying to be clever, just don't vote for nincompoops.

Oh - and if you're going to vote at all, vote in the primary. That's where most of your remaining bits and remaining variance have a chance to be exerted. It's a pretty good bet that a Republicrat will be elected. The primary is your only chance to choose between Hilliani and Opaula (or whatever your poison).

If anyone tells you that voting in a party's primary commits you to voting for that party in the general election, or that a political party owns the primary and you're stealing something from them, then laugh in their faces. They've taken nearly all the decision bits, moved first in the game, and now they think they can convince you not to exercise the bits you have left?

To boil it all down to an emotional argument that isn't necessarily wrong:

Why drive out to your polling place and stand in line for half an hour or more - when your vote isn't very likely to singlehandedly determine the Presidency - and then vote for someone you don't even want?

Referenced by: Today's Inspirational Tale

Original with comments: Stop Voting For Nincompoops

Rational vs. Scientific Ev-Psych

Eliezer Yudkowsky, 04 January 2008 07:01AM

Prerequisite: Evolutionary Psychology

Years ago, before I left my parents' nest, I was standing in front of a refrigerator, looking inside. My mother approached and said, "What are you doing?" I said, "Looking for the ketchup. I don't see it."

My mother reached behind a couple of bottles and took out the ketchup.

She said, "If you don't see the ketchup, why don't you move things around and look behind them, instead of just standing and staring into the refrigerator? Do you think the ketchup is magically going to appear if you stare into the refrigerator long enough?"

And lo, the light went on over my head, and I said: "Men are hunters, so if we can't find our prey, we instinctively freeze motionless and wait for it to wander into our field of vision. Women are gatherers, so they move things around and look behind them."

Now this sort of thing is not scientifically respectable; it is called a "just-so story", after Kipling's "Just-So Stories" like "How the Camel Got His Hump". The implication being that you can make up anything you like for an evolutionary story, but the difficult thing is finding a way to prove it.

Well, fine, but I bet it's still true.

The sexual division of labor in hunter-gatherer societies - "Men hunt, women gather" - is a human universal among hunter-gatherers, like it or not; and it has experimentally tested cognitive consequences: Men are better at throwing spears; women are better at remembering where things are. So that part is pretty much nailed down - the controversial thing is saying, "This is why men stand motionless and stare into the refrigerator."

But even the Refrigerator Hypothesis is not as untestable as one might think. For a start, you have to verify that this is a cross-cultural universal - you have to check to see if men and women in China and South Africa exhibit the same stereotypical behavior as in the US. And you also have to verify that hunters who don't see prey where they expect it, do indeed freeze motionless and wait for the prey to wander across their vision.

But that doesn't prove that the same psychology is at work in a man staring into a refrigerator, does it?

Well, you could install an eye-tracker on a hunter; look for a characteristic pattern of eye movements when they're frozen waiting for prey; and use simulations or game theory to show that the gaze pattern is efficient. Then you could put an eye-tracker on a man looking into a refrigerator, and see if they show the same gaze pattern. Then you would have demonstrated a detailed correspondence which enables you to say, "He is frozen, waiting for the ketchup to come into sight."

Here is an odd thing: There are some people who will, if you just tell them the Refrigerator Hypothesis, snort and say "That's an untestable just-so story" and dismiss it out of hand; but if you start by telling them about the gaze-tracking experiment and then explain the evolutionary motivation, they will say, "Huh, that might be right." Because then, you see, you are doing Science; but in the earlier case, you are just making up a story without testing it, which is very Unscientific. We all know that Scientific hypotheses are more likely to be true than Unscientific ones.

This pattern of belief is very hard to justify from a Bayesian perspective. It is just the same hypothesis in both cases. Even if, in the second case, I announce an experimental method and my intent to actually test it, I have not yet experimented and I have not yet received any observational evidence in favor of the hypothesis. So in either case, my current estimate should equal my prior probability, from the degree to which the "just-so story" seems "just" and "so". You can't revise your beliefs based on an expected experimental success, by Conservation of Expected Evidence; if you expect the experiment to succeed, that just says that it was a rationally convincing just-so story.

People confuse the distinction between rationality and science. Science is a special kind of strong evidence, a subset of rational evidence; if I say that my socks are currently white, you have a rational reason to believe, but it is not Science, because there is no experiment people can perform to independently verify the belief.

If you have a hypothesis you have not figured out how to test with an organized, rigorous experiment, then your hypothesis is not very scientific. When you figure out how to do an experiment, and more importantly, set out to do the experiment, then your hypothesis becomes very scientific indeed. If you are judging probabilities using the affect heuristic, and you know that science is a Good Thing, then making the jump from "merely rational" to "scientific" might seem to raise the probability.

But this itself is not rational. Figuring out a way to test a belief with an organized, rigorous, repeatable experiment is certainly a Good Thing - but it should not raise the belief's rational probability in advance of the experiment succeeding! A hypothesis may become "more scientific" because you are going to test it, but it doesn't get any of the power of scientific confirmation until the experiment succeeds. To whatever degree you guess that the experiment might work - that it's likely enough to be worth performing - you must have a rational probabilistic belief in the hypothesis being true, in advance of any scientific confirmation of it.

I conclude that an evolutionary just-so story, whose predictions you cannot figure out how to test in any organized, rigorous, repeatable way, may nonetheless have a substantial rational credibility - equalling the degree to which you would expect a rigorous experiment to succeed, if you could only figure one out.

Referenced by: A Failed Just-So Story • Why Does Power Corrupt?

Original with comments: Rational vs. Scientific Ev-Psych

A Failed Just-So Story

Eliezer Yudkowsky, 05 January 2008 06:35AM

Followup to: Rational vs. Scientific Ev-Psych, The Tragedy of Group Selectionism, Evolving to Extinction

Perhaps the real reason that evolutionary "just-so stories" got a bad name is that so many attempted stories are prima facie absurdities to serious students of the field.

As an example, consider a hypothesis I've heard a few times (though I didn't manage to dig up an example). The one says: Where does religion come from? It appears to be a human universal, and to have its own emotion backing it - the emotion of religious faith. Religion often involves costly sacrifices, even in hunter-gatherer tribes - why does it persist? What selection pressure could there possibly be for religion?

So, the one concludes, religion must have evolved because it bound tribes closer together, and enabled them to defeat other tribes that didn't have religion.

This, of course, is a group selection argument - an individual sacrifice for a group benefit - and see the referenced posts if you're not familiar with the math, simulations, and observations which show that group selection arguments are extremely difficult to make work. For example, a 3% individual fitness sacrifice which doubles the fitness of the tribe will fail to rise to universality, even under unrealistically liberal assumptions, if the tribe size is as large as fifty. Tribes would need to have no more than 5 members if the individual fitness cost were 10%. You can see at a glance from the sex ratio in human births that, in humans, individual selection pressures overwhelmingly dominate group selection pressures. This is an example of what I mean by prima facie absurdity.

So why religion, then?

Well, it might just be a side effect of our ability to do things like model other minds, which enables us to conceive of disembodied minds. Faith, as an emotion, might just be co-opted hope.

But if faith is a true religious adaptation, I don't see why it's even puzzling what the selection pressure could have been.

Heretics were routinely burned alive just a few centuries ago. Or stoned to death, or executed by whatever method local fashion demands. Questioning the local gods is the notional crime for which Socrates was made to drink hemlock.

Conversely, Huckabee just won Iowa's nomination for tribal-chieftain.

Why would you need to go anywhere near the accursèd territory of group selectionism in order to provide an evolutionary explanation for religious faith? Aren't the individual selection pressures obvious?

I don't know whether to suppose that (1) people are mapping the question onto the "clash of civilizations" issue in current affairs, (2) people want to make religion out to have some kind of nicey-nice group benefit (though exterminating other tribes isn't very nice), or (3) when people get evolutionary hypotheses wrong, they just naturally tend to get it wrong by postulating group selection.

But the problem with this hypothesis is not that it's "unscientific" because no replicable experiment was proposed to test it. The problem is that the hypothesis is, as a matter of prior probability, almost certainly wrong. If you did propose a valid experiment to test it, I would expect it to turn up negative.

Referenced by: Dissolving the Question

Original with comments: A Failed Just-So Story

But There's Still A Chance, Right?

Eliezer Yudkowsky, 06 January 2008 01:56AM

Years ago, I was speaking to someone when he casually remarked that he didn't believe in evolution. And I said, "This is not the nineteenth century. When Darwin first proposed evolution, it might have been reasonable to doubt it. But this is the twenty-first century. We can read the genes. Humans and chimpanzees have 98% shared DNA. We know humans and chimps are related. It's over."

He said, "Maybe the DNA is just similar by coincidence."

I said, "The odds of that are something like two to the power of seven hundred and fifty million to one."

He said, "But there's still a chance, right?"

Now, there's a number of reasons my past self cannot claim a strict moral victory in this conversation. One reason is that I have no memory of whence I pulled that 2^(750,000,000) figure, though it's probably the right meta-order of magnitude. The other reason is that my past self didn't apply the concept of a calibrated confidence. Of all the times over the history of humanity that a human being has calculated odds of 2^(750,000,000):1 against something, they have undoubtedly been wrong more often than once in 2^(750,000,000) times. E.g. the shared genes estimate was revised to 95%, not 98% - and that may even apply only to the 30,000 known genes and not the entire genome, in which case it's the wrong meta-order of magnitude.

But I think the other guy's reply is still pretty funny.

I don't recall what I said in further response - probably something like "No" - but I remember this occasion because it brought me several insights into the laws of thought as seen by the unenlightened ones.

It first occurred to me that human intuitions were making a qualitative distinction between "No chance" and "A very tiny chance, but worth keeping track of." You can see this in the OB lottery debate, where someone said, "There's a big difference between zero chance of winning and epsilon chance of winning," and I replied, "No, there's an order-of-epsilon difference; if you doubt this, let epsilon equal one over googolplex."

The problem is that probability theory sometimes lets us calculate a chance which is, indeed, too tiny to be worth the mental space to keep track of it - but by that time, you've already calculated it. People mix up the map with the territory, so that on a gut level, tracking a symbolically described probability feels like "a chance worth keeping track of", even if the referent of the symbolic description is a number so tiny that if it was a dust speck, you couldn't see it. We can use words to describe numbers that small, but not feelings - a feeling that small doesn't exist, doesn't fire enough neurons or release enough neurotransmitters to be felt. This is why people buy lottery tickets - no one can feel the smallness of a probability that small.

But what I found even more fascinating was the qualitative distinction between "certain" and "uncertain" arguments, where if an argument is not certain, you're allowed to ignore it. Like, if the likelihood is zero, then you have to give up the belief, but if the likelihood is one over googol, you're allowed to keep it.

Now it's a free country and no one should put you in jail for illegal reasoning, but if you're going to ignore an argument that says the likelihood is one over googol, why not also ignore an argument that says the likelihood is zero? I mean, as long as you're ignoring the evidence anyway, why is it so much worse to ignore certain evidence than uncertain evidence?

I have often found, in life, that I have learned from other people's nicely blatant bad examples, duly generalized to more subtle cases. In this case, the flip lesson is that, if you can't ignore a likelihood of one over googol because you want to, you can't ignore a likelihood of 0.9 because you want to. It's all the same slippery cliff.

Consider his example if you ever you find yourself thinking, "But you can't prove me wrong." If you're going to ignore a probabilistic counterargument, why not ignore a proof, too?

Sequence: Overly Convenient Excuses

Privileging the Hypothesis

The Fallacy of Gray

Referenced by: The Fallacy of Gray • Absolute Authority • The Allais Paradox • The "Intuitions" Behind "Utilitarianism" • Dark Side Epistemology

Original with comments: But There's Still A Chance, Right?

The Fallacy of Gray

Eliezer Yudkowsky, 07 January 2008 06:24AM

Followup to: Tsuyoku Naritai, But There's Still A Chance Right?

The Sophisticate: "The world isn't black and white. No one does pure good or pure bad. It's all gray. Therefore, no one is better than anyone else."
The Zetet: "Knowing only gray, you conclude that all grays are the same shade. You mock the simplicity of the two-color view, yet you replace it with a one-color view..."
-- Marc Stiegler, David's Sling

I don't know if the Sophisticate's mistake has an official name, but I call it the Fallacy of Gray. We saw it manifested in yesterday's post - the one who believed that odds of two to the power of seven hundred and fifty millon to one, against, meant "there was still a chance". All probabilities, to him, were simply "uncertain" and that meant he was licensed to ignore them if he pleased.

"The Moon is made of green cheese" and "the Sun is made of mostly hydrogen and helium" are both uncertainties, but they are not the same uncertainty.

Everything is shades of gray, but there are shades of gray so light as to be very nearly white, and shades of gray so dark as to be very nearly black. Or even if not, we can still compare shades, and say "it is darker" or "it is lighter".

Years ago, one of the strange little formative moments in my career as a rationalist was reading this paragraph from Player of Games by Iain M. Banks, especially the sentence in bold:

"A guilty system recognizes no innocents. As with any power apparatus which thinks everybody's either for it or against it, we're against it. You would be too, if you thought about it. The very way you think places you amongst its enemies. This might not be your fault, because every society imposes some of its values on those raised within it, but the point is that some societies try to maximize that effect, and some try to minimize it. You come from one of the latter and you're being asked to explain yourself to one of the former. Prevarication will be more difficult than you might imagine; neutrality is probably impossible. You cannot choose not to have the politics you do; they are not some separate set of entities somehow detachable from the rest of your being; they are a function of your existence. I know that and they know that; you had better accept it."

Now, don't write angry comments saying that, if societies impose fewer of their values, then each succeeding generation has more work to start over from scratch. That's not what I got out of the paragraph.

What I got out of the paragraph was something which seems so obvious in retrospect that I could have conceivably picked it up in a hundred places; but something about that one paragraph made it click for me.

It was the whole notion of the Quantitative Way applied to life-problems like moral judgments and the quest for personal self-improvement. That, even if you couldn't switch something from on to off, you could still tend to increase it or decrease it.

Is this too obvious to be worth mentioning? I say it is not too obvious, for many bloggers have said of Overcoming Bias: "It is impossible, no one can completely eliminate bias." I don't care if the one is a professional economist, it is clear that they have not yet grokked the Quantitative Way as it applies to everyday life and matters like personal self-improvement. That which I cannot eliminate may be well worth reducing.

Or consider this exchange between Robin Hanson and Tyler Cowen. Robin Hanson said that he preferred to put at least 75% weight on the prescriptions of economic theory versus his intuitions: "I try to mostly just straightforwardly apply economic theory, adding little personal or cultural judgment". Tyler Cowen replied:

In my view there is no such thing as "straightforwardly applying economic theory"... theories are always applied through our personal and cultural filters and there is no other way it can be.

Yes, but you can try to minimize that effect, or you can do things that are bound to increase it. And if you try to minimize it, then in many cases I don't think it's unreasonable to call the output "straightforward" - even in economics.

"Everyone is imperfect." Mohandas Gandhi was imperfect and Joseph Stalin was imperfect, but they were not the same shade of imperfection. "Everyone is imperfect" is an excellent example of replacing a two-color view with a one-color view. If you say, "No one is perfect, but some people are less imperfect than others," you may not gain applause; but for those who strive to do better, you have held out hope. No one is perfectly imperfect, after all.

(Whenever someone says to me, "Perfectionism is bad for you," I reply: "I think it's okay to be imperfect, but not so imperfect that other people notice.")

Likewise the folly of those who say, "Every scientific paradigm imposes some of its assumptions on how it interprets experiments," and then act like they'd proven science to occupy the same level with witchdoctoring. Every worldview imposes some of its structure on its observations, but the point is that there are worldviews which try to minimize that imposition, and worldviews which glory in it. There is no white, but there are shades of gray that are far lighter than others, and it is folly to treat them as if they were all on the same level.

If the moon has orbited the Earth these past few billion years, if you have seen it in the sky these last years, and you expect to see it in its appointed place and phase tomorrow, then that is not a certainty. And if you expect an invisible dragon to heal your daughter of cancer, that too is not a certainty. But they are rather different degrees of uncertainty - this business of expecting things to happen yet again in the same way you have previously predicted to twelve decimal places, versus expecting something to happen that violates the order previously observed. Calling them both "faith" seems a little too un-narrow.

It's a most peculiar psychology - this business of "Science is based on faith too, so there!" Typically this is said by people who claim that faith is a good thing. Then why do they say "Science is based on faith too!" in that angry-triumphal tone, rather than as a compliment? And a rather dangerous compliment to give, one would think, from their perspective. If science is based on 'faith', then science is of the same kind as religion - directly comparable. If science is a religion, it is the religion that heals the sick and reveals the secrets of the stars. It would make sense to say, "The priests of science can blatantly, publicly, verifiably walk on the Moon as a faith-based miracle, and your priests' faith can't do the same." Are you sure you wish to go there, oh faithist? Perhaps, on further reflection, you would prefer to retract this whole business of "Science is a religion too!"

There's a strange dynamic here: You try to purify your shade of gray, and you get it to a point where it's pretty light-toned, and someone stands up and says in a deeply offended tone, "But it's not white! It's gray!" It's one thing when someone says, "This isn't as light as you think, because of specific problems X, Y, and Z." It's a different matter when someone says angrily "It's not white! It's gray!" without pointing out any specific dark spots.

In this case, I begin to suspect psychology that is more imperfect than usual - that someone may have made a devil's bargain with their own mistakes, and now refuses to hear of any possibility of improvement. When someone finds an excuse not to try to do better, they often refuse to concede that anyone else can try to do better, and every mode of improvement is thereafter their enemy, and every claim that it is possible to move forward is an offense against them. And so they say in one breath proudly, "I'm glad to be gray," and in the next breath angrily, "And you're gray too!"

If there is no black and white, there is yet lighter and darker, and not all grays are the same.

Addendum: G points us to Asimov's The Relativity of Wrong: "When people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together."

This post is part of the sequence How To Actually Change Your Mind.

Sequence: Overly Convenient Excuses

But There's Still A Chance, Right?

Absolute Authority

Referenced by: Absolute Authority • Expecting Beauty • Trust in Math • Trust in Bayes • Newcomb's Problem and Regret of Rationality • The Argument from Common Usage • Qualitatively Confused • Zombies! Zombies? • Where Recursive Justification Hits Bottom • Rationalist Fiction

Original with comments: The Fallacy of Gray

Absolute Authority

Eliezer Yudkowsky, 08 January 2008 03:33AM

Followup to: But There's Still A Chance Right?, The Fallacy of Gray

The one comes to you and loftily says: "Science doesn't really know anything. All you have are theories - you can't know for certain that you're right. You scientists changed your minds about how gravity works - who's to say that tomorrow you won't change your minds about evolution?"

Behold the abyssal cultural gap. If you think you can cross it in a few sentences, you are bound to be sorely disappointed.

In the world of the unenlightened ones, there is authority and un-authority. What can be trusted, can be trusted; what cannot be trusted, you may as well throw away. There are good sources of information and bad sources of information. If scientists have changed their stories ever in their history, then science cannot be a true Authority, and can never again be trusted - like a witness caught in a contradiction, or like an employee found stealing from the till.

Plus, the one takes for granted that a proponent of an idea is expected to defend it against every possible counterargument and confess nothing. All claims are discounted accordingly. If even the proponent of science admits that science is less than perfect, why, it must be pretty much worthless.

When someone has lived their life accustomed to certainty, you can't just say to them, "Science is probabilistic, just like all other knowledge." They will accept the first half of the statement as a confession of guilt; and dismiss the second half as a flailing attempt to accuse everyone else to avoid judgment.

You have admitted you are not trustworthy - so begone, Science, and trouble us no more!

One obvious source for this pattern of thought is religion, where the scriptures are alleged to come from God; therefore to confess any flaw in them would destroy their authority utterly; so any trace of doubt is a sin, and claiming certainty is mandatory whether you're certain or not.

But I suspect that the traditional school regimen also has something to do with it. The teacher tells you certain things, and you have to believe them, and you have to recite them back on the test. But when a student makes a suggestion in class, you don't have to go along with it - you're free to agree or disagree (it seems) and no one will punish you.

This experience, I fear, maps the domain of belief onto the social domains of authority, of command, of law. In the social domain, there is a qualitative difference between absolute laws and nonabsolute laws, between commands and suggestions, between authorities and unauthorities. There seems to be strict knowledge and unstrict knowledge, like a strict regulation and an unstrict regulation. Strict authorities must be yielded to, while unstrict suggestions can be obeyed or discarded as a matter of personal preference. And Science, since it confesses itself to have a possibility of error, must belong in the second class.

(I note in passing that I see a certain similarity to they who think that if you don't get an Authoritative probability written on a piece of paper from the teacher in class, or handed down from some similar Unarguable Source, then your uncertainty is not a matter for Bayesian probability theory. Someone might - gasp! - argue with your estimate of the prior probability. It thus seems to the not-fully-enlightened ones that Bayesian priors belong to the class of beliefs proposed by students, and not the class of beliefs commanded you by teachers - it is not proper knowledge.)

The abyssal cultural gap between the Authoritative Way and the Quantitative Way is rather annoying to those of us staring across it from the rationalist side. Here is someone who believes they have knowledge more reliable than science's mere probabilistic guesses - such as the guess that the moon will rise in its appointed place and phase tomorrow, just like it has every observed night since the invention of astronomical record-keeping, and just as predicted by physical theories whose previous predictions have been successfully confirmed to fourteen decimal places. And what is this knowledge that the unenlightened ones set above ours, and why? It's probably some musty old scroll that has been contradicted eleventeen ways from Sunday, and from Monday, and from every day of the week. Yet this is more reliable than Science (they say) because it never admits to error, never changes its mind, no matter how often it is contradicted. They toss around the word "certainty" like a tennis ball, using it as lightly as a feather - while scientists are weighed down by dutiful doubt, struggling to achieve even a modicum of probability. "I'm perfect," they say without a care in the world, "I must be so far above you, who must still struggle to improve yourselves."

There is nothing simple you can say to them - no fast crushing rebuttal. By thinking carefully, you may be able to win over the audience, if this is a public debate. Unfortunately you cannot just blurt out, "Foolish mortal, the Quantitative Way is beyond your comprehension, and the beliefs you lightly name 'certain' are less assured than the least of our mighty hypotheses." It's a difference of life-gestalt that isn't easy to describe in words at all, let alone quickly.

What might you try, rhetorically, in front of an audience? Hard to say... maybe:

- "The power of science comes from having the ability to change our minds and admit we're wrong. If you've never admitted you're wrong, it doesn't mean you've made fewer mistakes."

- "Anyone can say they're absolutely certain. It's a bit harder to never, ever make any mistakes. Scientists understand the difference, so they don't say they're absolutely certain. That's all. It doesn't mean that they have any specific reason to doubt a theory - absolutely every scrap of evidence can be going the same way, all the stars and planets lined up like dominos in support of a single hypothesis, and the scientists still won't say they're absolutely sure, because they've just got higher standards. It doesn't mean scientists are less entitled to certainty than, say, the politicians who always seem so sure of everything."

- "Scientists don't use the phrase 'not absolutely certain' the way you're used to from regular conversation. I mean, suppose you went to the doctor, and got a blood test, and the doctor came back and said, 'We ran some tests, and it's not absolutely certain that you're not made out of cheese, and there's a non-zero chance that twenty fairies made out of sentient chocolate are singing the 'I love you' song from Barney inside your lower intestine.' Run for the hills, your doctor needs a doctor. When a scientist says the same thing, it means that he thinks the probability is so tiny that you couldn't see it with an electron microscope, but he's willing to see the evidence in the extremely unlikely event that you have it."

- "Would you be willing to change your mind about the things you call 'certain' if you saw enough evidence? I mean, suppose that God himself descended from the clouds and told you that your whole religion was true except for the Virgin Birth. If that would change your mind, you can't say you're absolutely certain of the Virgin Birth. For technical reasons of probability theory, if it's theoretically possible for you to change your mind about something, it can't have a probability exactly equal to one. The uncertainty might be smaller than a dust speck, but it has to be there. And if you wouldn't change your mind even if God told you otherwise, then you have a problem with refusing to admit you're wrong that transcends anything a mortal like me can say to you, I guess."

But, in a way, the more interesting question is what you say to someone not in front of an audience. How do you begin the long process of teaching someone to live in a universe without certainty?

I think the first, beginning step should be understanding that you can live without certainty - that if, hypothetically speaking, you couldn't be certain of anything, it would not deprive you of the ability to make moral or factual distinctions. To paraphrase Lois Bujold, "Don't push harder, lower the resistance."

One of the common defenses of Absolute Authority is something I call "The Argument From The Argument From Gray", which runs like this:

- Moral relativists say:

- The world isn't black and white, therefore:

- Everything is gray, therefore:

- No one is better than anyone else, therefore:

- I can do whatever I want and you can't stop me bwahahaha.

- But we've got to be able to stop people from committing murder.

- Therefore there has to be some way of being absolutely certain, or the moral relativists win.

Reversed stupidity is not intelligence. You can't arrive at a correct answer by reversing every single line of an argument that ends with a bad conclusion - it gives the fool too much detailed control over you. Every single line must be correct for a mathematical argument to carry. And it doesn't follow, from the fact that moral relativists say "The world isn't black and white", that this is false, any more than it follows from Stalin's belief that 2 + 2 = 4 that "2 + 2 = 4" is false. The error (and it only takes one) is in the leap from the two-color view to the single-color view, that all grays are the same shade.

It would concede far too much (indeed, concede the whole argument) to agree with the premise that you need absolute knowledge of absolutely good options and absolutely evil options in order to be moral. You can have uncertain knowledge of relatively better and relatively worse options, and still choose. It should be routine, in fact, not something to get all dramatic about.

I mean, yes, if you have to choose between two alternatives A and B, and you somehow succeed in establishing knowably certain well-calibrated 100% confidence that A is absolutely and entirely desirable and that B is the sum of everything evil and disgusting, then this is a sufficient condition for choosing A over B. It is not a necessary condition.

Oh, and: Logical fallacy: Appeal to consequences of belief.

Let's see, what else do they need to know? Well, there's the entire rationalist culture which says that doubt, questioning, and confession of error are not terrible shameful things.

There's the whole notion of gaining information by looking at things, rather than being proselytized. When you look at things harder, sometimes you find out that they're different from what you thought they were at first glance; but it doesn't mean that Nature lied to you, or that you should give up on seeing.

Then there's the concept of a calibrated confidence - that "probability" isn't the same concept as the little progress bar in your head that measures your emotional commitment to an idea. It's more like a measure of how often, pragmatically, in real life, people in a certain state of belief say things that are actually true. If you take one hundred people and ask them to list one hundred statements of which they are "absolutely certain", how many will be correct? Not one hundred.

If anything, the statements that people are really fanatic about are far less likely to be correct than statements like "the Sun is larger than the Moon" that seem too obvious to get excited about. For every statement you can find of which someone is "absolutely certain", you can probably find someone "absolutely certain" of its opposite, because such fanatic professions of belief do not arise in the absence of opposition. So the little progress bar in people's heads that measures their emotional commitment to a belief does not translate well into a calibrated confidence - it doesn't even behave monotonically.

As for "absolute certainty" - well, if you say that something is 99.9999% probable, it means you think you could make one million equally strong independent statements, one after the other, over the course of a solid year or so, and be wrong, on average, around once. This is incredible enough. (It's amazing to realize we can actually get that level of confidence for "Thou shalt not win the lottery.") So let us say nothing of probability 1.0. Once you realize you don't need probabilities of 1.0 to get along in life, you'll realize how absolutely ridiculous it is to think you could ever get to 1.0 with a human brain. A probability of 1.0 isn't just certainty, it's infinite certainty.

In fact, it seems to me that to prevent public misunderstanding, maybe scientists should go around saying "We are not INFINITELY certain" rather than "We are not certain". For the latter case, in ordinary discourse, suggests you know some specific reason for doubt.

Sequence: Overly Convenient Excuses

The Fallacy of Gray

How to Convince Me That 2 + 2 = 3

Referenced by: Infinite Certainty • 0 And 1 Are Not Probabilities • Zut Allais! • The Parable of Hemlock • The Argument from Common Usage • Perpetual Motion Beliefs • The Generalized Anti-Zombie Principle • GAZP vs. GLUT • Can You Prove Two Particles Are Identical? • Science Isn't Strict Enough • Crisis of Faith • Dark Side Epistemology • The Nature of Logic • That Crisis thing seems pretty useful • You're Entitled to Arguments, But Not (That Particular) Proof

Original with comments: Absolute Authority

Infinite Certainty

Eliezer Yudkowsky, 09 January 2008 06:49AM

Followup to: How To Convince Me That 2 + 2 = 3, Absolute Authority

In Absolute Authority, I argued that you don't need infinite certainty: "If you have to choose between two alternatives A and B, and you somehow succeed in establishing knowably certain well-calibrated 100% confidence that A is absolutely and entirely desirable and that B is the sum of everything evil and disgusting, then this is a sufficient condition for choosing A over B. It is not a necessary condition... You can have uncertain knowledge of relatively better and relatively worse options, and still choose. It should be routine, in fact."

However, might there not be some propositions in which we are entitled to infinite confidence? What about the proposition that 2 + 2 = 4?

We must distinguish between the the map and the territory. Given the seeming absolute stability and universality of physical laws, it's possible that never, in the whole history of the universe, has any particle exceeded the local lightspeed limit. That is, the lightspeed limit may be, not just true 99% of the time, or 99.9999% of the time, or (1 - 1/googolplex) of the time, but simply always and absolutely true.

But whether we can ever have absolute confidence in the lightspeed limit is a whole 'nother question. The map is not the territory.

It may be entirely and wholly true that a student plagiarized their assignment, but whether you have any knowledge of this fact at all - let alone absolute confidence in the belief - is a separate issue. If you flip a coin and then don't look at it, it may be completely true that the coin is showing heads, and you may be completely unsure of whether the coin is showing heads or tails. A degree of uncertainty is not the same as a degree of truth or a frequency of occurrence.

The same holds for mathematical truths. It's questionable whether the statement "2 + 2 = 4" or "In Peano arithmetic, SS0 + SS0 = SSSS0" can be said to be true in any purely abstract sense, apart from physical systems that seem to behave in ways similar to the Peano axioms. Having said this, I will charge right ahead and guess that, in whatever sense "2 + 2 = 4" is true at all, it is always and precisely true, not just roughly true ("2 + 2 actually equals 4.0000004") or true 999,999,999,999 times out of 1,000,000,000,000.

I'm not totally sure what "true" should mean in this case, but I stand by my guess. The credibility of "2 + 2 = 4 is always true" far exceeds the credibility of any particular philosophical position on what "true", "always", or "is" means in the statement above.

This doesn't mean, though, that I have absolute confidence that 2 + 2 = 4. See the previous discussion on how to convince me that 2 + 2 = 3, which could be done using much the same sort of evidence that convinced me that 2 + 2 = 4 in the first place. I could have hallucinated all that previous evidence, or I could be misremembering it. In the annals of neurology there are stranger brain dysfunctions than this.

So if we attach some probability to the statement "2 + 2 = 4", then what should the probability be? What you seek to attain in a case like this is good calibration - statements to which you assign "99% probability" come true 99 times out of 100. This is actually a hell of a lot more difficult than you might think. Take a hundred people, and ask each of them to make ten statements of which they are "99% confident". Of the 1000 statements, do you think that around 10 will be wrong?

I am not going to discuss the actual experiments that have been done on calibration - you can find them in my book chapter - because I've seen that when I blurt this out to people without proper preparation, they thereafter use it as a Fully General Counterargument, which somehow leaps to mind whenever they have to discount the confidence of someone whose opinion they dislike, and fails to be available when they consider their own opinions. So I try not to talk about the experiments on calibration except as part of a structured presentation of rationality that includes warnings against motivated skepticism.

But the observed calibration of human beings who say they are "99% confident" is not 99% accuracy.

Suppose you say that you're 99.99% confident that 2 + 2 = 4. Then you have just asserted that you could make 10,000 independent statements, in which you repose equal confidence, and be wrong, on average, around once. Maybe for 2 + 2 = 4 this extraordinary degree of confidence would be possible: "2 + 2 = 4" extremely simple, and mathematical as well as empirical, and widely believed socially (not with passionate affirmation but just quietly taken for granted). So maybe you really could get up to 99.99% confidence on this one.

I don't think you could get up to 99.99% confidence for assertions like "53 is a prime number". Yes, it seems likely, but by the time you tried to set up protocols that would let you assert 10,000 independent statements of this sort - that is, not just a set of statements about prime numbers, but a new protocol each time - you would fail more than once. Peter de Blanc has an amusing anecdote on this point, which he is welcome to retell in the comments.

Yet the map is not the territory: if I say that I am 99% confident that 2 + 2 = 4, it doesn't mean that I think "2 + 2 = 4" is true to within 99% precision, or that "2 + 2 = 4" is true 99 times out of 100. The proposition in which I repose my confidence is the proposition that "2 + 2 = 4 is always and exactly true", not the proposition "2 + 2 = 4 is mostly and usually true".

As for the notion that you could get up to 100% confidence in a mathematical proposition - well, really now! If you say 99.9999% confidence, you're implying that you could make one million equally fraught statements, one after the other, and be wrong, on average, about once. That's around a solid year's worth of talking, if you can make one assertion every 20 seconds and you talk for 16 hours a day.

Assert 99.9999999999% confidence, and you're taking it up to a trillion. Now you're going to talk for a hundred human lifetimes, and not be wrong even once?

Assert a confidence of (1 - 1/googolplex) and your ego far exceeds that of mental patients who think they're God.

And a googolplex is a lot smaller than even relatively small inconceivably huge numbers like 3^^^3.

But even a confidence of (1 - 1/3^^^3) isn't all that much closer to PROBABILITY 1 than being 90% sure of something.

If all else fails, the hypothetical Dark Lords of the Matrix, who are right now tampering with your brain's credibility assessment of this very sentence, will bar the path and defend us from the scourge of infinite certainty.

Am I absolutely sure of that?

Why, of course not.

As Rafal Smigrodski once said:

"I would say you should be able to assign a less than 1 certainty level to the mathematical concepts which are necessary to derive Bayes' rule itself, and still practically use it. I am not totally sure I have to be always unsure. Maybe I could be legitimately sure about something. But once I assign a probability of 1 to a proposition, I can never undo it. No matter what I see or learn, I have to reject everything that disagrees with the axiom. I don't like the idea of not being able to change my mind, ever."

Sequence: Overly Convenient Excuses

How to Convince Me That 2 + 2 = 3

0 And 1 Are Not Probabilities

Referenced by: 0 And 1 Are Not Probabilities • Expecting Beauty • The Parable of Hemlock • Perpetual Motion Beliefs • Bell's Theorem: No EPR "Reality" • Fundamental Doubts • Trying to Try • Shut up and do the impossible!

Original with comments: Infinite Certainty

0 And 1 Are Not Probabilities

Eliezer Yudkowsky, 10 January 2008 06:58AM

Followup to: Infinite Certainty

1, 2, and 3 are all integers, and so is -4. If you keep counting up, or keep counting down, you're bound to encounter a whole lot more integers. You will not, however, encounter anything called "positive infinity" or "negative infinity", so these are not integers.

Positive and negative infinity are not integers, but rather special symbols for talking about the behavior of integers. People sometimes say something like, "5 + infinity = infinity", because if you start at 5 and keep counting up without ever stopping, you'll get higher and higher numbers without limit. But it doesn't follow from this that "infinity - infinity = 5". You can't count up from 0 without ever stopping, and then count down without ever stopping, and then find yourself at 5 when you're done.

From this we can see that infinity is not only not-an-integer, it doesn't even behave like an integer. If you unwisely try to mix up infinities with integers, you'll need all sorts of special new inconsistent-seeming behaviors which you don't need for 1, 2, 3 and other actual integers.

Even though infinity isn't an integer, you don't have to worry about being left at a loss for numbers. Although people have seen five sheep, millions of grains of sand, and septillions of atoms, no one has ever counted an infinity of anything. The same with continuous quantities - people have measured dust specks a millimeter across, animals a meter across, cities kilometers across, and galaxies thousands of lightyears across, but no one has ever measured anything an infinity across. In the real world, you don't need a whole lot of infinity.

(I should note for the more sophisticated readers in the audience that they do not need to write me with elaborate explanations of, say, the difference between ordinal numbers and cardinal numbers. Yes, I possess various advanced set-theoretic definitions of infinity, but I don't see a good use for them in probability theory. See below.)

In the usual way of writing probabilities, probabilities are between 0 and 1. A coin might have a probability of 0.5 of coming up tails, or the weatherman might assign probability 0.9 to rain tomorrow.

This isn't the only way of writing probabilities, though. For example, you can transform probabilities into odds via the transformation O = (P / (1 - P)). So a probability of 50% would go to odds of 0.5/0.5 or 1, usually written 1:1, while a probability of 0.9 would go to odds of 0.9/0.1 or 9, usually written 9:1. To take odds back to probabilities you use P = (O / (1 + O)), and this is perfectly reversible, so the transformation is an isomorphism - a two-way reversible mapping. Thus, probabilities and odds are isomorphic, and you can use one or the other according to convenience.

For example, it's more convenient to use odds when you're doing Bayesian updates. Let's say that I roll a six-sided die: If any face except 1 comes up, there's an 10% chance of hearing a bell, but if the face 1 comes up, there's a 20% chance of hearing the bell. Now I roll the die, and hear a bell. What are the odds that the face showing is 1? Well, the prior odds are 1:5 (corresponding to the real number 1/5 = 0.20) and the likelihood ratio is 0.2:0.1 (corresponding to the real number 2) and I can just multiply these two together to get the posterior odds 2:5 (corresponding to the real number 2/5 or 0.40). Then I convert back into a probability, if I like, and get (0.4 / 1.4) = 2/7 = ~29%.

So odds are more manageable for Bayesian updates - if you use probabilities, you've got to deploy Bayes's Theorem in its complicated version. But probabilities are more convenient for answering questions like "If I roll a six-sided die, what's the chance of seeing a number from 1 to 4?" You can add up the probabilities of 1/6 for each side and get 4/6, but you can't add up the odds ratios of 0.2 for each side and get an odds ratio of 0.8.

Why am I saying all this? To show that "odd ratios" are just as legitimate a way of mapping uncertainties onto real numbers as "probabilities". Odds ratios are more convenient for some operations, probabilities are more convenient for others. A famous proof called Cox's Theorem (plus various extensions and refinements thereof) shows that all ways of representing uncertainties that obey some reasonable-sounding constraints, end up isomorphic to each other.

Why does it matter that odds ratios are just as legitimate as probabilities? Probabilities as ordinarily written are between 0 and 1, and both 0 and 1 look like they ought to be readily reachable quantities - it's easy to see 1 zebra or 0 unicorns. But when you transform probabilities onto odds ratios, 0 goes to 0, but 1 goes to positive infinity. Now absolute truth doesn't look like it should be so easy to reach.

A representation that makes it even simpler to do Bayesian updates is the log odds - this is how E. T. Jaynes recommended thinking about probabilities. For example, let's say that the prior probability of a proposition is 0.0001 - this corresponds to a log odds of around -40 decibels. Then you see evidence that seems 100 times more likely if the proposition is true than if it is false. This is 20 decibels of evidence. So the posterior odds are around -40 db + 20 db = -20 db, that is, the posterior probability is ~0.01.

When you transform probabilities to log odds, 0 goes onto negative infinity and 1 goes onto positive infinity. Now both infinite certainty and infinite improbability seem a bit more out-of-reach.

In probabilities, 0.9999 and 0.99999 seem to be only 0.00009 apart, so that 0.502 is much further away from 0.503 than 0.9999 is from 0.99999. To get to probability 1 from probability 0.99999, it seems like you should need to travel a distance of merely 0.00001.

But when you transform to odds ratios, 0.502 and .503 go to 1.008 and 1.012, and 0.9999 and 0.99999 go to 9,999 and 99,999. And when you transform to log odds, 0.502 and 0.503 go to 0.03 decibels and 0.05 decibels, but 0.9999 and 0.99999 go to 40 decibels and 50 decibels.

When you work in log odds, the distance between any two degrees of uncertainty equals the amount of evidence you would need to go from one to the other. That is, the log odds gives us a natural measure of spacing among degrees of confidence.

Using the log odds exposes the fact that reaching infinite certainty requires infinitely strong evidence, just as infinite absurdity requires infinitely strong counterevidence.

Furthermore, all sorts of standard theorems in probability have special cases if you try to plug 1s or 0s into them - like what happens if you try to do a Bayesian update on an observation to which you assigned probability 0.

So I propose that it makes sense to say that 1 and 0 are not in the probabilities; just as negative and positive infinity, which do not obey the field axioms, are not in the real numbers.

The main reason this would upset probability theorists is that we would need to rederive theorems previously obtained by assuming that we can marginalize over a joint probability by adding up all the pieces and having them sum to 1.

However, in the real world, when you roll a die, it doesn't literally have infinite certainty of coming up some number between 1 and 6. The die might land on its edge; or get struck by a meteor; or the Dark Lords of the Matrix might reach in and write "37" on one side.

If you made a magical symbol to stand for "all possibilities I haven't considered", then you could marginalize over the events including this magical symbol, and arrive at a magical symbol "T" that stands for infinite certainty.

But I would rather ask whether there's some way to derive a theorem without using magic symbols with special behaviors. That would be more elegant. Just as there are mathematicians who refuse to believe in double negation or infinite sets, I would like to be a probability theorist who doesn't believe in absolute certainty.

PS: Here's Peter de Blanc's "mathematical certainty" anecdote. (I told him not to do it again.)

Sequence: Overly Convenient Excuses

Infinite Certainty

Referenced by: Decoherence is Falsifiable and Testable

Original with comments: 0 And 1 Are Not Probabilities

Beautiful Math

Eliezer Yudkowsky, 10 January 2008 10:43PM

Consider the sequence {1, 4, 9, 16, 25, ...} You recognize these as the square numbers, the sequence Ak = k^2. Suppose you did not recognize this sequence at a first glance. Is there any way you could predict the next item in the sequence? Yes: You could take the first differences, and end up with:

{4 - 1, 9 - 4, 16 - 9, 25 - 16, ...} = {3, 5, 7, 9, ...}

And if you don't recognize these as successive odd numbers, you are still not defeated; if you produce a table of the second differences, you will find:

{5 - 3, 7 - 5, 9 - 7, ...} = {2, 2, 2, ...}

If you cannot recognize this as the number 2 repeating, then you're hopeless.

But if you predict that the next second difference is also 2, then you can see the next first difference must be 11, and the next item in the original sequence must be 36 - which, you soon find out, is correct.

Dig down far enough, and you discover hidden order, underlying structure, stable relations beneath changing surfaces.

The original sequence was generated by squaring successive numbers - yet we predicted it using what seems like a wholly different method, one that we could in principle use without ever realizing we were generating the squares. Can you prove the two methods are always equivalent? - for thus far we have not proven this, but only ventured an induction. Can you simplify the proof so that you can you see it at a glance? - as Polya was fond of asking.

This is a very simple example by modern standards, but it is a very simple example of the sort of thing that mathematicians spend their whole lives looking for.

The joy of mathematics is inventing mathematical objects, and then noticing that the mathematical objects that you just created have all sorts of wonderful properties that you never intentionally built into them. It is like building a toaster and then realizing that your invention also, for some unexplained reason, acts as a rocket jetpack and MP3 player.

Numbers, according to our best guess at history, have been invented and reinvented over the course of time. (Apparently some artifacts from 30,000 BC have marks cut that look suspiciously like tally marks.) But I doubt that a single one of the human beings who invented counting visualized the employment they would provide to generations of mathematicians. Or the excitement that would someday surround Fermat's Last Theorem, or the factoring problem in RSA cryptography... and yet these are as implicit in the definition of the natural numbers, as are the first and second difference tables implicit in the sequence of squares.

This is what creates the impression of a mathematical universe that is "out there" in Platonia, a universe which humans are exploring rather than creating. Our definitions teleport us to various locations in Platonia, but we don't create the surrounding environment. It seems this way, at least, because we don't remember creating all the wonderful things we find. The inventors of the natural numbers teleported to Countingland, but did not create it, and later mathematicians spent centuries exploring Countingland and discovering all sorts of things no one in 30,000 BC could begin to imagine.

To say that human beings "invented numbers" - or invented the structure implicit in numbers - seems like claiming that Neil Armstrong hand-crafted the Moon. The universe existed before there were any sentient beings to observe it, which implies that physics preceded physicists. This is a puzzle, I know; but if you claim the physicists came first, it is even more confusing because instantiating a physicist takes quite a lot of physics. Physics involves math, so math - or at least that portion of math which is contained in physics - must have preceded mathematicians. Otherwise, there would have no structured universe running long enough for innumerate organisms to evolve for the billions of years required to produce mathematicians.

The amazing thing is that math is a game without a designer, and yet it is eminently playable.

Oh, and to prove that the pattern in the difference tables always holds:

(k + 1)2 = k^2 + (2k + 1)

As for seeing it at a glance:

Think the square problem is too trivial to be worth your attention? Think there's nothing amazing about the tables of first and second differences? Think it's so obviously implicit in the squares as to not count as a separate discovery? Then consider the cubes:

1, 8, 27, 64...

Now, without calculating it directly, and without doing any algebra, can you see at a glance what the cubes' third differences must be?

And of course, when you know what the cubes' third difference is, you will realize that it could not possibly have been anything else...

Referenced by: Expecting Beauty • Is Reality Ugly? • Beautiful Probability

Original with comments: Beautiful Math

Expecting Beauty

Eliezer Yudkowsky, 12 January 2008 03:00AM

Followup to: Beautiful Math

If you looked at the sequence {1, 4, 9, 16, 25, ...} and didn't recognize the square numbers, you might still essay an excellent-seeming prediction of future items in the sequence by noticing that the table of first differences is {3, 5, 7, 9, ...}. Indeed, your prediction would be perfect, though you have no way of knowing this without peeking at the generator. The correspondence can be shown algebraically or even geometrically (see yesterday's post). It's really rather elegant.

Whatever people praise, they tend to praise too much; and there are skeptics who think that the pursuit of elegance is like unto a disease, which produces neat mathematics in opposition to the messiness of the real world. "You got lucky," they say, "and you won't always be lucky. If you expect that kind of elegance, you'll distort the world to match your expectations - chop off all the parts of Life that don't fit into your nice little pattern."

I mean, suppose Life hands you the sequence {1, 8, 27, 64, 125, ...}. When you take the first differences, you get {7, 19, 37, 61, ...}. All these numbers have in common is that they're primes, and they aren't even sequential primes. Clearly, there isn't the neat order here that we saw in the squares.

You might try to impose order, by insisting that the first differences must be evenly spaced, and any deviations are experimental errors - or better yet, we just won't think about them. "You will say," says the skeptic, "that 'The first differences are spaced around 20 apart and land on prime numbers, so that the next difference is probably 83, which makes the next number 208.' But reality comes back and says 216."

Serves you right, expecting neatness and elegance when there isn't any there. You were too addicted to absolutes; you had too much need for closure. Behold the perils of - gasp! - DUN DUN DUN - reductionism.

You can guess, from the example I chose, that I don't think this is the best way to look at the problem. Because, in the example I chose, it's not that no order exists, but that you have to look a little deeper to find it. The sequence {7, 19, 37, 61, ...} doesn't leap out at you - you might not recognize it, if you met it on the street - but take the second differences and you find {12, 18, 24, ...}. Take the third differences and you find {6, 6, ...}.

You had to dig deeper to find the stable level, but it was still there - in the example I chose.

Someone who grasped too quickly at order, who demanded closure right now, who forced the pattern, might never find the stable level. If you tweak the table of first differences to make them "more even", fit your own conception of aesthetics before you found the math's own rhythm, then the second differences and third differences will come out wrong. Maybe you won't even bother to take the second differences and third differences. Since, once you've forced the first differences to conform to your own sense of aesthetics, you'll be happy - or you'll insist in a loud voice that you're happy.

None of this says a word against - gasp! - reductionism. The order is there, it's just better-hidden. Is the moral of the tale (as I told it) to forsake the search for beauty? Is the moral to take pride in the glorious cosmopolitan sophistication of confessing, "It is ugly"? No, the moral is to reduce at the right time, to wait for an opening before you slice, to not prematurely terminate the search for beauty. So long as you can refuse to see beauty that isn't there, you have already taken the needful precaution if it all turns out ugly.

But doesn't it take - gasp! - faith to search for a beauty you haven't found yet?

As I recently remarked, if you say, "Many times I have witnessed the turning of the seasons, and tomorrow I expect the Sun will rise in its appointed place in the east," then that is not a certainty. And if you say, "I expect a purple polka-dot fairy to come out of my nose and give me a bag of money," that is not a certainty. But they are not the same shade of uncertainty, and it seems insufficiently narrow to call them both "faith".

Looking for mathematical beauty you haven't found yet, is not so sure as expecting the Sun to rise in the east. But neither does it seem like the same shade of uncertainty as expecting a purple polka-dot fairy - not after you ponder the last fifty-seven thousand cases where humanity found hidden order.

And yet in mathematics the premises and axioms are closed systems - can we expect the messy real world to reveal hidden beauty? Tune in next time on Overcoming Bias to find out!

Referenced by: Is Reality Ugly? • Beautiful Probability • Trust in Math

Original with comments: Expecting Beauty

Is Reality Ugly?

Eliezer Yudkowsky, 12 January 2008 10:26PM

Followup to: Beautiful Math, Expecting Beauty

Yesterday I talked about the cubes {1, 8, 27, 64, 125, ...} and how their first differences {7, 19, 37, 61, ...} might at first seem to lack an obvious pattern, but taking the second differences {12, 18, 24, ...} takes you down to the simply related level. Taking the third differences {6, 6, ...} brings us to the perfectly stable level, where chaos dissolves into order.

But this (as I noted) is a handpicked example. Perhaps the "messy real world" lacks the beauty of these abstract mathematical objects? Perhaps it would be more appropriate to talk about neuroscience or gene expression networks?

Abstract math, being constructed solely in imagination, arises from simple foundations - a small set of initial axioms - and is a closed system; conditions that might seem unnaturally conducive to neatness.

Which is to say: In pure math, you don't have to worry about a tiger leaping out of the bushes and eating Pascal's Triangle.

So is the real world uglier than mathematics?

Strange that people ask this. I mean, the question might have been sensible two and a half millennia ago...

Back when the Greek philosophers were debating what this "real world" thingy might be made of, there were many positions. Heraclitus said, "All is fire." Thales said, "All is water." Pythagoras said, "All is number."

Score: Heraclitus 0 Thales 0 Pythagoras 1

Beneath the complex forms and shapes of the surface world, there is a simple level, an exact and stable level, whose laws we name "physics". This discovery, the Great Surprise, has already taken place at our point in human history - but it does not do to forget that it was surprising. Once upon a time, people went in search of underlying beauty, with no guarantee of finding it; and once upon a time, they found it; and now it is a known thing, and taken for granted.

Then why can't we predict the location of every tiger in the bushes as easily as we predict the sixth cube?

I count three sources of uncertainty even within worlds of pure math - two obvious sources, and one not so obvious.

The first source of uncertainty is that even a creature of pure math, living embedded in a world of pure math, may not know the math. Humans walked the Earth long before Galileo/Newton/Einstein discovered the law of gravity that prevents us from being flung off into space. You can be governed by stable fundamental rules without knowing them. There is no law of physics which says that laws of physics must be explicitly represented, as knowledge, in brains that run under them.

We do not yet have the Theory of Everything. Our best current theories are things of math, but they are not perfectly integrated with each other. The most probable explanation is that - as has previously proved to be the case - we are seeing surface manifestations of deeper math. So by far the best guess is that reality is made of math; but we do not fully know which math, yet.

But physicists have to construct huge particle accelerators to distinguish between theories - to manifest their remaining uncertainty in any visible fashion. That physicists must go to such lengths to be unsure, suggests that this is not the source of our uncertainty about stock prices.

The second obvious source of uncertainty is that even when you know all the relevant laws of physics, you may not have enough computing power to extrapolate them. We know every fundamental physical law that is relevant to a chain of amino acids folding itself into a protein. But we still can't predict the shape of the protein from the amino acids. Some tiny little 5-nanometer molecule that folds in a microsecond is too much information for current computers to handle (never mind tigers and stock prices). Our frontier efforts in protein folding use clever approximations, rather than the underlying Schrödinger equation. When it comes to describing a 5-nanometer object using really basic physics, over quarks - well, you don't even bother trying.

We have to use instruments like X-ray crystallography and NMR to discover the shapes of proteins that are fully determined by physics we know and a DNA sequence we know. We are not logically omniscient; we cannot see all the implications of our thoughts; we do not know what we believe.

The third source of uncertainty is the most difficult to understand, and Nick Bostrom has written a book about it. Suppose that the sequence {1, 8, 27, 64, 125, ...} exists; suppose that this is a fact. And suppose that atop each cube is a little person - one person per cube - and suppose that this is also a fact.

If you stand on the outside and take a global perspective - looking down from above at the sequence of cubes and the little people perched on top - then these two facts say everything there is to know about the sequence and the people.

But if you are one of the little people perched atop a cube, and you know these two facts, there is still a third piece of information you need to make predictions: "Which cube am I standing on?"

You expect to find yourself standing on a cube; you do not expect to find yourself standing on the number 7. Your anticipations are definitely constrained by your knowledge of the basic physics; your beliefs are falsifiable. But you still have to look down to find out whether you're standing on 1728 or 5177717. If you can do fast mental arithmetic, then seeing that the first two digits of a four-digit cube are 17__ will be sufficient to guess that the last digits are 2 and 8. Otherwise you may have to look to discover the 2 and 8 as well.

To figure out what the night sky should look like, it's not enough to know the laws of physics. It's not even enough to have logical omniscience over their consequences. You have to know where you are in the universe. You have to know that you're looking up at the night sky from Earth. The information required is not just the information to locate Earth in the visible universe, but in the entire universe, including all the parts that our telescopes can't see because they are too distant, and different inflationary universes, and alternate Everett branches.

It's a good bet that "uncertainty about initial conditions at the boundary" is really indexical uncertainty. But if not, it's empirical uncertainty, uncertainty about how the universe is from a global perspective, which puts it in the same class as uncertainty about fundamental laws.

Wherever our best guess is that the "real world" has an irretrievably messy component, it is because of the second and third sources of uncertainty - logical uncertainty and indexical uncertainty.

Ignorance of fundamental laws does not tell you that a messy-looking pattern really is messy. It might just be that you haven't figured out the order yet.

But when it comes to messy gene expression networks, we've already found the hidden beauty - the stable level of underlying physics. Because we've already found the master order, we can guess that we won't find any additional secret patterns that will make biology as easy as a sequence of cubes. Knowing the rules of the game, we know that the game is hard. We don't have enough computing power to do protein chemistry from physics (the second source of uncertainty) and evolutionary pathways may have gone different ways on different planets (the third source of uncertainty). New discoveries in basic physics won't help us here.

If you were an ancient Greek staring at the raw data from a biology experiment, you would be much wiser to look for some hidden structure of Pythagorean elegance, all the proteins lining up in a perfect icosahedron. But in biology we already know where the Pythagorean elegance is, and we know it's too far down to help us overcome our indexical and logical uncertainty.

Similarly, we can be confident that no one will ever be able to predict the results of certain quantum experiments, only because our fundamental theory tells us quite definitely that different versions of us will see different results. If your knowledge of fundamental laws tells you that there's a sequence of cubes, and that there's one little person standing on top of each cube, and that the little people are all alike except for being on different cubes, and that you are one of these little people, then you know that you have no way of deducing which cube you're on except by looking.

The best current knowledge says that the "real world" is a perfectly regular, deterministic, and very large mathematical object which is highly expensive to simulate. So "real life" is less like predicting the next cube in a sequence of cubes, and more like knowing that lots of little people are standing on top of cubes, but not knowing who you personally are, and also not being very good at mental arithmetic. Our knowledge of the rules does constrain our anticipations, quite a bit, but not perfectly.

There, now doesn't that sound like real life?

But uncertainty exists in the map, not in the territory. If we are ignorant of a phenomenon, that is a fact about our state of mind, not a fact about the phenomenon itself. Empirical uncertainty, logical uncertainty, and indexical uncertainty are just names for our own bewilderment. The best current guess is that the world is math and the math is perfectly regular. The messiness is only in the eye of the beholder.

Even the huge morass of the blogosphere is embedded in this perfect physics, which is ultimately as orderly as {1, 8, 27, 64, 125, ...}.

So the Internet is not a big muck... it's a series of cubes.

Referenced by: Beautiful Probability • Bell's Theorem: No EPR "Reality"

Original with comments: Is Reality Ugly?

Beautiful Probability

Eliezer Yudkowsky, 14 January 2008 07:19AM

Followup to: Beautiful Math, Expecting Beauty, Is Reality Ugly?

Should we expect rationality to be, on some level, simple? Should we search and hope for underlying beauty in the arts of belief and choice?

Let me introduce this issue by borrowing a complaint of the late great Bayesian Master, E. T. Jaynes (1990):

"Two medical researchers use the same treatment independently, in different hospitals. Neither would stoop to falsifying the data, but one had decided beforehand that because of finite resources he would stop after treating N=100 patients, however many cures were observed by then. The other had staked his reputation on the efficacy of the treatment, and decided he would not stop until he had data indicating a rate of cures definitely greater than 60%, however many patients that might require. But in fact, both stopped with exactly the same data: n = 100 [patients], r = 70 [cures]. Should we then draw different conclusions from their experiments?" (Presumably the two control groups also had equal results.)

According to old-fashioned statistical procedure - which I believe is still being taught today - the two researchers have performed different experiments with different stopping conditions. The two experiments could have terminated with different data, and therefore represent different tests of the hypothesis, requiring different statistical analyses. It's quite possible that the first experiment will be "statistically significant", the second not.

Whether or not you are disturbed by this says a good deal about your attitude toward probability theory, and indeed, rationality itself.

Non-Bayesian statisticians might shrug, saying, "Well, not all statistical tools have the same strengths and weaknesses, y'know - a hammer isn't like a screwdriver - and if you apply different statistical tools you may get different results, just like using the same data to compute a linear regression or train a regularized neural network. You've got to use the right tool for the occasion. Life is messy -"

And then there's the Bayesian reply: "Excuse you? The evidential impact of a fixed experimental method, producing the same data, depends on the researcher's private thoughts? And you have the nerve to accuse us of being 'too subjective'?"

If Nature is one way, the likelihood of the data coming out the way we have seen will be one thing. If Nature is another way, the likelihood of the data coming out that way will be something else. But the likelihood of a given state of Nature producing the data we have seen, has nothing to do with the researcher's private intentions. So whatever our hypotheses about Nature, the likelihood ratio is the same, and the evidential impact is the same, and the posterior belief should be the same, between the two experiments. At least one of the two Old Style methods must discard relevant information - or simply do the wrong calculation - for the two methods to arrive at different answers.

The ancient war between the Bayesians and the accursèd frequentists stretches back through decades, and I'm not going to try to recount that elder history in this blog post.

But one of the central conflicts is that Bayesians expect probability theory to be... what's the word I'm looking for? "Neat?" "Clean?" "Self-consistent?"

As Jaynes says, the theorems of Bayesian probability are just that, theorems in a coherent proof system. No matter what derivations you use, in what order, the results of Bayesian probability theory should always be consistent - every theorem compatible with every other theorem.

If you want to know the sum of 10 + 10, you can redefine it as (2 * 5) + (7 + 3) or as (2 * (4 + 6)) or use whatever other legal tricks you like, but the result always has to come out to be the same, in this case, 20. If it comes out as 20 one way and 19 the other way, then you may conclude you did something illegal on at least one of the two occasions. (In arithmetic, the illegal operation is usually division by zero; in probability theory, it is usually an infinity that was not taken as a the limit of a finite process.)

If you get the result 19 = 20, look hard for that error you just made, because it's unlikely that you've sent arithmetic itself up in smoke. If anyone should ever succeed in deriving a real contradiction from Bayesian probability theory - like, say, two different evidential impacts from the same experimental method yielding the same results - then the whole edifice goes up in smoke. Along with set theory, 'cause I'm pretty sure ZF provides a model for probability theory.

Math! That's the word I was looking for. Bayesians expect probability theory to be math. That's why we're interested in Cox's Theorem and its many extensions, showing that any representation of uncertainty which obeys certain constraints has to map onto probability theory. Coherent math is great, but unique math is even better.

And yet... should rationality be math? It is by no means a foregone conclusion that probability should be pretty. The real world is messy - so shouldn't you need messy reasoning to handle it? Maybe the non-Bayesian statisticians, with their vast collection of ad-hoc methods and ad-hoc justifications, are strictly more competent because they have a strictly larger toolbox. It's nice when problems are clean, but they usually aren't, and you have to live with that.

After all, it's a well-known fact that you can't use Bayesian methods on many problems because the Bayesian calculation is computationally intractable. So why not let many flowers bloom? Why not have more than one tool in your toolbox?

That's the fundamental difference in mindset. Old School statisticians thought in terms of tools, tricks to throw at particular problems. Bayesians - at least this Bayesian, though I don't think I'm speaking only for myself - we think in terms of laws.

Looking for laws isn't the same as looking for especially neat and pretty tools. The second law of thermodynamics isn't an especially neat and pretty refrigerator.

The Carnot cycle is an ideal engine - in fact, the ideal engine. No engine powered by two heat reservoirs can be more efficient than a Carnot engine. As a corollary, all thermodynamically reversible engines operating between the same heat reservoirs are equally efficient.

But, of course, you can't use a Carnot engine to power a real car. A real car's engine bears the same resemblance to a Carnot engine that the car's tires bear to perfect rolling cylinders.

Clearly, then, a Carnot engine is a useless tool for building a real-world car. The second law of thermodynamics, obviously, is not applicable here. It's too hard to make an engine that obeys it, in the real world. Just ignore thermodynamics - use whatever works.

This is the sort of confusion that I think reigns over they who still cling to the Old Ways.

No, you can't always do the exact Bayesian calculation for a problem. Sometimes you must seek an approximation; often, indeed. This doesn't mean that probability theory has ceased to apply, any more than your inability to calculate the aerodynamics of a 747 on an atom-by-atom basis implies that the 747 is not made out of atoms. Whatever approximation you use, it works to the extent that it approximates the ideal Bayesian calculation - and fails to the extent that it departs.

Bayesianism's coherence and uniqueness proofs cut both ways. Just as any calculation that obeys Cox's coherency axioms (or any of the many reformulations and generalizations) must map onto probabilities, so too, anything that is not Bayesian must fail one of the coherency tests. This, in turn, opens you to punishments like Dutch-booking (accepting combinations of bets that are sure losses, or rejecting combinations of bets that are sure gains).

You may not be able to compute the optimal answer. But whatever approximation you use, both its failures and successes will be explainable in terms of Bayesian probability theory. You may not know the explanation; that does not mean no explanation exists.

So you want to use a linear regression, instead of doing Bayesian updates? But look to the underlying structure of the linear regression, and you see that it corresponds to picking the best point estimate given a Gaussian likelihood function and a uniform prior over the parameters.

You want to use a regularized linear regression, because that works better in practice? Well, that corresponds (says the Bayesian) to having a Gaussian prior over the weights.

Sometimes you can't use Bayesian methods literally; often, indeed. But when you can use the exact Bayesian calculation that uses every scrap of available knowledge, you are done. You will never find a statistical method that yields a better answer. You may find a cheap approximation that works excellently nearly all the time, and it will be cheaper, but it will not be more accurate. Not unless the other method uses knowledge, perhaps in the form of disguised prior information, that you are not allowing into the Bayesian calculation; and then when you feed the prior information into the Bayesian calculation, the Bayesian calculation will again be equal or superior.

When you use an Old Style ad-hoc statistical tool with an ad-hoc (but often quite interesting) justification, you never know if someone else will come up with an even more clever tool tomorrow. But when you can directly use a calculation that mirrors the Bayesian law, you're done - like managing to put a Carnot heat engine into your car. It is, as the saying goes, "Bayes-optimal".

It seems to me that the toolboxers are looking at the sequence of cubes {1, 8, 27, 64, 125, ...} and pointing to the first differences {7, 19, 37, 61, ...} and saying "Look, life isn't always so neat - you've got to adapt to circumstances." And the Bayesians are pointing to the third differences, the underlying stable level {6, 6, 6, 6, 6, ...}. And the critics are saying, "What the heck are you talking about? It's 7, 19, 37 not 6, 6, 6. You are oversimplifying this messy problem; you are too attached to simplicity."

It's not necessarily simple on a surface level. You have to dive deeper than that to find stability.

Think laws, not tools. Needing to calculate approximations to a law doesn't change the law. Planes are still atoms, they aren't governed by special exceptions in Nature for aerodynamic calculations. The approximation exists in the map, not in the territory. You can know the second law of thermodynamics, and yet apply yourself as an engineer to build an imperfect car engine. The second law does not cease to be applicable; your knowledge of that law, and of Carnot cycles, helps you get as close to the ideal efficiency as you can.

We aren't enchanted by Bayesian methods merely because they're beautiful. The beauty is a side effect. Bayesian theorems are elegant, coherent, optimal, and provably unique because they are laws.

Addendum: Cyan directs us to chapter 37 of MacKay's excellent statistics book, free online, for a more thorough explanation of the opening problem.

Jaynes, E. T. (1990.) Probability Theory as Logic. In: P. F. Fougere (Ed.), Maximum Entropy and Bayesian Methods. Kluwer Academic Publishers.

MacKay, D. (2003.) Information Theory, Inference, and Learning Algorithms. Cambridge: Cambridge University Press.

Referenced by: The Allais Paradox • Zut Allais! • The "Intuitions" Behind "Utilitarianism" • Trust in Bayes • Something to Protect • Newcomb's Problem and Regret of Rationality • Perpetual Motion Beliefs • Searching for Bayes-Structure • Zombies! Zombies? • Joint Configurations • Many Worlds, One Best Guess • Science Isn't Strict Enough • Timeless Beauty • A Premature Word on AI • When (Not) To Use Probabilities • My Bayesian Enlightenment

Original with comments: Beautiful Probability

Trust in Math

Eliezer Yudkowsky, 15 January 2008 04:25AM

Followup to: Expecting Beauty

I was once reading a Robert Heinlein story - sadly I neglected to note down which story, but I do think it was a Heinlein - where one of the characters says something like, "Logic is a fine thing, but I have seen a perfectly logical proof that 2 = 1." Authors are not to be confused with characters, but the line is voiced by one of Heinlein's trustworthy father figures. I find myself worried that Heinlein may have meant it.

The classic proof that 2 = 1 runs thus. First, let x = y = 1. Then:

1. x = y

2. x^2 = xy

3. x^2 - y^2 = xy - y^2

4. (x + y)(x - y) = y(x - y)

5. x + y = y

6. 2 = 1

Now, you could look at that, and shrug, and say, "Well, logic doesn't always work."

Or, if you felt that math had rightfully earned just a bit more credibility than that, over the last thirty thousand years, then you might suspect the flaw lay in your use of math, rather than Math Itself.

You might suspect that the proof was not, in fact, "perfectly logical".

The novice goes astray and says: "The Art failed me."
The master goes astray and says: "I failed my Art."

Is this - gasp! - faith? To believe that math is consistent, when you have seen with your own eyes a proof that it is not? Are you supposed to just ignore the contrary evidence, my good Bayesian?

As I have remarked before, it seems worthwhile to distinguish "faith" that the sun will rise in the east just like the last hundred thousand times observed, from "faith" that tomorrow a green goblin will give you a bag of gold doubloons. When first-order arithmetic has been observed to be internally consistent over the last ten million theorems proved in it, and you see a seeming proof of inconsistency, it is, perhaps, reasonable to double-check the proof.

You're not going to ignore the contrary evidence. You're going to double-check it. You're going to also take into account the last ten million times that first-order arithmetic has proven consistent, when you evaluate your new posterior confidence that 2 = 1 is not perfectly logical. On that basis, you are going to evaluate a high probability that, if you check for a flaw, you are likely to find one.

But isn't this motivated skepticism? The most fearful bane of students of bias? You're applying a stronger standard of checking to incongruent evidence than congruent evidence?

Yes. So it is necessary to be careful around this sort of reasoning, because it can induce belief hysteresis - a case where your final beliefs end up determined by the order in which you see the evidence. When you add decision theory, unlike the case of pure probability theory, you have to decide whether to take costly actions to look for additional evidence, and you will do this based on the evidence you have seen so far.

Perhaps you should think to yourself, "Huh, if I didn't spot this flaw at first sight, then I may have accepted some flawed congruent evidence too. What other mistaken proofs do I have in my head, whose absurdity is not at first apparent?" Maybe you should apply stronger scrutiny to the next piece of congruent evidence you hear, just to balance things out.

Real faith, blind faith, would be if you looked at the proof and shrugged and said, "Seems like a valid proof to me, but I don't care, I believe in math." That would be discarding the evidence.

You have a doubt. Move to resolve it. That is the purpose of a doubt. After all, if the proof does hold up, you will have to discard first-order arithmetic. It's not acceptable to be walking around with your mind containing both the belief that arithmetic is consistent, and what seems like a valid proof that 2 = 1.

Oh, and the flaw in the proof? Simple technique for finding it: Substitute 1 for both x and y, concretely evaluate the arithmetic on both sides of the equation, and find the first line where a true equation is followed by a false equation. Whatever step was performed between those two equations, must have been illegal - illegal for some general reason, mind you; not illegal just because it led to a conclusion you don't like.

That's what Heinlein should have looked for - if, perhaps, he'd had a bit more faith in algebra.

Added: Andrew2 says the character was Jubal from Stranger in a Strange Land.

Charlie says that Heinlein did graduate work in math at UCLA and was a hardcore formalist. I guess either Jubal wasn't expressing an authorial opinion, or Heinlein meant to convey "deceptively logical-seeming" by the phrase "perfectly logical".

If you don't already know the flaw in the algebra, there are spoilers in the comments ahead.

Referenced by: Trust in Bayes • The Cartoon Guide to Löb's Theorem

Original with comments: Trust in Math

Rationality Quotes 1

Eliezer Yudkowsky, 16 January 2008 07:41AM

I'll be moving to Redwood City, CA in a week, so forgive me if I don't get a regular post out every day between now and then. As a substitute offering, some items from my (offline) quotesfile:

"It appears to be a quite general principle that, whenever there is a randomized way of doing something, then there is a nonrandomized way that delivers better performance but requires more thought."
-- E. T. Jaynes

"When you're young, you look at television and think, There's a conspiracy. The networks have conspired to dumb us down. But when you get a little older, you realize that's not true. The networks are in business to give people exactly what they want. That's a far more depressing thought. Conspiracy is optimistic! You can shoot the bastards!"
-- Steve Jobs

"Saving a drowning child is no more a moral duty than understanding a syllogism is a logical one."
-- Sam Harris, The End of Faith

"Don't ask why (again, long story), but the number 5,479,863,282.86 has been stuck in my head for as long as I can remember."
-- Ultimatum479

"I've met these people, the ones from the glossy magazines. I've walked among them. I have seen, firsthand, their callow, empty lives. I have watched them from the shadows when they thought themselves alone. And I can tell you this: I'm afraid there is not one of them who would swap lives with you at gunpoint."
-- Neil Gaiman, Anansi Boys

"Before you can get to the end of this paragraph, another person will probably die because of what someone else believes about God."
-- Sam Harris, The End of Faith

"Someone walking down the street with absolutely no scars or calluses would look pretty odd. I suspect having a conversation with someone who'd never taken any emotional or mental damage would be even odder. The line between "experience" and "damage" is pretty thin."
-- Aliza, from the Open-Source Wish Project

"Fear and lies fester in darkness. The truth may wound, but it cuts clean."
-- Jacqueline Carey, Kushiel's Avatar

"I never make a prediction that can be proved wrong within 24 hours."
-- Louis Rukeyser

"You have been asking what you could do in the great events that are now stirring, and have found that you could do nothing. But that is because your suffering has caused you to phrase the question in the wrong way... Instead of asking what you could do, you ought to have been asking what needs to be done."
-- Steven Brust, The Paths of the Dead

"In The Brothers Karamazov, Alyosha expresses the idea which panicked Dostoyevski more than any other: Without God, 'everything is lawful'. But as Mohammed Atta can explain, the opposite is true. Without God, murder is forbidden by human law; it is only for those acting on behalf of God, that everything is permitted."
-- Jonathan Wallace

"It's possible to describe anything in mathematical notation. I recall seeing some paper once in which someone had created a mathematical description of C. (I forget whether or not this included the preprocessor.) As an achievement, this is somewhat like building a full-size model of the Eiffel Tower out of tongue depressors. It's clearly not the act of a talentless man, but you have to wonder what he said when he applied for his grant."
-- Mencius Moldbug

"If we are fervently passionate about the idea that fire is hot, we are more rational than the man who calmly and quietly says fire is cold."
-- Tom McCabe

"You are only as strong as your weakest delusion."
-- Common Sense Camp

Original with comments: Rationality Quotes 1

Rationality Quotes 2

Eliezer Yudkowsky, 16 January 2008 11:47PM

"I often have to arrange talks years in advance. If I am asked for a title, I suggest “The Current Crisis in the Middle East.” It has yet to fail."
-- Noam Chomsky

"We don't do science for the general public. We do it for each other. Good day."
-- Renato Dalbecco, complete text of interview with H F Judson

"Most witches don't believe in gods. They know that the gods exist, of course. They even deal with them occasionally. But they don't believe in them. They know them too well. It would be like believing in the postman."
-- Terry Pratchett, Witches Abroad

"He didn't reject the idea so much as not react to it and watch as it floated away."
-- David Foster Wallace, Infinite Jest

"People can't predict how long they will be happy with recently acquired objects, how long their marriages will last, how their new jobs will turn out, yet it's subatomic particles that they cite as "limits of prediction." They're ignoring a mammoth standing in front of them in favor of matter even a microscope would not allow them to see."
-- Nassim Taleb, The Black Swan

"I would strongly challenge the notion of “that's what's called growing up.” The most depressing part of the nomenclature around adolescence and college life is this bizarre connection between “experimentation” / “learning from your mistakes” and binge drinking, reckless sex, and drug use."
-- Ben Casnocha

"Behind every story of extraordinary heroism, there is a less exciting and more interesting story about the larger failures that made heroism necessary in the first place."
-- Black Belt Bayesian

"Anyone who claims that the brain is a total mystery should be slapped upside the head with the MIT Encyclopedia of the Cognitive Sciences. All one thousand ninety-six pages of it."
-- Tom McCabe

"Try explaining anything scientific to your friends -- you soon won't have any."
-- Soloport

"The definition of the word "meaning," is something that is conveyed. So, who is conveying this "meaning" that you speak of? To put it another way, if "life" is a painting, then who is painter? Whoever is the painter is the one who decides what meaning the painting has. Now, depending on your outlook, the painter is either yourself, or God. Depending upon how you answer that question, you should now be able to figure out the answer."
-- Flipside

"It is doubtful that most "Noble Lies" are at all noble."
-- Samantha Atkins

"In essence, we have to be more moral than God. A quick glance over God's rap sheet suggests that this is, indeed, possible."
-- Blake Stacey

"Are there wonderful Xians out there? Yes. Do I think they are assholes for believing I deserve to burn forever because I don't believe in fairies? Hell yes."
-- RRyan

"I am most often irritated by those who attack the bishop but somehow fall for the securities analyst."
-- Nassim Taleb, The Black Swan

"I've found that people who are great at something are not so much convinced of their own greatness as mystified at why everyone else seems so incompetent."
-- Paul Graham

"But then... it used to be so simple, once upon a time.
Because the universe was full of ignorance all around and the scientist panned through it like a prospector crouched over a mountain stream, looking for the gold of knowledge among the gravel of unreason, the sand of uncertainty and the little whiskery eight-legged swimming things of superstition.
Occasionally he would straighten up and say things like "Hurrah, I've discovered Boyle's Third Law." And everyone knew where they stood. But the trouble was that ignorance became more interesting, especially big fascinating ignorance about huge and important things like matter and creation, and people stopped patiently building their little houses of rational sticks in the chaos of the universe and started getting interested in the chaos itself -- partly because it was a lot easier to be an expert on chaos, but mostly because it made really good patterns that you could put on a t-shirt.
And instead of getting on with proper science scientists suddenly went around saying how impossible it was to know anything, and that there wasn't really anything you could call reality to know anything about, and how all this was tremendously exciting, and incidentally did you know there were possibly all these little universes all over the place but no one can see them because they are all curved in on themselves? Incidentally, don't you think this is a rather good t-shirt?"
-- Terry Pratchett, Witches Abroad

"To the extent there is a secret handshake among good hackers, it's when they know one another well enough to express opinions that would get them stoned to death by the general public."
-- Paul Graham

Original with comments: Rationality Quotes 2

Rationality Quotes 3

Eliezer Yudkowsky, 18 January 2008 02:20AM

"Reality is that which, when you stop believing in it, doesn't go away."
-- Philip K. Dick

"How many legs does a dog have, if you call the tail a leg? Four. Calling a tail a leg doesn't make it a leg."
-- Abraham Lincoln

"Faced with the choice of changing one's mind and proving that there is no need to do so, almost everyone gets busy on the proof."
-- John Kenneth Galbraith

"I'd rather live with a good question than a bad answer."
-- Aryeh Frimer

"It ain't a true crisis of faith unless things could just as easily go either way."
-- bunnyThor

"My best test for a libertarian so far is to ask what needs to be done to protect ancient sequoias. If you say you need to buy them, you pass."
-- Rafal Smigrodzki

"Mystical explanations are considered deep. The truth is that they are not even superficial."
-- Friedrich Nietzsche, The Gay Science

"Some people know better, and they still make the mistake. That's when ignorance becomes stupidity."
-- Aaron McBride

"Perfection is achieved, not when there is nothing left to add, but when there is nothing left to take away."
-- Antoine de Saint-Exupéry

"Skill is successfully walking a tightrope over Niagara Falls. Intelligence is not trying."
-- Unknown

"The fact that one apple added to one apple invariably gives two apples helps in the teaching of arithmetic, but has no bearing on the truth of the proposition that 1 + 1 = 2."
-- James R. Newman, The World of Mathematics

"Giving a person a high IQ is kind of like giving a person a million dollars. A few individuals will do something interesting with it, but most will piss it away on trinkets and pointless exercises."
-- J. Andrew Rogers

"Fatal stupidity is inefficient: idiots take other people out with them way too often."
-- Mike

"Surprises are things that you not only didn't know, but that contradict things you thought you knew. And so they're the most valuable sort of fact you can get. They're like a food that's not merely healthy, but counteracts the unhealthy effects of things you've already eaten."
-- Paul Graham

"So many times I found myself on the receiving end of unkind treatment, or on the giving end of the same, that there is not often a space in which I can find peace to escape from this woman I have become. I want so much to not know the things I've known, to not feel the things I've felt, to not have hurt the ones I've hurt."
-- Sara

"I am an undrawn Grand Master of the Game, and you cannot lose well against me, no matter the form. But as with all my children, I will play this game or another against you every day that you are here, and in time you will learn to lose well, and you may even learn to lose brilliantly."
-- John M. Ford, The Final Reflection

"When, however, the lay public rallies round an idea that is denounced by distinguished but elderly scientists and supports that idea with great fervor and emotion -- the distinguished but elderly scientists are then, after all, probably right."
-- Asimov's Corollary

"Stupid gods! Don't they realize how important this is?"
-- The Wings of Change

"Laws do inhibit some citizens from breaking them, and laws do specify punishments for crimes, but laws do not prevent anybody from doing anything."
-- Michael Roy Ames

"I have often been accused by friends and acquaintances of being very logical. What they really meant was that I take some principle or insight and apply it further than other people that they know."
-- Lee Corbin

"If the meanings of "true" and "false" were switched, then this sentence wouldn't be false."
-- Douglas Hofstadter

"Make changes based on your strongest opportunities, not your most convenient ones."
-- MegaTokyo

"You are not ready to count your enemy's losses until you have learned to count your own. And remember that some enemies will never have learned to count."
-- John M. Ford, The Final Reflection

"Our brains live in a dark, quiet, wet place. That is the reality. It is only by means of our senses that we get the illusion of being out there in the world. In a way, our bodies are a form of telepresence, operated by our brains, huddling safe in their little caves of bone."
-- Hal Finney

"If you do not wish a thing heard, do not say it."
-- John M. Ford, The Final Reflection

"The four points of the compass be logic, knowledge, wisdom and the unknown. Some do bow in that final direction. Others advance upon it. To bow before the one is to lose sight of the three. I may submit to the unknown, but never to the unknowable. The man who bows in that final direction is either a saint or a fool. I have no use for either."
-- Roger Zelazny, Lord of Light

"The assassin's gun may believe it is a surgeon's laser. But the assassin must know the task."
-- John M. Ford, The Final Reflection

"Man has Nature whacked," said someone to a friend of mine not long ago. In their context the words had a certain tragic beauty, for the speaker was dying of tuberculosis. "No matter," he said, "I know I'm one of the casualties. Of course there are casualties on the winning as well as on the losing side. But that doesn't alter the fact that it is winning."
-- C.S. Lewis, The Abolition of Man

Original with comments: Rationality Quotes 3

The Allais Paradox

Eliezer Yudkowsky, 19 January 2008 03:05AM

Followup to: But There's Still A Chance Right?, Beautiful Probability

Choose between the following two options:

1A. $24,000, with certainty.
1B. 33/34 chance of winning $27,000, and 1/34 chance of winning nothing.

Which seems more intuitively appealing? And which one would you choose in real life?

Now which of these two options would you intuitively prefer, and which would you choose in real life?

2A. 34% chance of winning $24,000, and 66% chance of winning nothing.
2B. 33% chance of winning $27,000, and 67% chance of winning nothing.

The Allais Paradox - as Allais called it, though it's not really a paradox - was one of the first conflicts between decision theory and human reasoning to be experimentally exposed, in 1953. I've modified it slightly for ease of math, but the essential problem is the same: Most people prefer 1A > 1B, and most people prefer 2B > 2A. Indeed, in within-subject comparisons, a majority of subjects express both preferences simultaneously.

This is a problem because the 2s are equal to a one-third chance of playing the 1s. That is, 2A is equivalent to playing gamble 1A with 34% probability, and 2B is equivalent to playing 1B with 34% probability.

Among the axioms used to prove that "consistent" decisionmakers can be viewed as maximizing expected utility, is the Axiom of Independence: If X is strictly preferred to Y, then a probability P of X and (1 - P) of Z should be strictly preferred to P chance of Y and (1 - P) chance of Z.

All the axioms are consequences, as well as antecedents, of a consistent utility function. So it must be possible to prove that the experimental subjects above can't have a consistent utility function over outcomes. And indeed, you can't simultaneously have:

- U($24,000) > 33/34 U($27,000) + 1/34 U($0)

- 0.34 U($24,000) + 0.66 U($0) < 0.33 U($27,000) + 0.67 U($0)

These two equations are algebraically inconsistent, regardless of U, so the Allais Paradox has nothing to do with the diminishing marginal utility of money.

Maurice Allais initially defended the revealed preferences of the experimental subjects - he saw the experiment as exposing a flaw in the conventional ideas of utility, rather than exposing a flaw in human psychology. This was 1953, after all, and the heuristics-and-biases movement wouldn't really get started for another two decades. Allais thought his experiment just showed that the Axiom of Independence clearly wasn't a good idea in real life.

(How naive, how foolish, how simplistic is Bayesian decision theory...)

Surely, the certainty of having $24,000 should count for something. You can feel the difference, right? The solid reassurance?

(I'm starting to think of this as "naive philosophical realism" - supposing that our intuitions directly expose truths about which strategies are wiser, as though it was a directly perceived fact that "1A is superior to 1B". Intuitions directly expose truths about human cognitive functions, and only indirectly expose (after we reflect on the cognitive functions themselves) truths about rationality.)

"But come now," you say, "is it really such a terrible thing, to depart from Bayesian beauty?" Okay, so the subjects didn't follow the neat little "independence axiom" espoused by the likes of von Neumann and Morgenstern. Yet who says that things must be neat and tidy?

Why fret about elegance, if it makes us take risks we don't want? Expected utility tells us that we ought to assign some kind of number to an outcome, and then multiply that value by the outcome's probability, add them up, etc. Okay, but why do we have to do that? Why not make up more palatable rules instead?

There is always a price for leaving the Bayesian Way. That's what coherence and uniqueness theorems are all about.

In this case, if an agent prefers 1A > 1B, and 2B > 2A, it introduces a form of preference reversal - a dynamic inconsistency in the agent's planning. You become a money pump.

Suppose that at 12:00PM I roll a hundred-sided die. If the die shows a number greater than 34, the game terminates. Otherwise, at 12:05PM I consult a switch with two settings, A and B. If the setting is A, I pay you $24,000. If the setting is B, I roll a 34-sided die and pay you $27,000 unless the die shows "34", in which case I pay you nothing.

Let's say you prefer 1A over 1B, and 2B over 2A, and you would pay a single penny to indulge each preference. The switch starts in state A. Before 12:00PM, you pay me a penny to throw the switch to B. The die comes up 12. After 12:00PM and before 12:05PM, you pay me a penny to throw the switch to A.

I have taken your two cents on the subject.

If you indulge your intuitions, and dismiss mere elegance as a pointless obsession with neatness, then don't be surprised when your pennies get taken from you...

(I think the same failure to proportionally devalue the emotional impact of small probabilities is responsible for the lottery.)

Allais, M. (1953). Le comportement de l'homme rationnel devant le risque: Critique des postulats et axiomes de l'école américaine. Econometrica, 21, 503-46.

Kahneman, D. and Tversky, A. (1979.) Prospect Theory: An Analysis of Decision Under Risk. Econometrica, 47, 263-92.

Referenced by: Zut Allais! • Allais Malaise • Against Discount Rates • Timeless Identity • Moral Error and Moral Disagreement • Dunbar's Function

Original with comments: The Allais Paradox

Zut Allais!

Eliezer Yudkowsky, 20 January 2008 03:18AM

Continuation of: The Allais Paradox

Huh! I was not expecting that response. Looks like I ran into an inferential distance.

It probably helps in interpreting the Allais Paradox to have absorbed more of the gestalt of the field of heuristics and biases, such as:

- Experimental subjects tend to defend incoherent preferences even when they're really silly.

- People put very high values on small shifts in probability away from 0 or 1 (the certainty effect).

Let's start with the issue of incoherent preferences - preference reversals, dynamic inconsistency, money pumps, that sort of thing.

Anyone who knows a little prospect theory will have no trouble constructing cases where people say they would prefer to play gamble A rather than gamble B; but when you ask them to price the gambles they put a higher value on gamble B than gamble A. There are different perceptual features that become salient when you ask "Which do you prefer?" in a direct comparison, and "How much would you pay?" with a single item.

My books are packed up for the move, but from what I remember, this should typically generate a preference reversal:

1. 1/3 to win $18 and 2/3 to lose $1.50

2. 19/20 to win $4 and 1/20 to lose $0.25

Most people will (IIRC) rather play 2 than 1. But if you ask them to price the bets separately - ask for a price at which they would be indifferent between having that amount of money, and having a chance to play the gamble - people will (IIRC) put a higher price on 1 than on 2. If I'm wrong about this exact example, nonetheless, there are plenty of cases where such a pattern is exhibited experimentally.

So first you sell them a chance to play bet 1, at their stated price. Then you offer to trade bet 1 for bet 2. Then you buy bet 2 back from them, at their stated price. Then you do it again. Hence the phrase, "money pump".

Or to paraphrase Steve Omohundro: If you would rather be in Oakland than San Francisco, and you would rather be in San Jose than Oakland, and you would rather be in San Francisco than San Jose, you're going to spend an awful lot of money on taxi rides.

Amazingly, people defend these preference patterns. Some subjects abandon them after the money-pump effect is pointed out - revise their price or revise their preference - but some subjects defend them.

On one occasion, gamblers in Las Vegas played these kinds of bets for real money, using a roulette wheel. And afterward, one of the researchers tried to explain the problem with the incoherence between their pricing and their choices. From the transcript:

Experimenter: Well, how about the bid for Bet A? Do you have any further feelings about it now that you know you are choosing one but bidding more for the other one?
Subject: It's kind of strange, but no, I don't have any feelings at all whatsoever really about it. It's just one of those things. It shows my reasoning process isn't so good, but, other than that, I... no qualms.
...
E: Can I persuade you that it is an irrational pattern?
S: No, I don't think you probably could, but you could try.
...
E: Well, now let me suggest what has been called a money-pump game and try this out on you and see how you like it. If you think Bet A is worth 550 points [points were converted to dollars after the game, though not on a one-to-one basis] then you ought to be willing to give me 550 points if I give you the bet...
...
E: So you have Bet A, and I say, "Oh, you'd rather have Bet B wouldn't you?"
...
S: I'm losing money.
E: I'll buy Bet B from you. I'll be generous; I'll pay you more than 400 points. I'll pay you 401 points. Are you willing to sell me Bet B for 401 points?
S: Well, certainly.
...
E: I'm now ahead 149 points.
S: That's good reasoning on my part. (laughs) How many times are we going to go through this?
...
E: Well, I think I've pushed you as far as I know how to push you short of actually insulting you.
S: That's right.

You want to scream, "Just give up already! Intuition isn't always right!"

And then there's the business of the strange value that people attach to certainty. Again, I don't have my books, but I believe that one experiment showed that a shift from 100% probability to 99% probability weighed larger in people's minds than a shift from 80% probability to 20% probability.

The problem with attaching a huge extra value to certainty is that one time's certainty is another time's probability.

Yesterday I talked about the Allais Paradox:

- 1A. $24,000, with certainty.

- 1B. 33/34 chance of winning $27,000, and 1/34 chance of winning nothing.

- 2A. 34% chance of winning $24,000, and 66% chance of winning nothing.

- 2B. 33% chance of winning $27,000, and 67% chance of winning nothing.

The naive preference pattern on the Allais Paradox is 1A > 1B and 2B > 2A. Then you will pay me to throw a switch from A to B because you'd rather have a 33% chance of winning $27,000 than a 34% chance of winning $24,000. Then a die roll eliminates a chunk of the probability mass. In both cases you had at least a 66% chance of winning nothing. This die roll eliminates that 66%. So now option B is a 33/34 chance of winning $27,000, but option A is a certainty of winning $24,000. Oh, glorious certainty! So you pay me to throw the switch back from B to A.

Now, if I've told you in advance that I'm going to do all that, do you really want to pay me to throw the switch, and then pay me to throw it back? Or would you prefer to reconsider?

Whenever you try to price a probability shift from 24% to 23% as being less important than a shift from ~1 to 99% - every time you try to make an increment of probability have more value when it's near an end of the scale - you open yourself up to this kind of exploitation. I can always set up a chain of events that eliminates the probability mass, a bit at a time, until you're left with "certainty" that flips your preferences. One time's certainty is another time's uncertainty, and if you insist on treating the distance from ~1 to 0.99 as special, I can cause you to invert your preferences over time and pump some money out of you.

Can I persuade you, perhaps, that this is an irrational pattern?

Surely, if you've been reading this blog for a while, you realize that you - the very system and process that reads these very words - are a flawed piece of machinery. Your intuitions are not giving you direct, veridical information about good choices. If you don't believe that, there are some gambling games I'd like to play with you.

There are various other games you can also play with certainty effects. For example, if you offer someone a certainty of $400, or an 80% probability of $500 and a 20% probability of $300, they'll usually take the $400. But if you ask people to imagine themselves $500 richer, and ask if they would prefer a certain loss of $100 or a 20% chance of losing $200, they'll usually take the chance of losing $200. Same probability distribution over outcomes, different descriptions, different choices.

Yes, Virginia, you really should try to multiply the utility of outcomes by their probability. You really should. Don't be embarrassed to use clean math.

In the Allais paradox, figure out whether 1 unit of the difference between getting $24,000 and getting nothing, outweighs 33 units of the difference between getting $24,000 and $27,000. If it does, prefer 1A to 1B and 2A to 2B. If the 33 units outweigh the 1 unit, prefer 1B to 1A and 2B to 2A. As for calculating the utility of money, I would suggest using an approximation that assumes money is logarithmic in utility. If you've got plenty of money already, pick B. If $24,000 would double your existing assets, pick A. Case 2 or case 1, makes no difference. Oh, and be sure to assess the utility of total asset values - the utility of final outcome states of the world - not changes in assets, or you'll end up inconsistent again.

A number of commenters, yesterday, claimed that the preference pattern wasn't irrational because of "the utility of certainty", or something like that. One commenter even wrote U(Certainty) into an expected utility equation.

Does anyone remember that whole business about expected utility and utility being of fundamentally different types? Utilities are over outcomes. They are values you attach to particular, solid states of the world. You cannot feed a probability of 1 into a utility function. It makes no sense.

And before you sniff, "Hmph... you just want the math to be neat and tidy," remember that, in this case, the price of departing the Bayesian Way was paying someone to throw a switch and then throw it back.

But what about that solid, warm feeling of reassurance? Isn't that a utility?

That's being human. Humans are not expected utility maximizers. Whether you want to relax and have fun, or pay some extra money for a feeling of certainty, depends on whether you care more about satisfying your intuitions or actually achieving the goal.

If you're gambling at Las Vegas for fun, then by all means, don't think about the expected utility - you're going to lose money anyway.

But what if it were 24,000 lives at stake, instead of $24,000? The certainty effect is even stronger over human lives. Will you pay one human life to throw the switch, and another to switch it back?

Tolerating preference reversals makes a mockery of claims to optimization. If you drive from San Jose to San Francisco to Oakland to San Jose, over and over again, then you may get a lot of warm fuzzy feelings out of it, but you can't be interpreted as having a destination - as trying to go somewhere.

When you have circular preferences, you're not steering the future - just running in circles. If you enjoy running for its own sake, then fine. But if you have a goal - something you're trying to actually accomplish - a preference reversal reveals a big problem. At least one of the choices you're making must not be working to actually optimize the future in any coherent sense.

If what you care about is the warm fuzzy feeling of certainty, then fine. If someone's life is at stake, then you had best realize that your intuitions are a greasy lens through which to see the world. Your feelings are not providing you with direct, veridical information about strategic consequences - it feels that way, but they're not. Warm fuzzies can lead you far astray.

There are mathematical laws governing efficient strategies for steering the future. When something truly important is at stake - something more important than your feelings of happiness about the decision - then you should care about the math, if you truly care at all.

Referenced by: Allais Malaise • Circular Altruism • Buy Now Or Forever Hold Your Peace • Unspeakable Morality

Original with comments: Zut Allais!

Rationality Quotes 4

Eliezer Yudkowsky, 20 January 2008 05:00PM

"Altruistic behavior: An act done without any intent for personal gain in any form. Altruism requires that there is no want for material, physical, spiritual, or egoistic gain."
-- Glossary of Zen

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him."
-- Miyamoto Musashi, The Book of Five Rings

"You can win with a long weapon, and yet you can also win with a short weapon. In short, the Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size."
-- Miyamoto Musashi, The Book of Five Rings

"Well," Rowlands said carefully, turning the Land-Rover into the road. "I am not at all sure what it is that is going on all around us, Will bach, or where it is leading. But those men who know anything at all about the Light also know that there is a fierceness to its power, like the bare sword of the law, or the white burning of the sun." Suddenly his voice sounded to Will very strong, and very Welsh. "At the very heart, that is. Other things, like humanity, and mercy, and charity, that most good men hold more precious than all else, they do not come first for the Light. Oh, sometimes they are there; often, indeed. But in the very long run the concern of you people is with the absolute good, ahead of all else. You are like fanatics. Your masters, at any rate. Like the old Crusaders -- oh, like certain groups in every belief, though this is not a matter of religion, of course. At the centre of the Light there is a cold white flame, just as at the centre of the Dark there is a great black pit bottomless as the Universe."
His warm, deep voice ended, and there was only the roar of the engine. Will looked out over the grey-misted fields, silent.
"There was a great long speech, now," John Rowlands said awkwardly. "But I was only saying, be careful not to forget that there are people in this valley who can be hurt, even in the pursuit of good ends."
Will heard again in his mind Bran's anguished cry as the dog Cafall was shot dead, and heard his cold dismissal: go away, go away... And for a second another image, unexpected, flashed into his mind out of the past: the strong, bony face of Merriman his master, first of the Old Ones, cold in judgment of a much-loved figure who, through the frailty of being no more than a man, had once betrayed the cause of the Light.
He sighed. "I understand what you are saying," he said sadly. "But you misjudge us, because you are a man yourself. For us, there is only the destiny. Like a job to be done. We are here simply to save the world from the Dark. Make no mistake, John, the Dark is rising, and will take the world to itself very soon if nothing stands in its way. And if that should happen, then there would be no question ever, for anyone, either of warm charity or of cold absolute good, because nothing would exist in the world or in the hearts of men except that bottomless black pit. The charity and the mercy and the humanitarianism are for you, they are the only things by which men are able to exist together in peace. But in this hard case that we the Light are in, confronting the Dark, we can make no use of them. We are fighting a war. We are fighting for life or death -- not for our life, remember, since we cannot die. For yours."
He reached his hand behind him, over the back of the seat, and Pen licked it with his floppy wet tongue.
"Sometimes," Will said slowly, "in this sort of a war, it is not possible to pause, to smooth the way for one human being, because even that one small thing could mean an end of the world for all the rest."
A fine rain began to mist the windscreen. John Rowlands turned on the wipers, peering forward at the grey world as he drove. He said, "It is a cold world you live in, bachgen. I do not think so far ahead, myself. I would take the one human being over all the principle, all the time."
Will slumped down low in his seat, curling into a ball, pulling up his knees. "Oh, so would I," he said sadly. "So would I, if I could. It would feel a lot better inside me. But it wouldn't work."
-- Susan Cooper, The Grey King

Referenced by: Circular Altruism

Original with comments: Rationality Quotes 4

Allais Malaise

Eliezer Yudkowsky, 21 January 2008 12:40AM

Continuation of: The Allais Paradox, Zut Allais!

Judging by the comments on Zut Allais, I failed to emphasize the points that needed emphasis.

The problem with the Allais Paradox is the incoherent pattern 1A > 1B, 2B > 2A. If you need $24,000 for a lifesaving operation and an extra $3,000 won't help that much, then you choose 1A > 1B and 2A > 2B. If you have a million dollars in the bank account and your utility curve doesn't change much with an extra $25,000 or so, then you should choose 1B > 1A and 2B > 2A. Neither the individual choice 1A > 1B, nor the individual choice 2B > 2A, are of themselves irrational. It's the combination that's the problem.

Expected utility is not expected dollars. In the case above, the utility-distance from $24,000 to $27,000 is a tiny fraction of the distance from $21,000 to $24,000. So, as stated, you should choose 1A > 1B and 2A > 2B, a quite coherent combination. The Allais Paradox has nothing to do with believing that every added dollar is equally useful. That idea has been rejected since the dawn of decision theory.

If satisfying your intuitions is more important to you than money, do whatever the heck you want. Drop the money over Niagara falls. Blow it all on expensive champagne. Set fire to your hair. Whatever. If the largest utility you care about is the utility of feeling good about your decision, then any decision that feels good is the right one. If you say that different trajectories to the same outcome "matter emotionally", then you're attaching an inherent utility to conforming to the brain's native method of optimization, whether or not it actually optimizes. Heck, running around in circles from preference reversals could feel really good too. But if you care enough about the stakes that winning is more important than your brain's good feelings about an intuition-conforming strategy, then use decision theory.

If you suppose the problem is different from the one presented - that the gambles are untrustworthy and that, after this mistrust is taken into account, the payoff probabilities are not as described - then, obviously, you can make the answer anything you want.

Let's say you're dying of thirst, you only have $1.00, and you have to choose between a vending machine that dispenses a drink with certainty for $0.90, versus spending $0.75 on a vending machine that dispenses a drink with 99% probability. Here, the 1% chance of dying is worth more to you than $0.15, so you would pay the extra fifteen cents. You would also pay the extra fifteen cents if the two vending machines dispensed drinks with 75% probability and 74% probability respectively. The 1% probability is worth the same amount whether or not it's the last increment towards certainty. This pattern of decisions is perfectly coherent. Don't confuse being rational with being shortsighted or greedy.

Added: A 50% probability of $30K and a 50% probability of $20K, is not the same as a 50% probability of $26K and a 50% probability of $24K. If your utility is logarithmic in money (the standard assumption) then you will definitely prefer the latter to the former: 0.5 log(30) + 0.5 log(20) < 0.5 log(26) + 0.5 log(24). You take the expectation of the utility of the money, not the utility of the expectation of the money.

Referenced by: The "Intuitions" Behind "Utilitarianism" • How An Algorithm Feels From Inside

Original with comments: Allais Malaise

Against Discount Rates

Eliezer Yudkowsky, 21 January 2008 10:00AM

I've never been a fan of the notion that we should (normatively) have a discount rate in our pure preferences - as opposed to a pseudo-discount rate arising from monetary inflation, or from opportunity costs of other investments, or from various probabilistic catastrophes that destroy resources or consumers. The idea that it is literally, fundamentally 5% more important that a poverty-stricken family have clean water in 2008, than that a similar family have clean water in 2009, seems like pure discrimination to me - just as much as if you were to discriminate between blacks and whites.

And there's worse: If your temporal discounting follows any curve other than the exponential, you'll have time-inconsistent goals that force you to wage war against your future selves - preference reversals - cases where your self of 2008 will pay a dollar to ensure that your future self gets option A in 2011 rather than B in 2010; but then your future self in 2009 will pay another dollar to get B in 2010 rather than A in 2011.

But a 5%-per-year discount rate, compounded exponentially, implies that it is worth saving a single person from torture today, at the cost of 168 people being tortured a century later, or a googol persons being tortured 4,490 years later.

People who deal in global catastrophic risks sometimes have to wrestle with the discount rate assumed by standard economics. Is a human civilization spreading through the Milky Way, 100,000 years hence - the Milky Way being about 100K lightyears across - really to be valued at a discount of 10-2,227 relative to our own little planet today?

And when it comes to artificial general intelligence... I encounter wannabe AGI-makers who say, "Well, I don't know how to make my math work for an infinite time horizon, so... um... I've got it! I'll build an AGI whose planning horizon cuts out in a thousand years." Such a putative AGI would be quite happy to take an action that causes the galaxy to explode, so long as the explosion happens at least 1,001 years later. (In general, I've observed that most wannabe AGI researchers confronted with Singularity-level problems ponder for ten seconds and then propose the sort of clever programming trick one would use for data-mining the Netflix Prize, without asking if it makes deep sense for Earth-originating civilization over the next million years.)

The discount question is an old debate in economics, I know. I'm writing this blog post just now, because I recently had a conversation with Carl Shulman, who proposed an argument against temporal discounting that is, as far as I know, novel: namely that an AI with a 5% temporal discount rate has a nearly infinite incentive to expend all available resources on attempting time travel - maybe hunting for wormholes with a terminus in the past.

Or to translate this back out of transhumanist discourse: If you wouldn't burn alive 1,226,786,652 people today to save Giordano Bruno from the stake in 1600, then clearly, you do not have a 5%-per-year temporal discount rate in your pure preferences.

Maybe it's easier to believe in a temporal discount rate when you - the you of today - are the king of the hill, part of the most valuable class of persons in the landscape of present and future. But you wouldn't like it if there were other people around deemed more valuable than yourself, to be traded off against you. You wouldn't like a temporal discount if the past was still around.

Discrimination always seems more justifiable, somehow, when you're not the person who is discriminated against -

- but you will be.

(Just to make it clear, I'm not advocating against the idea that Treasury bonds can exist. But I am advocating against the idea that you should intrinsically care less about the future than the present; and I am advocating against the idea that you should compound a 5% discount rate a century out when you are valuing global catastrophic risk management.)

Original with comments: Against Discount Rates

Circular Altruism

Eliezer Yudkowsky, 22 January 2008 06:00PM

Followup to: Torture vs. Dust Specks, Zut Allais, Rationality Quotes 4

Suppose that a disease, or a monster, or a war, or something, is killing people. And suppose you only have enough resources to implement one of the following two options:

1. Save 400 lives, with certainty.

2. Save 500 lives, with 90% probability; save no lives, 10% probability.

Most people choose option 1. Which, I think, is foolish; because if you multiply 500 lives by 90% probability, you get an expected value of 450 lives, which exceeds the 400-life value of option 1. (Lives saved don't diminish in marginal utility, so this is an appropriate calculation.)

"What!" you cry, incensed. "How can you gamble with human lives? How can you think about numbers when so much is at stake? What if that 10% probability strikes, and everyone dies? So much for your damned logic! You're following your rationality off a cliff!"

Ah, but here's the interesting thing. If you present the options this way:

1. 100 people die, with certainty.

2. 90% chance no one dies; 10% chance 500 people die.

Then a majority choose option 2. Even though it's the same gamble. You see, just as a certainty of saving 400 lives seems to feel so much more comfortable than an unsure gain, so too, a certain loss feels worse than an uncertain one.

You can grandstand on the second description too: "How can you condemn 100 people to certain death when there's such a good chance you can save them? We'll all share the risk! Even if it was only a 75% chance of saving everyone, it would still be worth it - so long as there's a chance - everyone makes it, or no one does!"

You know what? This isn't about your feelings. A human life, with all its joys and all its pains, adding up over the course of decades, is worth far more than your brain's feelings of comfort or discomfort with a plan. Does computing the expected utility feel too cold-blooded for your taste? Well, that feeling isn't even a feather in the scales, when a life is at stake. Just shut up and multiply.

Previously on Overcoming Bias, I asked what was the least bad, bad thing that could happen, and suggested that it was getting a dust speck in your eye that irritated you for a fraction of a second, barely long enough to notice, before it got blinked away. And conversely, a very bad thing to happen, if not the worst thing, would be getting tortured for 50 years.

Now, would you rather that a googolplex people got dust specks in their eyes, or that one person was tortured for 50 years? I originally asked this question with a vastly larger number - an incomprehensible mathematical magnitude - but a googolplex works fine for this illustration.

Most people chose the dust specks over the torture. Many were proud of this choice, and indignant that anyone should choose otherwise: "How dare you condone torture!"

This matches research showing that there are "sacred values", like human lives, and "unsacred values", like money. When you try to trade off a sacred value against an unsacred value, subjects express great indignation (sometimes they want to punish the person who made the suggestion).

My favorite anecdote along these lines - though my books are packed at the moment, so no citation for now - comes from a team of researchers who evaluated the effectiveness of a certain project, calculating the cost per life saved, and recommended to the government that the project be implemented because it was cost-effective. The governmental agency rejected the report because, they said, you couldn't put a dollar value on human life. After rejecting the report, the agency decided not to implement the measure.

Trading off a sacred value (like refraining from torture) against an unsacred value (like dust specks) feels really awful. To merely multiply utilities would be too cold-blooded - it would be following rationality off a cliff...

But let me ask you this. Suppose you had to choose between one person being tortured for 50 years, and a googol people being tortured for 49 years, 364 days, 23 hours, 59 minutes and 59 seconds. You would choose one person being tortured for 50 years, I do presume; otherwise I give up on you.

And similarly, if you had to choose between a googol people tortured for 49.9999999 years, and a googol-squared people being tortured for 49.9999998 years, you would pick the former.

A googolplex is ten to the googolth power. That's a googol/100 factors of a googol. So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort, and multiplying by a factor of a googol each time, until we choose between a googolplex people getting a dust speck in their eye, and a googolplex/googol people getting two dust specks in their eye.

If you find your preferences are circular here, that makes rather a mockery of moral grandstanding. If you drive from San Jose to San Francisco to Oakland to San Jose, over and over again, you may have fun driving, but you aren't going anywhere. Maybe you think it a great display of virtue to choose for a googolplex people to get dust specks rather than one person being tortured. But if you would also trade a googolplex people getting one dust speck for a googolplex/googol people getting two dust specks et cetera, you sure aren't helping anyone. Circular preferences may work for feeling noble, but not for feeding the hungry or healing the sick.

Altruism isn't the warm fuzzy feeling you get from being altruistic. If you're doing it for the spiritual benefit, that is nothing but selfishness. The primary thing is to help others, whatever the means. So shut up and multiply!

And if it seems to you that there is a fierceness to this maximization, like the bare sword of the law, or the burning of the sun - if it seems to you that at the center of this rationality there is a small cold flame -

Well, the other way might feel better inside you. But it wouldn't work.

And I say also this to you: That if you set aside your regret for all the spiritual satisfaction you could be having - if you wholeheartedly pursue the Way, without thinking that you are being cheated - if you give yourself over to rationality without holding back, you will find that rationality gives to you in return.

But that part only works if you don't go around saying to yourself, "It would feel better inside me if only I could be less rational."

Chimpanzees feel, but they don't multiply. Should you be sad that you have the opportunity to do better? You cannot attain your full potential if you regard your gift as a burden.

Added: If you'd still take the dust specks, see Unknown's comment on the problem with qualitative versus quantitative distinctions.

Referenced by: The "Intuitions" Behind "Utilitarianism" • Something to Protect • Optimization • (Moral) Truth in Fiction? • Purchase Fuzzies and Utilons Separately • The End (of Sequences)

Original with comments: Circular Altruism

Rationality Quotes 5

Eliezer Yudkowsky, 23 January 2008 06:00PM

If you're seeing this, it means that I've moved, but that my Internet access isn't set up yet. I've set up quotes to be posted automatically for the next few days. Don't be surprised if I don't respond to comments!

"Perfection is our goal. Excellence will be tolerated."
-- J. Yahl

"Morality is objective within a given frame of reference."
-- Gordon Worley

"If there were a verb meaning "to believe falsely," it would not have any significant first person, present indicative."
-- Ludwig Wittgenstein

"It takes 50 years for your parents to mature enough that they see you as an independent person."
-- Ralph Lewis

"Impatience is a flaw. There's always just enough time when you do something right, no more, no less. Your sword has no blade. It has only your intention. When that goes astray you have no weapon."
-- C.J. Cherryh, The Paladin

"They're space cannibals. They only eat other space cannibals. Q.E.D."
-- Nikolai Kingsley

"My father said whoever tells the longest story is always the liar. The truth isn't that complicated."
-- Bill Joy, cofounder and Chief Scientist of Sun Microsystems

"My experience tells me that in this complicated world the simplest explanation is usually dead wrong. But I've noticed that the simplest explanation usually sounds right and is far more convincing than any complicated explanation could hope to be."
-- Scott Adams, cartoonist

"There are really good and thoroughly bad people on each side in all wars."
"Nothing is more truly horrifying than the limits of human behavior."
"Determined efforts are better than a miracle."
"Things only get weirder the longer they go on."
"People will love you for who you are ... as long as you're secretly a super-hero."
"Lack of communication leads to 90% of all problems. The other causes being 5% magic and 5% giant robots."
-- Tonbo, Things I've Learned From Anime

"And I heard a voice saying "Give up! Give up!" And that really scared me 'cause it sounded like Ben Kenobi."
-- Rebel Pilot's Lament

"If you're a literary critic, keep in mind that I hate you, too, and I said it first."
-- Scott Adams

"In any war there are always more of the enemy than you think, and there are always allies you never knew you had."
-- John M. Ford, Web of Angels

"He told her about the Afterglow: that brief, brilliant period after the Big Bang, when matter gathered briefly in clumps and burned by fusion light."
-- Stephen Baxter, The Gravity Mine

Original with comments: Rationality Quotes 5

Rationality Quotes 6

Eliezer Yudkowsky, 24 January 2008 06:00PM

"Gnostic: Knowledge that is so pure that it cannot be explained or proven wrong."
-- Glossary of Zen

"I haven't been wrong since 1961, when I thought I made a mistake."
-- Bob Hudson

"Sometimes I envy that groundless confidence."
-- Vandread

"I remember when I first learned the word 'omniscient'. I thought that was a reasonable goal to aim for."
-- Samantha Atkins

"Our language for describing emotions is very crude... that's what music is for, I guess."
-- Ben Goertzel

"How much money would it take to get YOU to change your sexual orientation? Now add a dollar to that. That's how much Disney could offer you."
-- Leader Kibo

"I don't need The Media to tell me that I should be outraged about a brutal murder. All I need is to be informed that it has happened, and I'll form my own opinion about it."
-- The_Morlock

"Now Charlie, don't forget what happened to the man who suddenly got everything he wished for."
"What?"
"He lived happily ever after."
-- Roald Dahl, Willy Wonka and the Chocolate Factory

"He was never at ease with politics, where good and bad were just, apparently, two ways of looking at the same thing or, at least, were described like that by the people who were on the side Vimes thought of as 'bad'."
-- Terry Pratchett, "The Fifth Elephant"

"Philosophy is the art of asking the wrong questions."
-- J.R. Molloy

"The future is coming, Bill Joy. Like a juggernaut. There'll be no slowing it down. A billion people in India (substantial numbers of programmers riding the wave of the silicon revolution) and more than a billion people in China want a decent standard of living and they're counting on future technologies to deliver it for them. To slow down the future, you would have to nuke them. Are you going to nuke them, Bill? I didn't think so."
-- Jeff Davis

"Yikes, we may need to revise the Wechsler-Bellevue to accomodate some hitherto unrecognized nadir of the intelligence quotient scale."
-- Tom Morse

"Adultery always begins with the adulterer(s) claiming to themselves and to others that the relationship is "harmless" because it hasn't crossed a certain line. The line where it becomes wrong is the line where you start having to rationalize like that."
-- Gelfin

Original with comments: Rationality Quotes 6

Rationality Quotes 7

Eliezer Yudkowsky, 25 January 2008 06:00PM

"People don't buy three-eighths-inch drill bits. People buy three-eighths-inch holes."
-- Michael Porter

"I feel more like I do now than I did a while ago."
-- Arifel

"Know thyself, because in the end there's no one else."
-- Living Colour, Solace of You

"Student motivation? I'm gonna start hooking you all up to electrodes."
-- Kevin Giffhorn

"A rock pile ceases to be a rock pile the moment a single man contemplates it, bearing within him the image of a cathedral."
-- Antoine de Saint-Exupery

"I found one day in school a boy of medium size ill-treating a smaller boy. I expostulated, but he replied: 'The bigs hit me, so I hit the babies; that's fair.' In these words he epitomized the history of the human race."
-- Bertrand Russell

"Open Source Software: There are days when I can't figure out whether I'm living in a Socialist utopia or a Libertarian one."
-- Alex Future Bokov

"Supposing you got a crate of oranges that you opened, and you found all the top layer of oranges bad, you would not argue, `The underneath ones must be good, so as to redress the balance.' You would say, `Probably the whole lot is a bad consignment'; and that is really what a scientific person would say about the universe."
-- Bertrand Russell

"This thing not only has all the earmarks of a hoax, it has "HOAX" branded into its flank and it's regulated by the U.S. Department of Hoaxes, Pranks and Shenanigans."
-- Brunching Shuttlecocks

"Natural selection gave a big advantage to those who were good at spotting the pattern of a saber toothed tiger hiding in the bushes, but no advantage to those who were good at solving partial differential equations. It is not mere rhetoric to say that in an absolute sense a janitor has a more intellectually challenging job than a professor of mathematics."
-- John K. Clark

"The simple fact is that non-violent means do not work against Evil. Gandhi's non-violent resistance against the British occupiers had some effect because Britain was wrong, but not Evil. The same is true of the success of non-violent civil rights resistance against de jure racism. Most people, including those in power, knew that what was being done was wrong. But Evil is an entirely different beast. Gandhi would have gone to the ovens had he attempted non-violent resistance against the Nazis. When one encounters Evil, the only solution is violence, actual or threatened. That's all Evil understands."
-- Robert Bruce Thompson

Original with comments: Rationality Quotes 7

Rationality Quotes 8

Eliezer Yudkowsky, 26 January 2008 06:00PM

"Like a lot of people in the computer industry, Keith Malinowski had spent his whole life being the smartest person in the room, and like most of his fellows the experience left him with a rather high opinion of his opinions."
-- Rick Cook, The Wizardry Quested

"The fact that we can become accustomed to anything, however disgusting at first, makes it necessary to examine carefully everything we have become accustomed to."
-- George Bernard Shaw

"Beware `we should...', extend a hand to `how do I...'"
-- Alan Cox

"I assign higher credibility to an institution if liberals accuse it of being conservative and conservatives accuse it of being liberal."
-- Alex F. Bokov

"An open mind is a great place for other people to dump their garbage."
-- Rev. Rock, Church of the Subgenius

"A.D. 1517: Martin Luther nails his 95 Theses to the church door and is promptly moderated down to (-1, Flamebait)."
-- Yu Suzuki

"A committee cannot be wrong - only divided. Once it resolves its division, then every part reinforces every other part and its rightness becomes unassailable."
-- John M. Ford, The Princes of the Air

"Gee, thanks for the elementary lesson in metaphysics. And there I thought Being was an undifferentiated unity."
-- Mark Walker

"The old political syllogism "something must be done: this is something: therefore this will be done" appears to be at work here, in spades."
-- Charlie Stross

"The fact that I beat a drum has nothing to do with the fact that I do theoretical physics. Theoretical physics is a human endeavour, one of the higher developments of human beings — and this perpetual desire to prove that people who do it are human by showing that they do other things that a few other humans do (like playing bongo drums) is insulting to me. I am human enough to tell you to go to hell."
-- Richard Feynman

"Java sucks. Java on TV set top boxes will suck so hard it might well inhale people from off their sofa until their heads get wedged in the card slots."
-- Jon Rabone

"Susan was bright enough to know that the phrase "Someone ought to do something" was not, by itself, a helpful one. People who used it never added the rider "and that someone is me." But someone ought to do something, and right now the whole pool of someones consisted of her, and no one else."
-- Terry Pratchett, Hogfather

Original with comments: Rationality Quotes 8

Rationality Quotes 9

Eliezer Yudkowsky, 27 January 2008 06:00PM

"A world ought to have a few genuine good guys, and not just a spectrum of people running from bad to worse."
-- Glen Cook, A Shadow of All Night Falling

"You couldn't get a clue during the clue mating season in a field full of horny clues if you smeared your body with clue musk and did the clue mating dance."
-- Edward Flaherty

"We all enter this world in the same way: naked; screaming; soaked in blood. But if you live your life right, that kind of thing doesn't have to stop there."
-- Dana Gould

"Love is a snowmobile racing across the tundra and then suddenly it flips over, pinning you underneath. At night, the ice weasels come."
-- Matt Groening

"Things do get better, all the time, maybe just not as fast as I'd like. I do what I can. Don't ask me to hate, too."
-- Michael Wiik

"Political or military commentators, like astrologers, can survive almost any mistake, because their more devoted followers do not look to them for an appraisal of the facts but for the stimulation of nationalistic loyalties."
-- George Orwell, Notes on Nationalism

"People are always amazed by how much "free time" I have.
They're also amazed that I don't know who Ally McBeal is.
Frankly, I'm amazed that they can't make the connection."
-- Robert Wenzlaff

"Throughout the technology revolution, mankind has consistently sought to improve life by reducing the number of tasks that require physical activity, then sought to counteract the inevitable weight gain by manufacturing food that only looks like food and barely tastes like it."
-- Samuel Stoddard

"Any person who claims to have deep feeling for other human beings should think a long, long time before he votes to have other men kept behind bars - caged. I am not saying there shouldn't be prisons, but there shouldn't be bars. Behind bars, a man never reforms. He will never forget. He never will get completely over the memory of the bars."
-- Malcolm X

"What funding committee will agree to fund a book describing an entire new field that has never before been dreamt of? Committees base their conclusions on a shared understanding of a common body of knowledge. Their members are drawn from an existing society of experts to evaluate the next incremental improvement. What do you do when there are no experts? Who lays claim to expertise in nanomedicine? Who has spent their life in this field which is just being conceived? No one. The committee process breaks down when we move into truly new terrain. It fails us just when failure is most expensive: at the beginnings of new things. Here we must fall back on individuals - individuals who are bold enough to believe in themselves when there are no experts to turn to for help and support. Individuals who are willing to back up their own beliefs with action, who will nurture the truly new and the truly groundbreaking without having to first seek the approval of others. For there are no others! On the far frontiers there are very few, and sometimes there is only one."
-- Ralph Merkle, afterword to Nanomedicine

Original with comments: Rationality Quotes 9

The "Intuitions" Behind "Utilitarianism"

Eliezer Yudkowsky, 28 January 2008 04:29PM

Followup to: Circular Altruism. Response to: Knowing your argumentative limitations, OR "one [rationalist's] modus ponens is another's modus tollens."

(Still no Internet access. Hopefully they manage to repair the DSL today.)

I haven't said much about metaethics - the nature of morality - because that has a forward dependency on a discussion of the Mind Projection Fallacy that I haven't gotten to yet. I used to be very confused about metaethics. After my confusion finally cleared up, I did a postmortem on my previous thoughts. I found that my object-level moral reasoning had been valuable and my meta-level moral reasoning had been worse than useless. And this appears to be a general syndrome - people do much better when discussing whether torture is good or bad than when they discuss the meaning of "good" and "bad". Thus, I deem it prudent to keep moral discussions on the object level wherever I possibly can.

Occasionally people object to any discussion of morality on the grounds that morality doesn't exist, and in lieu of jumping over the forward dependency to explain that "exist" is not the right term to use here, I generally say, "But what do you do anyway?" and take the discussion back down to the object level.

Paul Gowder, though, has pointed out that both the idea of choosing a googolplex dust specks in a googolplex eyes over 50 years of torture for one person, and the idea of "utilitarianism", depend on "intuition". He says I've argued that the two are not compatible, but charges me with failing to argue for the utilitarian intuitions that I appeal to.

Now "intuition" is not how I would describe the computations that underlie human morality and distinguish us, as moralists, from an ideal philosopher of perfect emptiness and/or a rock. But I am okay with using the word "intuition" as a term of art, bearing in mind that "intuition" in this sense is not to be contrasted to reason, but is, rather, the cognitive building block out of which both long verbal arguments and fast perceptual arguments are constructed.

I see the project of morality as a project of renormalizing intuition. We have intuitions about things that seem desirable or undesirable, intuitions about actions that are right or wrong, intuitions about how to resolve conflicting intuitions, intuitions about how to systematize specific intuitions into general principles.

Delete all the intuitions, and you aren't left with an ideal philosopher of perfect emptiness, you're left with a rock.

Keep all your specific intuitions and refuse to build upon the reflective ones, and you aren't left with an ideal philosopher of perfect spontaneity and genuineness, you're left with a grunting caveperson running in circles, due to cyclical preferences and similar inconsistencies.

"Intuition", as a term of art, is not a curse word when it comes to morality - there is nothing else to argue from. Even modus ponens is an "intuition" in this sense - it's just that modus ponens still seems like a good idea after being formalized, reflected on, extrapolated out to see if it has sensible consequences, etcetera.

So that is "intuition".

However, Gowder did not say what he meant by "utilitarianism". Does utilitarianism say...

1. That right actions are strictly determined by good consequences?

2. That praiseworthy actions depend on justifiable expectations of good consequences?

3. That probabilities of consequences should normatively be discounted by their probability, so that a 50% probability of something bad should weigh exactly half as much in our tradeoffs?

4. That virtuous actions always correspond to maximizing expected utility under some utility function?

5. That two harmful events are worse than one?

6. That two independent occurrences of a harm (not to the same person, not interacting with each other) are exactly twice as bad as one?

7. That for any two harms A and B, with A much worse than B, there exists some tiny probability such that gambling on this probability of A is preferable to a certainty of B?

If you say that I advocate something, or that my argument depends on something, and that it is wrong, do please specify what this thingy is... anyway, I accept 3, 5, 6, and 7, but not 4; I am not sure about the phrasing of 1; and 2 is true, I guess, but phrased in a rather solipsistic and selfish fashion: you should not worry about being praiseworthy.

Now, what are the "intuitions" upon which my "utilitarianism" depends?

This is a deepish sort of topic, but I'll take a quick stab at it.

First of all, it's not just that someone presented me with a list of statements like those above, and I decided which ones sounded "intuitive". Among other things, if you try to violate "utilitarianism", you run into paradoxes, contradictions, circular preferences, and other things that aren't symptoms of moral wrongness so much as moral incoherence.

After you think about moral problems for a while, and also find new truths about the world, and even discover disturbing facts about how you yourself work, you often end up with different moral opinions than when you started out. This does not quite define moral progress, but it is how we experience moral progress.

As part of my experienced moral progress, I've drawn a conceptual separation between questions of type Where should we go? and questions of type How should we get there? (Could that be what Gowder means by saying I'm "utilitarian"?)

The question of where a road goes - where it leads - you can answer by traveling the road and finding out. If you have a false belief about where the road leads, this falsity can be destroyed by the truth in a very direct and straightforward manner.

When it comes to wanting to go to a particular place, this want is not entirely immune from the destructive powers of truth. You could go there and find that you regret it afterward (which does not define moral error, but is how we experience moral error).

But, even so, wanting to be in a particular place seems worth distinguishing from wanting to take a particular road to a particular place.

Our intuitions about where to go are arguable enough, but our intuitions about how to get there are frankly messed up. After the two hundred and eighty-seventh research study showing that people will chop their own feet off if you frame the problem the wrong way, you start to distrust first impressions.

When you've read enough research on scope insensitivity - people will pay only 28% more to protect all 57 wilderness areas in Ontario than one area, people will pay the same amount to save 50,000 lives as 5,000 lives... that sort of thing...

Well, the worst case of scope insensitivity I've ever heard of was described here by Slovic:

Other recent research shows similar results. Two Israeli psychologists asked people to contribute to a costly life-saving treatment. They could offer that contribution to a group of eight sick children, or to an individual child selected from the group. The target amount needed to save the child (or children) was the same in both cases. Contributions to individual group members far outweighed the contributions to the entire group.

There's other research along similar lines, but I'm just presenting one example, 'cause, y'know, eight examples would probably have less impact.

If you know the general experimental paradigm, then the reason for the above behavior is pretty obvious - focusing your attention on a single child creates more emotional arousal than trying to distribute attention around eight children simultaneously. So people are willing to pay more to help one child than to help eight.

Now, you could look at this intuition, and think it was revealing some kind of incredibly deep moral truth which shows that one child's good fortune is somehow devalued by the other children's good fortune.

But what about the billions of other children in the world? Why isn't it a bad idea to help this one child, when that causes the value of all the other children to go down? How can it be significantly better to have 1,329,342,410 happy children than 1,329,342,409, but then somewhat worse to have seven more at 1,329,342,417?

Or you could look at that and say: "The intuition is wrong: the brain can't successfully multiply by eight and get a larger quantity than it started with. But it ought to, normatively speaking."

And once you realize that the brain can't multiply by eight, then the other cases of scope neglect stop seeming to reveal some fundamental truth about 50,000 lives being worth just the same effort as 5,000 lives, or whatever. You don't get the impression you're looking at the revelation of a deep moral truth about nonagglomerative utilities. It's just that the brain doesn't goddamn multiply. Quantities get thrown out the window.

If you have $100 to spend, and you spend $20 each on each of 5 efforts to save 5,000 lives, you will do worse than if you spend $100 on a single effort to save 50,000 lives. Likewise if such choices are made by 10 different people, rather than the same person. As soon as you start believing that it is better to save 50,000 lives than 25,000 lives, that simple preference of final destinations has implications for the choice of paths, when you consider five different events that save 5,000 lives.

(It is a general principle that Bayesians see no difference between the long-run answer and the short-run answer; you never get two different answers from computing the same question two different ways. But the long run is a helpful intuition pump, so I am talking about it anyway.)

The aggregative valuation strategy of "shut up and multiply" arises from the simple preference to have more of something - to save as many lives as possible - when you have to describe general principles for choosing more than once, acting more than once, planning at more than one time.

Aggregation also arises from claiming that the local choice to save one life doesn't depend on how many lives already exist, far away on the other side of the planet, or far away on the other side of the universe. Three lives are one and one and one. No matter how many billions are doing better, or doing worse. 3 = 1 + 1 + 1, no matter what other quantities you add to both sides of the equation. And if you add another life you get 4 = 1 + 1 + 1 + 1. That's aggregation.

When you've read enough heuristics and biases research, and enough coherence and uniqueness proofs for Bayesian probabilities and expected utility, and you've seen the "Dutch book" and "money pump" effects that penalize trying to handle uncertain outcomes any other way, then you don't see the preference reversals in the Allais Paradox as revealing some incredibly deep moral truth about the intrinsic value of certainty. It just goes to show that the brain doesn't goddamn multiply.

The primitive, perceptual intuitions that make a choice "feel good" don't handle probabilistic pathways through time very skillfully, especially when the probabilities have been expressed symbolically rather than experienced as a frequency. So you reflect, devise more trustworthy logics, and think it through in words.

When you see people insisting that no amount of money whatsoever is worth a single human life, and then driving an extra mile to save $10; or when you see people insisting that no amount of money is worth a decrement of health, and then choosing the cheapest health insurance available; then you don't think that their protestations reveal some deep truth about incommensurable utilities.

Part of it, clearly, is that primitive intuitions don't successfully diminish the emotional impact of symbols standing for small quantities - anything you talk about seems like "an amount worth considering".

And part of it has to do with preferring unconditional social rules to conditional social rules. Conditional rules seem weaker, seem more subject to manipulation. If there's any loophole that lets the government legally commit torture, then the government will drive a truck through that loophole.

So it seems like there should be an unconditional social injunction against preferring money to life, and no "but" following it. Not even "but a thousand dollars isn't worth a 0.0000000001% probability of saving a life". Though the latter choice, of course, is revealed every time we sneeze without calling a doctor.

The rhetoric of sacredness gets bonus points for seeming to express an unlimited commitment, an unconditional refusal that signals trustworthiness and refusal to compromise. So you conclude that moral rhetoric espouses qualitative distinctions, because espousing a quantitative tradeoff would sound like you were plotting to defect.

On such occasions, people vigorously want to throw quantities out the window, and they get upset if you try to bring quantities back in, because quantities sound like conditions that would weaken the rule.

But you don't conclude that there are actually two tiers of utility with lexical ordering. You don't conclude that there is actually an infinitely sharp moral gradient, some atom that moves a Planck distance (in our continuous physical universe) and sends a utility from 0 to infinity. You don't conclude that utilities must be expressed using hyper-real numbers. Because the lower tier would simply vanish in any equation. It would never be worth the tiniest effort to recalculate for it. All decisions would be determined by the upper tier, and all thought spent thinking about the upper tier only, if the upper tier genuinely had lexical priority.

As Peter Norvig once pointed out, if Asimov's robots had strict priority for the First Law of Robotics ("A robot shall not harm a human being, nor through inaction allow a human being to come to harm") then no robot's behavior would ever show any sign of the other two Laws; there would always be some tiny First Law factor that would be sufficient to determine the decision.

Whatever value is worth thinking about at all, must be worth trading off against all other values worth thinking about, because thought itself is a limited resource that must be traded off. When you reveal a value, you reveal a utility.

I don't say that morality should always be simple. I've already said that the meaning of music is more than happiness alone, more than just a pleasure center lighting up. I would rather see music composed by people than by nonsentient machine learning algorithms, so that someone should have the joy of composition; I care about the journey, as well as the destination. And I am ready to hear if you tell me that the value of music is deeper, and involves more complications, than I realize - that the valuation of this one event is more complex than I know.

But that's for one event. When it comes to multiplying by quantities and probabilities, complication is to be avoided - at least if you care more about the destination than the journey. When you've reflected on enough intuitions, and corrected enough absurdities, you start to see a common denominator, a meta-principle at work, which one might phrase as "Shut up and multiply."

Where music is concerned, I care about the journey.

When lives are at stake, I shut up and multiply.

It is more important that lives be saved, than that we conform to any particular ritual in saving them. And the optimal path to that destination is governed by laws that are simple, because they are math.

And that's why I'm a utilitarian - at least when I am doing something that is overwhelmingly more important than my own feelings about it - which is most of the time, because there are not many utilitarians, and many things left undone.

</rant>

Referenced by: Something to Protect • Timeless Identity • Living in Many Worlds

Original with comments: The "Intuitions" Behind "Utilitarianism"

Trust in Bayes

Eliezer Yudkowsky, 29 January 2008 11:12PM

Followup to: Beautiful Probability, Trust in Math

In Trust in Math, I presented an algebraic proof that 1 = 2, which turned out to be - surprise surprise - flawed. Trusting that algebra, correctly used, will not carry you to an absurd result, is not a matter of blind faith. When we see apparent evidence against algebra's trustworthiness, we should also take into account the massive evidence favoring algebra which we have previously encountered. We should take into account our past experience of seeming contradictions which turned out to be themselves flawed. Based on our inductive faith that we may likely have a similar experience in the future, we look for a flaw in the contrary evidence.

This seems like a dangerous way to think, and it is dangerous, as I noted in "Trust in Math". But, faced with a proof that 2 = 1, I can't convince myself that it's genuinely reasonable to think any other way.

The novice goes astray and says, "The Art failed me."
The master goes astray and says, "I failed my Art."

To get yourself to stop saying "The Art failed me", it's helpful to know the history of people crying wolf on Bayesian math - to be familiar with seeming paradoxes that have been discovered and refuted. Here an invaluable resource is "Paradoxes of Probability Theory", Chapter 15 of E. T. Jaynes's Probability Theory: The Logic of Science (available online).

I'll illustrate with one of Jaynes's examples:

If you have a conditional probability distribution P(X|C), the unconditional probability P(X) should be a weighted average of the various P(X|C), and therefore intermediate between the various P(X|C) in value - somewhere between the minimum and the maximum P(X|C). That is: If you flip a coin before rolling a die, and the die is a four-sided die if the coin comes up heads, or ten-sided if the coin comes up tails, then (even without doing an exact calculation) you know that the compound probability of rolling a "1" occupies the range [0.1, 0.25].

Now suppose a two-dimensional array, M cells wide and N cells tall, with positions written (i, j) with i as the horizontal coordinate and j as the vertical coordinate. And suppose a uniform probability distribution over the array: p(i, j) = 1/MN for all i, j. Finally, let X be the event that i < j. We'll be asking about P(X).

If we think about just the top row - that is, condition on the information j=N - then the probability of X is p(i < N) = (N - 1)/M, or 1 if N > M.

If we think about just the bottom row - condition on the information j=1 - then the probability of X is p(i < 1) = 0.

Similarly, if we think about just the rightmost column, condition on i=M, then the probability of X is p(j > M) = (N - M)/N, or 0 if M > N.

And thinking about the leftmost column, conditioning on i=1, the probability of X is p(j > 1) = (N - 1)/N.

So for the whole array, the probability of X must be between (N - 1)/M and 0 (by reasoning about rows) or between (N - 1)/N and (N - M)/N (by reasoning about columns).

This is actually correct, so no paradox so far. If the array is 5 x 7, then the probability of X on the top row is 1, the probability of X on the bottom row is 0. The probability of X in the rightmost row is 2/7, and the probability of X in the leftmost row is 6/7. The probability of X over the whole array is 4/7, which obeys both constraints.

But now suppose that the array is infinite. Reasoning about the rows, we see that, for every row, there is a finite number of points where i < j, and an infinite number of points where i >= j. So for every row, the probability of the event X must be 0. Reasoning about the columns, we see in every column a finite number of points where j <= i, and an infinite number of points where i < j. So for every column, the probability of the event X must be 1. This is a paradox, since the compound probability of X must be both a weighted mix of the probability for each row, and a weighted mix of the probability for each column.

If to you this seems like a perfectly reasonable paradox, then you really need to read Jaynes's "Paradoxes of Probability Theory" all the way through.

In "paradoxes" of algebra, there is always an illegal operation that produces the contradiction. For algebraic paradoxes, the illegal operation is usually a disguised division by zero. For "paradoxes" of probability theory and decision theory, the illegal operation is usually assuming an infinity that has not been obtained as the limit of a finite calculation.

In the case above, the limiting probability of i < j approaches the ratio M / N, so just assuming that M and N are "infinite" will naturally produce all sorts of paradoxes in the ratio - it depends on how M and N approach infinity.

It's all too tempting to just talk about infinities, instead of constructing them. As Jaynes observes, this is a particularly pernicious habit because it may work 95% of the time and then lead you into trouble on the last 5% of occasions - like how the really deadly bugs in a computer program are not those that appear all the time, but those that only appear 1% of the time.

Apparently there was a whole cottage industry in this kind of paradox, where, assuming infinite sets, the marginal probability seemed not to be inside the conditional probabilities of some partition, and this was called "nonconglomerability". Jaynes again:

"Obviously, nonconglomerability cannot arise from a correct application of the rules of probability on finite sets. It cannot, therefore, occur in an infinite set which is approached as a well-defined limit of a sequence of finite sets. Yet nonconglomerability has become a minor industry, with a large and growing literature. There are writers who believe that it is a real phenomenon, and that they are proving theorems about the circumstances in which it occurs, which are important for the foundations of probability theory."

We recently ran into a similar problem here on Overcoming Bias: A commenter cited a paper, "An Air-Tight Dutch Book" by Vann McGee, which purports to show that if your utility function is not bounded, then a dutch book can be constructed against you. The paper is gated, but Neel Krishnaswami passed me a copy. A summary of McGee's argument can also be found in the ungated paper "Bayesianism, infinite decisions, and binding".

Rephrasing somewhat, McGee's argument goes as follows. Suppose that you are an expected utility maximizer and your utility function is unbounded in some quantity, such as human lives or proving math theorems. We'll write $27 to indicate a quantity worth 27 units of utility; by the hypothesis of an unbounded utility function, you can always find some amount of fun that is worth at least 27 units of utility (where the reference unit can be any positive change in the status quo).

Two important notes are that, (1) this does not require your utility function to be linear in anything, just that it grow monotonically and without bound; and (2) your utility function does not have to assign infinite utility to any outcome, just ever-larger finite utilities to ever-larger finite outcomes.

Now for the seeming Dutch Book - a sequence of bets that (McGee argues) a Bayesian will take, but which produce a guaranteed loss.

McGee produces a fair coin, and proceeds to offer us the following bet: We lose $1 (one unit of utility) if the coin comes up "tails" on the first round, and gain $3 (three units of utility) if the coin comes up "heads" on the first round and "tails" on the second round. Otherwise nothing happens - the bet has no payoff. The probability of the first outcome in the bet is 1/2, so it has an expected payoff of -$.50; and the probability of the second outcome is 1/4, so it has an expected payoff of +$.75. All other outcomes have no payoff, so the net value is +$0.25. We take the bet.

Now McGee offers us a second bet, which loses $4 if the coin first comes up "tails" on the second round, but pays $9 if the coin first comes up "tails" on the third round, with no consequence otherwise. The probabilities of a fair coin producing a sequence that begins HT or HHT are respectively 1/4 and 1/8, so the expected values are -$1.00 and +$1.125. The net expectation is positive, so we take this bet as well.

Then McGee offers us a third bet which loses $10 if the coin first comes up "tails" on the third round, but gains $21 if the coin first comes up "tails" on the fourth round; then a bet which loses $22 if the coin shows "tails" first on round 4 and gains $45 if the coin shows "tails" first on round 5. Etc.

If we accept all these bets together, then we lose $1 no matter when the coin first shows "tails". So, McGee says, we have accepted a Dutch Book. From which McGee argues that every rational mind must have a finite upper bound on its utility function.

Now, y'know, there's a number of replies I could give to this. I won't challenge the possibility of the game, which would be my typical response as an infinite set atheist, because I never actually encounter an infinity. I can imagine living in a universe where McGee actually does have the ability to increase his resources exponentially, and so could actually offer me that series of bets.

But if McGee is allowed to deploy a scenario where the expected value of the infinite sequence does not equal the limit of the expected values of the finite sequences, then why should a good Bayesian's decision in the infinite case equal the limit of the Bayesian's decisions in the finite cases?

It's easy to demonstrate that for every finite N, a Bayesian will accept the first N bets. It's also easy to demonstrate that for every finite N, accepting the first N bets has a positive expected payoff. The decision in every finite scenario is to accept all N bets - from which you might say that the "limit" decision is to accept all offered bets - and the limit of the expected payoffs of these finite decisions goes to +$.50.

But now McGee wants to talk about the infinite scenario directly, rather than as a limiting strategy that applies to any one of a series of finite scenarios. Jaynes would not let you get away with this at all, but I accept that I might live in an unbounded universe and I might just have to shut up and deal with infinite games. Well, if so, the expected payoff of the infinite scenario does not equal the limit of the expected payoffs of the finite scenarios. One equals -$1, the other equals +$.50.

So there is no particular reason why the rational decision in the infinite scenario should equal the limit of the rational decisions in the finite scenarios, given that the payoff in the infinite scenario does not equal the limit of the payoffs in the finite scenarios.

And from this, McGee wants to deduce that all rational entities must have bounded utility functions? If it turns out that I live in an infinite universe, you can bet that there isn't any positive real number such that I would decline to have more fun than that.

Arntzenius, Elga, and Hawthorne give a more detailed argument in "Bayesianism, Infinite Decisions, and Binding" that the concept of provable dominance only applies to finite option sets and not infinite option sets. If you show me a compound planning problem with a sub-option X, such that for every possible compound plan I am better off taking X1 than X2, then this shows that a maximal plan must include X1 when the set of possible plans is finite. But when there is no maximal plan, no "optimal" decision - because there are an infinite number of possible plans whose upper bound (if any) isn't in the set - then proving local dominance obviously can't show anything about the "optimal" decision. See Arntzenius et. al's section on "Satan's Apple" for their full argument.

An even better version of McGee's scenario, in my opinion, would use a different sequence of bets: -$1 on the first round versus +$6 on the second round; -$6 on the second round and +$20 on the third round; -$20 on the third round and +$56 on the fourth round. Now we've picked the sequence so that if you accept all bets up to the Nth bet, your expected value is $N.

So really McGee's argument can be simplified as follows: Pick any positive integer, and I'll give you that amount of money. Clearly you shouldn't pick 1, because 1 is always inferior to 2 and above. Clearly you shouldn't pick 2, because it's always inferior to 3 and above. By induction, you shouldn't pick any number, so you don't get any money. So (McGee concludes) if you're really rational, there must be some upper bound on how much you care about anything.

(Actually, McGee's proposed upper bound doesn't really solve anything. Once you allow infinite times, you can be put into the same dilemma if I offer you $.50, but then offer to trade $.50 today for $.75 tomorrow, and then, tomorrow, offer to trade $.75 now for $.875 the next day, and so on. Even if my utility function is bounded at $1, this doesn't save me from problems where the limit of the payoffs of the finite plans doesn't seem to equal the payoff of the limit of the finite plans. See the comments for further arguments.)

The meta-moral is that Bayesian probability theory and decision theory are math: the formalism provably follows from axioms, and the formalism provably obeys those axioms. When someone shows you a purported paradox of probability theory or decision theory, don't shrug and say, "Well, I guess 2 = 1 in that case" or "Haha, look how dumb Bayesians are" or "The Art failed me... guess I'll resort to irrationality." Look for the division by zero; or the infinity that is assumed rather than being constructed as the limit of a finite operation; or the use of different implicit background knowledge in different parts of the calculation; or the improper prior that is not treated as the limit of a series of proper priors... something illegal.

Trust Bayes. Bayes has earned it.

Referenced by: Newcomb's Problem and Regret of Rationality • The Second Law of Thermodynamics, and Engines of Cognition • The Weighted Majority Algorithm

Original with comments: Trust in Bayes

Something to Protect

Eliezer Yudkowsky, 30 January 2008 05:52PM

Followup to: Tsuyoku Naritai, Circular Altruism

In the gestalt of (ahem) Japanese fiction, one finds this oft-repeated motif: Power comes from having something to protect.

I'm not just talking about superheroes that power up when a friend is threatened, the way it works in Western fiction. In the Japanese version it runs deeper than that.

In the X saga it's explicitly stated that each of the good guys draw their power from having someone - one person - who they want to protect. Who? That question is part of X's plot - the "most precious person" isn't always who we think. But if that person is killed, or hurt in the wrong way, the protector loses their power - not so much from magical backlash, as from simple despair. This isn't something that happens once per week per good guy, the way it would work in a Western comic. It's equivalent to being Killed Off For Real - taken off the game board.

The way it works in Western superhero comics is that the good guy gets bitten by a radioactive spider; and then he needs something to do with his powers, to keep him busy, so he decides to fight crime. And then Western superheroes are always whining about how much time their superhero duties take up, and how they'd rather be ordinary mortals so they could go fishing or something.

Similarly, in Western real life, unhappy people are told that they need a "purpose in life", so they should pick out an altruistic cause that goes well with their personality, like picking out nice living-room drapes, and this will brighten up their days by adding some color, like nice living-room drapes. You should be careful not to pick something too expensive, though.

In Western comics, the magic comes first, then the purpose: Acquire amazing powers, decide to protect the innocent. In Japanese fiction, often, it works the other way around.

Of course I'm not saying all this to generalize from fictional evidence. But I want to convey a concept whose deceptively close Western analogue is not what I mean.

I have touched before on the idea that a rationalist must have something they value more than "rationality": The Art must have a purpose other than itself, or it collapses into infinite recursion. But do not mistake me, and think I am advocating that rationalists should pick out a nice altruistic cause, by way of having something to do, because rationality isn't all that important by itself. No. I am asking: Where do rationalists come from? How do we acquire our powers?

It is written in the Twelve Virtues of Rationality:

How can you improve your conception of rationality? Not by saying to yourself, "It is my duty to be rational." By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, "The sky is green," and you look up at the sky and see blue. If you think: "It may look like the sky is blue, but rationality is to believe the words of the Great Teacher," you lose a chance to discover your mistake.

Historically speaking, the way humanity finally left the trap of authority and began paying attention to, y'know, the actual sky, was that beliefs based on experiment turned out to be much more useful than beliefs based on authority. Curiosity has been around since the dawn of humanity, but the problem is that spinning campfire tales works just as well for satisfying curiosity.

Historically speaking, science won because it displayed greater raw strength in the form of technology, not because science sounded more reasonable. To this very day, magic and scripture still sound more reasonable to untrained ears than science. That is why there is continuous social tension between the belief systems. If science not only worked better than magic, but also sounded more intuitively reasonable, it would have won entirely by now.

Now there are those who say: "How dare you suggest that anything should be valued more than Truth? Must not a rationalist love Truth more than mere usefulness?"

Forget for a moment what would have happened historically to someone like that - that people in pretty much that frame of mind defended the Bible because they loved Truth more than mere accuracy. Propositional morality is a glorious thing, but it has too many degrees of freedom.

No, the real point is that a rationalist's love affair with the Truth is, well, just more complicated as an emotional relationship.

One doesn't become an adept rationalist without caring about the truth, both as a purely moral desideratum and as something that's fun to have. I doubt there are many master composers who hate music.

But part of what I like about rationality is the discipline imposed by requiring beliefs to yield predictions, which ends up taking us much closer to the truth than if we sat in the living room obsessing about Truth all day. I like the complexity of simultaneously having to love True-seeming ideas, and also being ready to drop them out the window at a moment's notice. I even like the glorious aesthetic purity of declaring that I value mere usefulness above aesthetics. That is almost a contradiction, but not quite; and that has an aesthetic quality as well, a delicious humor.

And of course, no matter how much you profess your love of mere usefulness, you should never actually end up deliberately believing a useful false statement.

So don't oversimplify the relationship between loving truth and loving usefulness. It's not one or the other. It's complicated, which is not necessarily a defect in the moral aesthetics of single events.

But morality and aesthetics alone, believing that one ought to be "rational" or that certain ways of thinking are "beautiful", will not lead you to the center of the Way. It wouldn't have gotten humanity out of the authority-hole.

In Circular Altruism, I discussed this dilemma: Which of these options would you prefer:

1. Save 400 lives, with certainty

2. Save 500 lives, 90% probability; save no lives, 10% probability.

You may be tempted to grandstand, saying, "How dare you gamble with people's lives?" Even if you, yourself, are one of the 500 - but you don't know which one - you may still be tempted to rely on the comforting feeling of certainty, because our own lives are often worth less to us than a good intuition.

But if your precious daughter is one of the 500, and you don't know which one, then, perhaps, you may feel more impelled to shut up and multiply - to notice that you have an 80% chance of saving her in the first case, and a 90% chance of saving her in the second.

And yes, everyone in that crowd is someone's son or daughter. Which, in turn, suggests that we should pick the second option as altruists, as well as concerned parents.

My point is not to suggest that one person's life is more valuable than 499 people. What I am trying to say is that more than one's own life has to be at stake, before someone becomes desperate enough to override comfortable intuitions with math. It takes visceral panic, channeled through cold calculation, to cut away all the distractions.

What if you believe that it is "rational" to choose the certainty of option 1? Lots of people think that "rationality" is about choosing only methods that are certain to work, and rejecting all uncertainty. But, hopefully, you care more about your daughter's life than about "rationality".

Will pride in your own virtue as a rationalist save you? Not if you believe that it is virtuous to choose certainty. You will only be able to learn something about rationality if your daughter's life matters more to you than your pride as a rationalist.

You may even learn something about rationality from the experience, if you are already far enough grown in your Art to say, "I must have had the wrong conception of rationality," and not, "Look at how rationality gave me the wrong answer!"

(The essential difficulty in becoming a master rationalist is that you need quite a bit of rationality to bootstrap the learning process.)

Is your belief that you ought to be rational, more important than your life? Because, as I've previously observed, risking your life isn't comparatively all that scary. Being the lone voice of dissent in the crowd and having everyone look at you funny is much scarier than a mere threat to your life, according to the revealed preferences of teenagers who drink at parties and then drive home. It will take something terribly important to make you willing to leave the pack. A threat to your life won't be enough.

Is your will to rationality stronger than your pride? Can it be, if your will to rationality stems from your pride in your self-image as a rationalist? It's helpful - very helpful - to have a self-image which says that you are the sort of person who confronts harsh truth. It's helpful to have too much self-respect to knowingly lie to yourself or refuse to face evidence. But there may come a time when you have to admit that you've been doing rationality all wrong. Then your pride, your self-image as a rationalist, may make that too hard to face.

If you've prided yourself on believing what the Great Teacher says - even when it seems harsh, even when you'd rather not - that may make it all the more bitter a pill to swallow, to admit that the Great Teacher is a fraud, and all your noble self-sacrifice was for naught.

Where do you get the will to keep moving forward?

When I look back at my own personal journey toward rationality - not just humanity's historical journey - well, I grew up believing very strongly that I ought to be rational. This made me an above-average Traditional Rationalist a la Feynman and Heinlein, and nothing more. It did not drive me to go beyond the teachings I had received. I only began to grow further as a rationalist once I had something terribly important that I needed to do. Something more important than my pride as a rationalist, never mind my life.

Only when you become more wedded to success than to any of your beloved techniques of rationality, do you begin to appreciate these words of Miyamoto Musashi:

"You can win with a long weapon, and yet you can also win with a short weapon. In short, the Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size."
-- Miyamoto Musashi, The Book of Five Rings

Don't mistake this for a specific teaching of rationality. It describes how you learn the Way, beginning with a desperate need to succeed. No one masters the Way until more than their life is at stake. More than their comfort, more even than their pride.

You can't just pick out a Cause like that because you feel you need a hobby. Go looking for a "good cause", and your mind will just fill in a standard cliche. Learn how to multiply, and perhaps you will recognize a drastically important cause when you see one.

But if you have a cause like that, it is right and proper to wield your rationality in its service.

To strictly subordinate the aesthetics of rationality to a higher cause, is part of the aesthetic of rationality. You should pay attention to that aesthetic: You will never master rationality well enough to win with any weapon, if you do not appreciate the beauty for its own sake.

Sequence: Letting Go

The Meditation on Curiosity

No One Can Exempt You From Rationality's Laws

Sequence: Challenging the Difficult

The Quantum Physics Sequence

Einstein's Superpowers

Referenced by: Newcomb's Problem and Regret of Rationality • Probability is in the Mind • Reductive Reference • Zombies! Zombies? • Zombie Responses • No Safe Defense, Not Even Science • Einstein's Superpowers • A Premature Word on AI • The Moral Void • Where Recursive Justification Hits Bottom • Probability is Subjectively Objective • When (Not) To Use Probabilities • Changing Your Metaethics • Inseparably Right; or, Joy in the Merely Good • Beyond the Reach of God • Which Parts Are "Me"? • Higher Purpose • The Thing That I Protect • Rationality: Common Interest of Many Causes • Incremental Progress and the Valley • The End (of Sequences)

Original with comments: Something to Protect

Newcomb's Problem and Regret of Rationality

Eliezer Yudkowsky, 31 January 2008 07:36PM

Followup to: Something to Protect

The following may well be the most controversial dilemma in the history of decision theory:

A superintelligence from another galaxy, whom we shall call Omega, comes to Earth and sets about playing a strange little game. In this game, Omega selects a human being, sets down two boxes in front of them, and flies away.

Box A is transparent and contains a thousand dollars.
Box B is opaque, and contains either a million dollars, or nothing.

You can take both boxes, or take only box B.

And the twist is that Omega has put a million dollars in box B iff Omega has predicted that you will take only box B.

Omega has been correct on each of 100 observed occasions so far - everyone who took both boxes has found box B empty and received only a thousand dollars; everyone who took only box B has found B containing a million dollars. (We assume that box A vanishes in a puff of smoke if you take only box B; no one else can take box A afterward.)

Before you make your choice, Omega has flown off and moved on to its next game. Box B is already empty or already full.

Omega drops two boxes on the ground in front of you and flies off.

Do you take both boxes, or only box B?

And the standard philosophical conversation runs thusly:

One-boxer: "I take only box B, of course. I'd rather have a million than a thousand."

Two-boxer: "Omega has already left. Either box B is already full or already empty. If box B is already empty, then taking both boxes nets me $1000, taking only box B nets me $0. If box B is already full, then taking both boxes nets $1,001,000, taking only box B nets $1,000,000. In either case I do better by taking both boxes, and worse by leaving a thousand dollars on the table - so I will be rational, and take both boxes."

One-boxer: "If you're so rational, why ain'cha rich?"

Two-boxer: "It's not my fault Omega chooses to reward only people with irrational dispositions, but it's already too late for me to do anything about that."

There is a large literature on the topic of Newcomblike problems - especially if you consider the Prisoner's Dilemma as a special case, which it is generally held to be. "Paradoxes of Rationality and Cooperation" is an edited volume that includes Newcomb's original essay. For those who read only online material, this PhD thesis summarizes the major standard positions.

I'm not going to go into the whole literature, but the dominant consensus in modern decision theory is that one should two-box, and Omega is just rewarding agents with irrational dispositions. This dominant view goes by the name of "causal decision theory".

As you know, the primary reason I'm blogging is that I am an incredibly slow writer when I try to work in any other format. So I'm not going to try to present my own analysis here. Way too long a story, even by my standards.

But it is agreed even among causal decision theorists that if you have the power to precommit yourself to take one box, in Newcomb's Problem, then you should do so. If you can precommit yourself before Omega examines you; then you are directly causing box B to be filled.

Now in my field - which, in case you have forgotten, is self-modifying AI - this works out to saying that if you build an AI that two-boxes on Newcomb's Problem, it will self-modify to one-box on Newcomb's Problem, if the AI considers in advance that it might face such a situation. Agents with free access to their own source code have access to a cheap method of precommitment.

What if you expect that you might, in general, face a Newcomblike problem, without knowing the exact form of the problem? Then you would have to modify yourself into a sort of agent whose disposition was such that it would generally receive high rewards on Newcomblike problems.

But what does an agent with a disposition generally-well-suited to Newcomblike problems look like? Can this be formally specified?

Yes, but when I tried to write it up, I realized that I was starting to write a small book. And it wasn't the most important book I had to write, so I shelved it. My slow writing speed really is the bane of my existence. The theory I worked out seems, to me, to have many nice properties besides being well-suited to Newcomblike problems. It would make a nice PhD thesis, if I could get someone to accept it as my PhD thesis. But that's pretty much what it would take to make me unshelve the project. Otherwise I can't justify the time expenditure, not at the speed I currently write books.

I say all this, because there's a common attitude that "Verbal arguments for one-boxing are easy to come by, what's hard is developing a good decision theory that one-boxes" - coherent math which one-boxes on Newcomb's Problem without producing absurd results elsewhere. So I do understand that, and I did set out to develop such a theory, but my writing speed on big papers is so slow that I can't publish it. Believe it or not, it's true.

Nonetheless, I would like to present some of my motivations on Newcomb's Problem - the reasons I felt impelled to seek a new theory - because they illustrate my source-attitudes toward rationality. Even if I can't present the theory that these motivations motivate...

First, foremost, fundamentally, above all else:

Rational agents should WIN.

Don't mistake me, and think that I'm talking about the Hollywood Rationality stereotype that rationalists should be selfish or shortsighted. If your utility function has a term in it for others, then win their happiness. If your utility function has a term in it for a million years hence, then win the eon.

But at any rate, WIN. Don't lose reasonably, WIN.

Now there are defenders of causal decision theory who argue that the two-boxers are doing their best to win, and cannot help it if they have been cursed by a Predictor who favors irrationalists. I will talk about this defense in a moment. But first, I want to draw a distinction between causal decision theorists who believe that two-boxers are genuinely doing their best to win; versus someone who thinks that two-boxing is the reasonable or the rational thing to do, but that the reasonable move just happens to predictably lose, in this case. There are a lot of people out there who think that rationality predictably loses on various problems - that, too, is part of the Hollywood Rationality stereotype, that Kirk is predictably superior to Spock.

Next, let's turn to the charge that Omega favors irrationalists. I can conceive of a superbeing who rewards only people born with a particular gene, regardless of their choices. I can conceive of a superbeing who rewards people whose brains inscribe the particular algorithm of "Describe your options in English and choose the last option when ordered alphabetically," but who does not reward anyone who chooses the same option for a different reason. But Omega rewards people who choose to take only box B, regardless of which algorithm they use to arrive at this decision, and this is why I don't buy the charge that Omega is rewarding the irrational. Omega doesn't care whether or not you follow some particular ritual of cognition; Omega only cares about your predicted decision.

We can choose whatever reasoning algorithm we like, and will be rewarded or punished only according to that algorithm's choices, with no other dependency - Omega just cares where we go, not how we got there.

It is precisely the notion that Nature does not care about our algorithm, which frees us up to pursue the winning Way - without attachment to any particular ritual of cognition, apart from our belief that it wins. Every rule is up for grabs, except the rule of winning.

As Miyamoto Musashi said - it's really worth repeating:

"You can win with a long weapon, and yet you can also win with a short weapon. In short, the Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size."

(Another example: It was argued by McGee that we must adopt bounded utility functions or be subject to "Dutch books" over infinite times. But: The utility function is not up for grabs. I love life without limit or upper bound: There is no finite amount of life lived N where I would prefer a 80.0001% probability of living N years to an 0.0001% chance of living a googolplex years and an 80% chance of living forever. This is a sufficient condition to imply that my utility function is unbounded. So I just have to figure out how to optimize for that morality. You can't tell me, first, that above all I must conform to a particular ritual of cognition, and then that, if I conform to that ritual, I must change my morality to avoid being Dutch-booked. Toss out the losing ritual; don't change the definition of winning. That's like deciding to prefer $1000 to $1,000,000 so that Newcomb's Problem doesn't make your preferred ritual of cognition look bad.)

"But," says the causal decision theorist, "to take only one box, you must somehow believe that your choice can affect whether box B is empty or full - and that's unreasonable! Omega has already left! It's physically impossible!"

Unreasonable? I am a rationalist: what do I care about being unreasonable? I don't have to conform to a particular ritual of cognition. I don't have to take only box B because I believe my choice affects the box, even though Omega has already left. I can just... take only box B.

I do have a proposed alternative ritual of cognition which computes this decision, which this margin is too small to contain; but I shouldn't need to show this to you. The point is not to have an elegant theory of winning - the point is to win; elegance is a side effect.

Or to look at it another way: Rather than starting with a concept of what is the reasonable decision, and then asking whether "reasonable" agents leave with a lot of money, start by looking at the agents who leave with a lot of money, develop a theory of which agents tend to leave with the most money, and from this theory, try to figure out what is "reasonable". "Reasonable" may just refer to decisions in conformance with our current ritual of cognition - what else would determine whether something seems "reasonable" or not?

From James Joyce (no relation), Foundations of Causal Decision Theory:

Rachel has a perfectly good answer to the "Why ain't you rich?" question. "I am not rich," she will say, "because I am not the kind of person the psychologist thinks will refuse the money. I'm just not like you, Irene. Given that I know that I am the type who takes the money, and given that the psychologist knows that I am this type, it was reasonable of me to think that the $1,000,000 was not in my account. The $1,000 was the most I was going to get no matter what I did. So the only reasonable thing for me to do was to take it."

Irene may want to press the point here by asking, "But don't you wish you were like me, Rachel? Don't you wish that you were the refusing type?" There is a tendency to think that Rachel, a committed causal decision theorist, must answer this question in the negative, which seems obviously wrong (given that being like Irene would have made her rich). This is not the case. Rachel can and should admit that she does wish she were more like Irene. "It would have been better for me," she might concede, "had I been the refusing type." At this point Irene will exclaim, "You've admitted it! It wasn't so smart to take the money after all." Unfortunately for Irene, her conclusion does not follow from Rachel's premise. Rachel will patiently explain that wishing to be a refuser in a Newcomb problem is not inconsistent with thinking that one should take the $1,000 whatever type one is. When Rachel wishes she was Irene's type she is wishing for Irene's options, not sanctioning her choice.

It is, I would say, a general principle of rationality - indeed, part of how I define rationality - that you never end up envying someone else's mere choices. You might envy someone their genes, if Omega rewards genes, or if the genes give you a generally happier disposition. But Rachel, above, envies Irene her choice, and only her choice, irrespective of what algorithm Irene used to make it. Rachel wishes just that she had a disposition to choose differently.

You shouldn't claim to be more rational than someone and simultaneously envy them their choice - only their choice. Just do the act you envy.

I keep trying to say that rationality is the winning-Way, but causal decision theorists insist that taking both boxes is what really wins, because you can't possibly do better by leaving $1000 on the table... even though the single-boxers leave the experiment with more money. Be careful of this sort of argument, any time you find yourself defining the "winner" as someone other than the agent who is currently smiling from on top of a giant heap of utility.

Yes, there are various thought experiments in which some agents start out with an advantage - but if the task is to, say, decide whether to jump off a cliff, you want to be careful not to define cliff-refraining agents as having an unfair prior advantage over cliff-jumping agents, by virtue of their unfair refusal to jump off cliffs. At this point you have covertly redefined "winning" as conformance to a particular ritual of cognition. Pay attention to the money!

Or here's another way of looking at it: Faced with Newcomb's Problem, would you want to look really hard for a reason to believe that it was perfectly reasonable and rational to take only box B; because, if such a line of argument existed, you would take only box B and find it full of money? Would you spend an extra hour thinking it through, if you were confident that, at the end of the hour, you would be able to convince yourself that box B was the rational choice? This too is a rather odd position to be in. Ordinarily, the work of rationality goes into figuring out which choice is the best - not finding a reason to believe that a particular choice is the best.

Maybe it's too easy to say that you "ought to" two-box on Newcomb's Problem, that this is the "reasonable" thing to do, so long as the money isn't actually in front of you. Maybe you're just numb to philosophical dilemmas, at this point. What if your daughter had a 90% fatal disease, and box A contained a serum with a 20% chance of curing her, and box B might contain a serum with a 95% chance of curing her? What if there was an asteroid rushing toward Earth, and box A contained an asteroid deflector that worked 10% of the time, and box B might contain an asteroid deflector that worked 100% of the time?

Would you, at that point, find yourself tempted to make an unreasonable choice?

If the stake in box B was something you could not leave behind? Something overwhelmingly more important to you than being reasonable? If you absolutely had to win - really win, not just be defined as winning?

Would you wish with all your power that the "reasonable" decision was to take only box B?

Then maybe it's time to update your definition of reasonableness.

Alleged rationalists should not find themselves envying the mere decisions of alleged nonrationalists, because your decision can be whatever you like. When you find yourself in a position like this, you shouldn't chide the other person for failing to conform to your concepts of reasonableness. You should realize you got the Way wrong.

So, too, if you ever find yourself keeping separate track of the "reasonable" belief, versus the belief that seems likely to be actually true. Either you have misunderstood reasonableness, or your second intuition is just wrong.

Now one can't simultaneously define "rationality" as the winning Way, and define "rationality" as Bayesian probability theory and decision theory. But it is the argument that I am putting forth, and the moral of my advice to Trust In Bayes, that the laws governing winning have indeed proven to be math. If it ever turns out that Bayes fails - receives systematically lower rewards on some problem, relative to a superior alternative, in virtue of its mere decisions - then Bayes has to go out the window. "Rationality" is just the label I use for my beliefs about the winning Way - the Way of the agent smiling from on top of the giant heap of utility. Currently, that label refers to Bayescraft.

I realize that this is not a knockdown criticism of causal decision theory - that would take the actual book and/or PhD thesis - but I hope it illustrates some of my underlying attitude toward this notion of "rationality".

You shouldn't find yourself distinguishing the winning choice from the reasonable choice. Nor should you find yourself distinguishing the reasonable belief from the belief that is most likely to be true.

That is why I use the word "rational" to denote my beliefs about accuracy and winning - not to denote verbal reasoning, or strategies which yield certain success, or that which is logically provable, or that which is publicly demonstrable, or that which is reasonable.

As Miyamoto Musashi said:

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him."

Referenced by: Replace the Symbol with the Substance • Zombies! Zombies? • Decoherent Essences • Many Worlds, One Best Guess • Science Isn't Strict Enough • Einstein's Speed • Timeless Physics • Why Quantum? • The Moral Void • My Kind of Reflection • Can Counterfactuals Be True? • The True Prisoner's Dilemma • The Truly Iterated Prisoner's Dilemma • Prices or Bindings? • Ethics Notes • Today's Inspirational Tale • Is That Your True Rejection? • High Challenge • Harmful Options • Free to Optimize • What Do We Mean By "Rationality"? • Why Our Kind Can't Cooperate • Your Price for Joining • Rationality is Systematized Winning • Newcomb's Problem standard positions • Bayesians vs. Barbarians • The Sin of Underconfidence • Timeless Decision Theory: Problems I Can't Solve • Ingredients of Timeless Decision Theory • The Lifespan Dilemma

Original with comments: Newcomb's Problem and Regret of Rationality

OB Meetup: Millbrae, Thu 21 Feb, 7pm

Eliezer Yudkowsky, 31 January 2008 11:18PM

The Overcoming Bias meetup has been scheduled for Thursday, February 21st, at 7pm. We're going to look at locating this in Millbrae within walking distance of the BART / Caltrain station. The particular restaurant I had in mind turns out to be booked for Thursdays, so if you know a good Millbrae restaurant (with a private room?) in walking distance of the train station, please post in the comments. I'll be looking at restaurants shortly.

Why not schedule to a day other than Thursday, you ask?

Because:

Robin Hanson will be in the Bay Area and attending! Woohoo!

If you would be able to make Thursday the 21st, 7pm, in Millbrae, somewhere near the BART/Caltrain, please vote below. No, seriously, please vote, now - the kind of restaurant I have to find depends on how many people will be attending.

Opinion Polls & Market Research

Referenced by: Classic Sichuan in Millbrae, Thu Feb 21, 7pm

Original with comments: OB Meetup: Millbrae, Thu 21 Feb, 7pm

The Parable of the Dagger

Eliezer Yudkowsky, 01 February 2008 08:53PM

Once upon a time, there was a court jester who dabbled in logic.

The jester presented the king with two boxes. Upon the first box was inscribed:

"Either this box contains an angry frog, or the box with a false inscription contains an angry frog, but not both."

On the second box was inscribed:

"Either this box contains gold and the box with a false inscription contains an angry frog, or this box contains an angry frog and the box with a true inscription contains gold."

And the jester said to the king: "One box contains an angry frog, the other box gold; and one, and only one, of the inscriptions is true."

The king opened the wrong box, and was savaged by an angry frog.

"You see," the jester said, "let us hypothesize that the first inscription is the true one. Then suppose the first box contains gold. Then the other box would have an angry frog, while the box with a true inscription would contain gold, which would make the second statement true as well. Now hypothesize that the first inscription is false, and that the first box contains gold. Then the second inscription would be -"

The king ordered the jester thrown in the dungeons.

A day later, the jester was brought before the king in chains, and shown two boxes.

"One box contains a key," said the king, "to unlock your chains; and if you find the key you are free. But the other box contains a dagger for your heart, if you fail."

And the first box was inscribed:

"Either both inscriptions are true, or both inscriptions are false."

And the second box was inscribed:

"This box contains the key."

The jester reasoned thusly: "Suppose the first inscription is true. Then the second inscription must also be true. Now suppose the first inscription is false. Then again the second inscription must be true. So the second box must contain the key, if the first inscription is true, and also if the first inscription is false. Therefore, the second box must logically contain the key."

The jester opened the second box, and found a dagger.

"How?!" cried the jester in horror, as he was dragged away. "It's logically impossible!"

"It is entirely possible," replied the king. "I merely wrote those inscriptions on two boxes, and then I put the dagger in the second one."

(Adapted from Raymond Smullyan.)

Sequence: A Human's Guide to Words

The Parable of Hemlock

Referenced by: The Parable of Hemlock • 37 Ways That Words Can Be Wrong • The Generalized Anti-Zombie Principle • Decoherent Essences

Original with comments: The Parable of the Dagger

The Parable of Hemlock

Eliezer Yudkowsky, 03 February 2008 02:01AM

Followup to: The Parable of the Dagger

"All men are mortal. Socrates is a man. Therefore Socrates is mortal."
-- Aristotle(?)

Socrates raised the glass of hemlock to his lips...
"Do you suppose," asked one of the onlookers, "that even hemlock will not be enough to kill so wise and good a man?"
"No," replied another bystander, a student of philosophy; "all men are mortal, and Socrates is a man; and if a mortal drink hemlock, surely he dies."
"Well," said the onlooker, "what if it happens that Socrates isn't mortal?"
"Nonsense," replied the student, a little sharply; "all men are mortal by definition; it is part of what we mean by the word 'man'. All men are mortal, Socrates is a man, therefore Socrates is mortal. It is not merely a guess, but a logical certainty."
"I suppose that's right..." said the onlooker. "Oh, look, Socrates already drank the hemlock while we were talking."
"Yes, he should be keeling over any minute now," said the student.
And they waited, and they waited, and they waited...
"Socrates appears not to be mortal," said the onlooker.
"Then Socrates must not be a man," replied the student. "All men are mortal, Socrates is not mortal, therefore Socrates is not a man. And that is not merely a guess, but a logical certainty."

The fundamental problem with arguing that things are true "by definition" is that you can't make reality go a different way by choosing a different definition.

You could reason, perhaps, as follows: "All things I have observed which wear clothing, speak language, and use tools, have also shared certain other properties as well, such as breathing air and pumping red blood. The last thirty 'humans' belonging to this cluster, whom I observed to drink hemlock, soon fell over and stopped moving. Socrates wears a toga, speaks fluent ancient Greek, and drank hemlock from a cup. So I predict that Socrates will keel over in the next five minutes."

But that would be mere guessing. It wouldn't be, y'know, absolutely and eternally certain. The Greek philosophers - like most prescientific philosophers - were rather fond of certainty.

Luckily the Greek philosophers have a crushing rejoinder to your questioning. You have misunderstood the meaning of "All humans are mortal," they say. It is not a mere observation. It is part of the definition of the word "human". Mortality is one of several properties that are individually necessary, and together sufficient, to determine membership in the class "human". The statement "All humans are mortal" is a logically valid truth, absolutely unquestionable. And if Socrates is human, he must be mortal: it is a logical deduction, as certain as certain can be.

But then we can never know for certain that Socrates is a "human" until after Socrates has been observed to be mortal. It does no good to observe that Socrates speaks fluent Greek, or that Socrates has red blood, or even that Socrates has human DNA. None of these characteristics are logically equivalent to mortality. You have to see him die before you can conclude that he was human.

(And even then it's not infinitely certain. What if Socrates rises from the grave a night after you see him die? Or more realistically, what if Socrates is signed up for cryonics? If mortality is defined to mean finite lifespan, then you can never really know if someone was human, until you've observed to the end of eternity - just to make sure they don't come back. Or you could think you saw Socrates keel over, but it could be an illusion projected onto your eyes with a retinal scanner. Or maybe you just hallucinated the whole thing...)

The problem with syllogisms is that they're always valid. "All humans are mortal; Socrates is human; therefore Socrates is mortal" is - if you treat it as a logical syllogism - logically valid within our own universe. It's also logically valid within neighboring Everett branches in which, due to a slightly different evolved biochemistry, hemlock is a delicious treat rather than a poison. And it's logically valid even in universes where Socrates never existed, or for that matter, where humans never existed.

The Bayesian definition of evidence favoring a hypothesis is evidence which we are more likely to see if the hypothesis is true than if it is false. Observing that a syllogism is logically valid can never be evidence favoring any empirical proposition, because the syllogism will be logically valid whether that proposition is true or false.

Syllogisms are valid in all possible worlds, and therefore, observing their validity never tells us anything about which possible world we actually live in.

This doesn't mean that logic is useless - just that logic can only tell us that which, in some sense, we already know. But we do not always believe what we know. Is the number 29384209 prime? By virtue of how I define my decimal system and my axioms of arithmetic, I have already determined my answer to this question - but I do not know what my answer is yet, and I must do some logic to find out.

Similarly, if I form the uncertain empirical generalization "Humans are vulnerable to hemlock", and the uncertain empirical guess "Socrates is human", logic can tell me that my previous guesses are predicting that Socrates will be vulnerable to hemlock.

It's been suggested that we can view logical reasoning as resolving our uncertainty about impossible possible worlds - eliminating probability mass in logically impossible worlds which we did not know to be logically impossible. In this sense, logical argument can be treated as observation.

But when you talk about an empirical prediction like "Socrates is going to keel over and stop breathing" or "Socrates is going to do fifty jumping jacks and then compete in the Olympics next year", that is a matter of possible worlds, not impossible possible worlds.

Logic can tell us which hypotheses match up to which observations, and it can tell us what these hypotheses predict for the future - it can bring old observations and previous guesses to bear on a new problem. But logic never flatly says, "Socrates will stop breathing now." Logic never dictates any empirical question; it never settles any real-world query which could, by any stretch of the imagination, go either way.

Just remember the Litany Against Logic:

Logic stays true, wherever you may go,
So logic never tells you where you live.

Sequence: A Human's Guide to Words

The Parable of the Dagger

Words as Hidden Inferences

Referenced by: Words as Hidden Inferences • Disputing Definitions • Empty Labels • Taboo Your Words • Sneaking in Connotations • 37 Ways That Words Can Be Wrong • The Generalized Anti-Zombie Principle • Could Anything Be Right? • The Nature of Logic • What Do We Mean By "Rationality"? • Conversation Halters

Original with comments: The Parable of Hemlock

Words as Hidden Inferences

Eliezer Yudkowsky, 03 February 2008 11:36PM

Followup to: The Parable of Hemlock

Suppose I find a barrel, sealed at the top, but with a hole large enough for a hand. I reach in, and feel a small, curved object. I pull the object out, and it's blue - a bluish egg. Next I reach in and feel something hard and flat, with edges - which, when I extract it, proves to be a red cube. I pull out 11 eggs and 8 cubes, and every egg is blue, and every cube is red.

Now I reach in and I feel another egg-shaped object. Before I pull it out and look, I have to guess: What will it look like?

The evidence doesn't prove that every egg in the barrel is blue, and every cube is red. The evidence doesn't even argue this all that strongly: 19 is not a large sample size. Nonetheless, I'll guess that this egg-shaped object is blue - or as a runner-up guess, red. If I guess anything else, there's as many possibilities as distinguishable colors - and for that matter, who says the egg has to be a single shade? Maybe it has a picture of a horse painted on.

So I say "blue", with a dutiful patina of humility. For I am a sophisticated rationalist-type person, and I keep track of my assumptions and dependencies - I guess, but I'm aware that I'm guessing... right?

But when a large yellow striped feline-shaped object leaps out at me from the shadows, I think, "Yikes! A tiger!" Not, "Hm... objects with the properties of largeness, yellowness, stripedness, and feline shape, have previously often possessed the properties 'hungry' and 'dangerous', and thus, although it is not logically necessary, it may be an empirically good guess that aaauuughhhh CRUNCH CRUNCH GULP."

The human brain, for some odd reason, seems to have been adapted to make this inference quickly, automatically, and without keeping explicit track of its assumptions.

And if I name the egg-shaped objects "bleggs" (for blue eggs) and the red cubes "rubes", then, when I reach in and feel another egg-shaped object, I may think: Oh, it's a blegg, rather than considering all that problem-of-induction stuff.

It is a common misconception that you can define a word any way you like.

This would be true if the brain treated words as purely logical constructs, Aristotelian classes, and you never took out any more information than you put in.

Yet the brain goes on about its work of categorization, whether or not we consciously approve. "All humans are mortal, Socrates is a human, therefore Socrates is mortal" - thus spake the ancient Greek philosophers. Well, if mortality is part of your logical definition of "human", you can't logically classify Socrates as human until you observe him to be mortal. But - this is the problem - Aristotle knew perfectly well that Socrates was a human. Aristotle's brain placed Socrates in the "human" category as efficiently as your own brain categorizes tigers, apples, and everything else in its environment: Swiftly, silently, and without conscious approval.

Aristotle laid down rules under which no one could conclude Socrates was "human" until after he died. Nonetheless, Aristotle and his students went on concluding that living people were humans and therefore mortal; they saw distinguishing properties such as human faces and human bodies, and their brains made the leap to inferred properties such as mortality.

Misunderstanding the working of your own mind does not, thankfully, prevent the mind from doing its work. Otherwise Aristotelians would have starved, unable to conclude that an object was edible merely because it looked and felt like a banana.

So the Aristotelians went on classifying environmental objects on the basis of partial information, the way people had always done. Students of Aristotelian logic went on thinking exactly the same way, but they had acquired an erroneous picture of what they were doing.

If you asked an Aristotelian philosopher whether Carol the grocer was mortal, they would say "Yes." If you asked them how they knew, they would say "All humans are mortal, Carol is human, therefore Carol is mortal." Ask them whether it was a guess or a certainty, and they would say it was a certainty (if you asked before the sixteenth century, at least). Ask them how they knew that humans were mortal, and they would say it was established by definition.

The Aristotelians were still the same people, they retained their original natures, but they had acquired incorrect beliefs about their own functioning. They looked into the mirror of self-awareness, and saw something unlike their true selves: they reflected incorrectly.

Your brain doesn't treat words as logical definitions with no empirical consequences, and so neither should you. The mere act of creating a word can cause your mind to allocate a category, and thereby trigger unconscious inferences of similarity. Or block inferences of similarity; if I create two labels I can get your mind to allocate two categories. Notice how I said "you" and "your brain" as if they were different things?

Making errors about the inside of your head doesn't change what's there; otherwise Aristotle would have died when he concluded that the brain was an organ for cooling the blood. Philosophical mistakes usually don't interfere with blink-of-an-eye perceptual inferences.

But philosophical mistakes can severely mess up the deliberate thinking processes that we use to try to correct our first impressions. If you believe that you can "define a word any way you like", without realizing that your brain goes on categorizing without your conscious oversight, then you won't take the effort to choose your definitions wisely.

Sequence: A Human's Guide to Words

The Parable of Hemlock

Extensions and Intensions

Referenced by: Extensions and Intensions • Typicality and Asymmetrical Similarity • Feel the Meaning • The Argument from Common Usage • Empty Labels • Sneaking in Connotations • Searching for Bayes-Structure • 37 Ways That Words Can Be Wrong

Original with comments: Words as Hidden Inferences

Extensions and Intensions

Eliezer Yudkowsky, 04 February 2008 09:34PM

Followup to: Words as Hidden Inferences

"What is red?"
"Red is a color."
"What's a color?"
"A color is a property of a thing."

But what is a thing? And what's a property? Soon the two are lost in a maze of words defined in other words, the problem that Steven Harnad once described as trying to learn Chinese from a Chinese/Chinese dictionary.

Alternatively, if you asked me "What is red?" I could point to a stop sign, then to someone wearing a red shirt, and a traffic light that happens to be red, and blood from where I accidentally cut myself, and a red business card, and then I could call up a color wheel on my computer and move the cursor to the red area. This would probably be sufficient, though if you know what the word "No" means, the truly strict would insist that I point to the sky and say "No."

I think I stole this example from S. I. Hayakawa - though I'm really not sure, because I heard this way back in the indistinct blur of my childhood. (When I was 12, my father accidentally deleted all my computer files. I have no memory of anything before that.)

But that's how I remember first learning about the difference between intensional and extensional definition. To give an "intensional definition" is to define a word or phrase in terms of other words, as a dictionary does. To give an "extensional definition" is to point to examples, as adults do when teaching children. The preceding sentence gives an intensional definition of "extensional definition", which makes it an extensional example of "intensional definition".

In Hollywood Rationality and popular culture generally, "rationalists" are depicted as word-obsessed, floating in endless verbal space disconnected from reality.

But the actual Traditional Rationalists have long insisted on maintaining a tight connection to experience:

"If you look into a textbook of chemistry for a definition of lithium, you may be told that it is that element whose atomic weight is 7 very nearly. But if the author has a more logical mind he will tell you that if you search among minerals that are vitreous, translucent, grey or white, very hard, brittle, and insoluble, for one which imparts a crimson tinge to an unluminous flame, this mineral being triturated with lime or witherite rats-bane, and then fused, can be partly dissolved in muriatic acid; and if this solution be evaporated, and the residue be extracted with sulphuric acid, and duly purified, it can be converted by ordinary methods into a chloride, which being obtained in the solid state, fused, and electrolyzed with half a dozen powerful cells, will yield a globule of a pinkish silvery metal that will float on gasolene; and the material of that is a specimen of lithium."
-- Charles Sanders Peirce

That's an example of "logical mind" as described by a genuine Traditional Rationalist, rather than a Hollywood scriptwriter.

But note: Peirce isn't actually showing you a piece of lithium. He didn't have pieces of lithium stapled to his book. Rather he's giving you a treasure map - an intensionally defined procedure which, when executed, will lead you to an extensional example of lithium. This is not the same as just tossing you a hunk of lithium, but it's not the same as saying "atomic weight 7" either. (Though if you had sufficiently sharp eyes, saying "3 protons" might let you pick out lithium at a glance...)

So that is intensional and extensional definition., which is a way of telling someone else what you mean by a concept. When I talked about "definitions" above, I talked about a way of communicating concepts - telling someone else what you mean by "red", "tiger", "human", or "lithium". Now let's talk about the actual concepts themselves.

The actual intension of my "tiger" concept would be the neural pattern (in my temporal cortex) that inspects an incoming signal from the visual cortex to determine whether or not it is a tiger.

The actual extension of my "tiger" concept is everything I call a tiger.

Intensional definitions don't capture entire intensions; extensional definitions don't capture entire extensions. If I point to just one tiger and say the word "tiger", the communication may fail if they think I mean "dangerous animal" or "male tiger" or "yellow thing". Similarly, if I say "dangerous yellow-black striped animal", without pointing to anything, the listener may visualize giant hornets.

You can't capture in words all the details of the cognitive concept - as it exists in your mind - that lets you recognize things as tigers or nontigers. It's too large. And you can't point to all the tigers you've ever seen, let alone everything you would call a tiger.

The strongest definitions use a crossfire of intensional and extensional communication to nail down a concept. Even so, you only communicate maps to concepts, or instructions for building concepts - you don't communicate the actual categories as they exist in your mind or in the world.

(Yes, with enough creativity you can construct exceptions to this rule, like "Sentences Eliezer Yudkowsky has published containing the term 'huragaloni' as of Feb 4, 2008". I've just shown you this concept's entire extension. But except in mathematics, definitions are usually treasure maps, not treasure.)

So that's another reason you can't "define a word any way you like": You can't directly program concepts into someone else's brain.

Even within the Aristotelian paradigm, where we pretend that the definitions are the actual concepts, you don't have simultaneous freedom of intension and extension. Suppose I define Mars as "A huge red rocky sphere, around a tenth of Earth's mass and 50% further away from the Sun". It's then a separate matter to show that this intensional definition matches some particular extensional thing in my experience, or indeed, that it matches any real thing whatsoever. If instead I say "That's Mars" and point to a red light in the night sky, it becomes a separate matter to show that this extensional light matches any particular intensional definition I may propose - or any intensional beliefs I may have - such as "Mars is the God of War".

But most of the brain's work of applying intensions happens sub-deliberately. We aren't consciously aware that our identification of a red light as "Mars" is a separate matter from our verbal definition "Mars is the God of War". No matter what kind of intensional definition I make up to describe Mars, my mind believes that "Mars" refers to this thingy, and that it is the fourth planet in the Solar System.

When you take into account the way the human mind actually, pragmatically works, the notion "I can define a word any way I like" soon becomes "I can believe anything I want about a fixed set of objects" or "I can move any object I want in or out of a fixed membership test". Just as you can't usually convey a concept's whole intension in words because it's a big complicated neural membership test, you can't control the concept's entire intension because it's applied sub-deliberately. This is why arguing that XYZ is true "by definition" is so popular. If definition changes behaved like the empirical nullops they're supposed to be, no one would bother arguing them. But abuse definitions just a little, and they turn into magic wands - in arguments, of course; not in reality.

Sequence: A Human's Guide to Words

Words as Hidden Inferences

Similarity Clusters

Referenced by: Similarity Clusters • Disputing Definitions • The Argument from Common Usage • Where to Draw the Boundary? • Superexponential Conceptspace, and Simple Words • 37 Ways That Words Can Be Wrong • Reductive Reference • The Generalized Anti-Zombie Principle

Original with comments: Extensions and Intensions

Buy Now Or Forever Hold Your Peace

Eliezer Yudkowsky, 04 February 2008 09:42PM

The Intrade prediction market is giving Hillary a 53% chance and Obama a 47% chance of winning the Democratic presidential nomination. Hillary is down 7.5 percentage points in just the last day. (Note: Between when I wrote the above, and when I posted this, Hillary went up to 54.)

From what I've read on Intrade, you can fund your account with up to $250 using a credit card, and it should land in your account immediately. (More than this takes time.) Also, remember that you can sell contracts at any time afterward - you don't have to wait months to collect your payout.

If you think that Hillary is going to do better than the polls on Super Tuesday, and you're going to sneer afterward and say that Intrade was "just tracking the polls", buy Hillary now.

If you think that Obama is going to do better than the polls on Super Tuesday, and you're going to gloat about how prediction markets didn't call this surprise in advance, buy Obama now.

If you don't do either, then clearly you do not really believe that you know anything the prediction markets don't. (Or you don't understand expected utility, or your utilities over final outcomes drop off improbably fast in the vicinity of your current wealth minus fifty bucks - you don't have to bet the full $250.) It is free money, going now for anyone who genuinely thinks they know better than the prediction markets what will happen next.

Prediction markets do not have supernatural insight. If they give the candidates fifty-fifty odds, it means that the market collectively doesn't know what will happen next. Even if you're well-calibrated, you get surprised on 90% probabilities one time out of ten.

The point is not that prediction markets are a good predictor but that they are the best predictor. If you think you can do better, why ain'cha rich? Any person, group, or method that does better can pump money out of the prediction markets.

If prediction markets react to polls, they're getting new information, that they didn't predict in advance, which happens. Being the best predictor doesn't make you omniscient.

Everyone's going to find it real easy to make a better prediction afterward, but if you think you can call it in advance, there's FREE MONEY GOING NOW.

Buy now, or forever hold your peace.

Referenced by: No Safe Defense, Not Even Science

Original with comments: Buy Now Or Forever Hold Your Peace

Similarity Clusters

Eliezer Yudkowsky, 06 February 2008 03:34AM

Followup to: Extensions and Intensions

Once upon a time, the philosophers of Plato's Academy claimed that the best definition of human was a "featherless biped". Diogenes of Sinope, also called Diogenes the Cynic, is said to have promptly exhibited a plucked chicken and declared "Here is Plato's man." The Platonists promptly changed their definition to "a featherless biped with broad nails".

No dictionary, no encyclopedia, has ever listed all the things that humans have in common. We have red blood, five fingers on each of two hands, bony skulls, 23 pairs of chromosomes - but the same might be said of other animal species. We make complex tools to make complex tools, we use syntactical combinatorial language, we harness critical fission reactions as a source of energy: these things may serve out to single out only humans, but not all humans - many of us have never built a fission reactor. With the right set of necessary-and-sufficient gene sequences you could single out all humans, and only humans - at least for now - but it would still be far from all that humans have in common.

But so long as you don't happen to be near a plucked chicken, saying "Look for featherless bipeds" may serve to pick out a few dozen of the particular things that are humans, as opposed to houses, vases, sandwiches, cats, colors, or mathematical theorems.

Once the definition "featherless biped" has been bound to some particular featherless bipeds, you can look over the group, and begin harvesting some of the other characteristics - beyond mere featherfree twolegginess - that the "featherless bipeds" seem to share in common. The particular featherless bipeds that you see seem to also use language, build complex tools, speak combinatorial language with syntax, bleed red blood if poked, die when they drink hemlock.

Thus the category "human" grows richer, and adds more and more characteristics; and when Diogenes finally presents his plucked chicken, we are not fooled: This plucked chicken is obviously not similar to the other "featherless bipeds".

(If Aristotelian logic were a good model of human psychology, the Platonists would have looked at the plucked chicken and said, "Yes, that's a human; what's your point?")

If the first featherless biped you see is a plucked chicken, then you may end up thinking that the verbal label "human" denotes a plucked chicken; so I can modify my treasure map to point to "featherless bipeds with broad nails", and if I am wise, go on to say, "See Diogenes over there? That's a human, and I'm a human, and you're a human; and that chimpanzee is not a human, though fairly close."

The initial clue only has to lead the user to the similarity cluster - the group of things that have many characteristics in common. After that, the initial clue has served its purpose, and I can go on to convey the new information "humans are currently mortal", or whatever else I want to say about us featherless bipeds.

A dictionary is best thought of, not as a book of Aristotelian class definitions, but a book of hints for matching verbal labels to similarity clusters, or matching labels to properties that are useful in distinguishing similarity clusters.

Sequence: A Human's Guide to Words

Extensions and Intensions

Typicality and Asymmetrical Similarity

Referenced by: Typicality and Asymmetrical Similarity • The Argument from Common Usage • Categorizing Has Consequences • Arguing "By Definition" • 37 Ways That Words Can Be Wrong • The Design Space of Minds-In-General • The Nature of Logic

Original with comments: Similarity Clusters

Typicality and Asymmetrical Similarity

Eliezer Yudkowsky, 06 February 2008 09:20PM

Followup to: Similarity Clusters

Birds fly. Well, except ostriches don't. But which is a more typical bird - a robin, or an ostrich?
Which is a more typical chair: A desk chair, a rocking chair, or a beanbag chair?

Most people would say that a robin is a more typical bird, and a desk chair is a more typical chair. The cognitive psychologists who study this sort of thing experimentally, do so under the heading of "typicality effects" or "prototype effects" (Rosch and Lloyd 1978). For example, if you ask subjects to press a button to indicate "true" or "false" in response to statements like "A robin is a bird" or "A penguin is a bird", reaction times are faster for more central examples. (I'm still unpacking my books, but I'm reasonably sure my source on this is Lakoff 1986.) Typicality measures correlate well using different investigative methods - reaction times are one example; you can also ask people to directly rate, on a scale of 1 to 10, how well an example (like a specific robin) fits a category (like "bird").

So we have a mental measure of typicality - which might, perhaps, function as a heuristic - but is there a corresponding bias we can use to pin it down?

Well, which of these statements strikes you as more natural: "98 is approximately 100", or "100 is approximately 98"? If you're like most people, the first statement seems to make more sense. (Sadock 1977.) For similar reasons, people asked to rate how similar Mexico is to the United States, gave consistently higher ratings than people asked to rate how similar the United States is to Mexico. (Tversky and Gati 1978.)

And if that still seems harmless, a study by Rips (1975) showed that people were more likely to expect a disease would spread from robins to ducks on an island, than from ducks to robins. Now this is not a logical impossibility, but in a pragmatic sense, whatever difference separates a duck from a robin and would make a disease less likely to spread from a duck to a robin, must also be a difference between a robin and a duck, and would make a disease less likely to spread from a robin to a duck.

Yes, you can come up with rationalizations, like "Well, there could be more neighboring species of the robins, which would make the disease more likely to spread initially, etc.," but be careful not to try too hard to rationalize the probability ratings of subjects who didn't even realize there was a comparison going on. And don't forget that Mexico is more similar to the United States than the United States is to Mexico, and that 98 is closer to 100 than 100 is to 98. A simpler interpretation is that people are using the (demonstrated) similarity heuristic as a proxy for the probability that a disease spreads, and this heuristic is (demonstrably) asymmetrical.

Kansas is unusually close to the center of the United States, and Alaska is unusually far from the center of the United States; so Kansas is probably closer to most places in the US and Alaska is probably farther. It does not follow, however, that Kansas is closer to Alaska than is Alaska to Kansas. But people seem to reason (metaphorically speaking) as if closeness is an inherent property of Kansas and distance is an inherent property of Alaska; so that Kansas is still close, even to Alaska; and Alaska is still distant, even from Kansas.

So once again we see that Aristotle's notion of categories - logical classes with membership determined by a collection of properties that are individually strictly necessary, and together strictly sufficient - is not a good model of human cognitive psychology. (Science's view has changed somewhat over the last 2350 years? Who would've thought?) We don't even reason as if set membership is a true-or-false property: Statements of set membership can be more or less true. (Note: This is not the same thing as being more or less probable.)

One more reason not to pretend that you, or anyone else, is really going to treat words as Aristotelian logical classes.

Lakoff, George. (1986). Women, Fire and Dangerous Things: What Categories Tell Us About the Nature of Thought. University of Chicago Press, Chicago.

Rips, Lance J. (1975). "Inductive judgments about natural categories." Journal of Verbal Learning and Verbal Behavior. 14:665-81.

Rosch, Eleanor and B. B. Lloyd, eds. (1978). Cognition and Categorization. Hillsdale, N.J.: Lawrence Erlbaum Associates.

Sadock, Jerrold. (1977). "Truth and Approximations." In Papers from the Third Annual Meeting of the Berkeley Linguistics Society, pp. 430-39. Berkeley: Berkeley Linguistics Society.

Tversky, Amos and Itamar Gati. (1978). "Studies of Similarity". In Rosch and Lloyd (1978).

Sequence: A Human's Guide to Words

Similarity Clusters

The Cluster Structure of Thingspace

Referenced by: The Cluster Structure of Thingspace • 37 Ways That Words Can Be Wrong

Original with comments: Typicality and Asymmetrical Similarity

The Cluster Structure of Thingspace

Eliezer Yudkowsky, 08 February 2008 12:07AM

Followup to: Typicality and Asymmetrical Similarity

The notion of a "configuration space" is a way of translating object descriptions into object positions. It may seem like blue is "closer" to blue-green than to red, but how much closer? It's hard to answer that question by just staring at the colors. But it helps to know that the (proportional) color coordinates in RGB are 0:0:5, 0:3:2 and 5:0:0. It would be even clearer if plotted on a 3D graph.

In the same way, you can see a robin as a robin - brown tail, red breast, standard robin shape, maximum flying speed when unladen, its species-typical DNA and individual alleles. Or you could see a robin as a single point in a configuration space whose dimensions described everything we knew, or could know, about the robin.

A robin is bigger than a virus, and smaller than an aircraft carrier - that might be the "volume" dimension. Likewise a robin weighs more than a hydrogen atom, and less than a galaxy; that might be the "mass" dimension. Different robins will have strong correlations between "volume" and "mass", so the robin-points will be lined up in a fairly linear string, in those two dimensions - but the correlation won't be exact, so we do need two separate dimensions.

This is the benefit of viewing robins as points in space: You couldn't see the linear lineup as easily if you were just imagining the robins as cute little wing-flapping creatures.

A robin's DNA is a highly multidimensional variable, but you can still think of it as part of a robin's location in thingspace - millions of quaternary coordinates, one coordinate for each DNA base - or maybe a more sophisticated view that . The shape of the robin, and its color (surface reflectance), you can likewise think of as part of the robin's position in thingspace, even though they aren't single dimensions.

Just like the coordinate point 0:0:5 contains the same information as the actual HTML color blue, we shouldn't actually lose information when we see robins as points in space. We believe the same statement about the robin's mass whether we visualize a robin balancing the scales opposite a 0.07-kilogram weight, or a robin-point with a mass-coordinate of +70.

We can even imagine a configuration space with one or more dimensions for every distinct characteristic of an object, so that the position of an object's point in this space corresponds to all the information in the real object itself. Rather redundantly represented, too - dimensions would include the mass, the volume, and the density.

If you think that's extravagant, quantum physicists use an infinite-dimensional configuration space, and a single point in that space describes the location of every particle in the universe. So we're actually being comparatively conservative in our visualization of thingspace - a point in thingspace describes just one object, not the entire universe.

If we're not sure of the robin's exact mass and volume, then we can think of a little cloud in thingspace, a volume of uncertainty, within which the robin might be. The density of the cloud is the density of our belief that the robin has that particular mass and volume. If you're more sure of the robin's density than of its mass and volume, your probability-cloud will be highly concentrated in the density dimension, and concentrated around a slanting line in the subspace of mass/volume. (Indeed, the cloud here is actually a surface, because of the relation VD = M.)

"Radial categories" are how cognitive psychologists describe the non-Aristotelian boundaries of words. The central "mother" conceives her child, gives birth to it, and supports it. Is an egg donor who never sees her child a mother? She is the "genetic mother". What about a woman who is implanted with a foreign embryo and bears it to term? She is a "surrogate mother". And the woman who raises a child that isn't hers genetically? Why, she's an "adoptive mother". The Aristotelian syllogism would run, "Humans have ten fingers, Fred has nine fingers, therefore Fred is not a human" but the way we actually think is "Humans have ten fingers, Fred is a human, therefore Fred is a 'nine-fingered human'."

We can think about the radial-ness of categories in intensional terms, as described above - properties that are usually present, but optionally absent. If we thought about the intension of the word "mother", it might be like a distributed glow in thingspace, a glow whose intensity matches the degree to which that volume of thingspace matches the category "mother". The glow is concentrated in the center of genetics and birth and child-raising; the volume of egg donors would also glow, but less brightly.

Or we can think about the radial-ness of categories extensionally. Suppose we mapped all the birds in the world into thingspace, using a distance metric that corresponds as well as possible to perceived similarity in humans: A robin is more similar to another robin, than either is similar to a pigeon, but robins and pigeons are all more similar to each other than either is to a penguin, etcetera.

Then the center of all birdness would be densely populated by many neighboring tight clusters, robins and sparrows and canaries and pigeons and many other species. Eagles and falcons and other large predatory birds would occupy a nearby cluster. Penguins would be in a more distant cluster, and likewise chickens and ostriches.

The result might look, indeed, something like an astronomical cluster: many galaxies orbiting the center, and a few outliers.

Or we could think simultaneously about both the intension of the cognitive category "bird", and its extension in real-world birds: The central clusters of robins and sparrows glowing brightly with highly typical birdness; satellite clusters of ostriches and penguins glowing more dimly with atypical birdness, and Abraham Lincoln a few megaparsecs away and glowing not at all.

I prefer that last visualization - the glowing points - because as I see it, the structure of the cognitive intension followed from the extensional cluster structure. First came the structure-in-the-world, the empirical distribution of birds over thingspace; then, by observing it, we formed a category whose intensional glow roughly overlays this structure.

This gives us yet another view of why words are not Aristotelian classes: the empirical clustered structure of the real universe is not so crystalline. A natural cluster, a group of things highly similar to each other, may have no set of necessary and sufficient properties - no set of characteristics that all group members have, and no non-members have.

But even if a category is irrecoverably blurry and bumpy, there's no need to panic. I would not object if someone said that birds are "feathered flying things". But penguins don't fly! - well, fine. The usual rule has an exception; it's not the end of the world. Definitions can't be expected to exactly match the empirical structure of thingspace in any event, because the map is smaller and much less complicated than the territory. The point of the definition "feathered flying things" is to lead the listener to the bird cluster, not to give a total description of every existing bird down to the molecular level.

When you draw a boundary around a group of extensional points empirically clustered in thingspace, you may find at least one exception to every simple intensional rule you can invent.

But if a definition works well enough in practice to point out the intended empirical cluster, objecting to it may justly be called "nitpicking".

Sequence: A Human's Guide to Words

Typicality and Asymmetrical Similarity

Disguised Queries

Referenced by: Disguised Queries • Neural Categories • Categorizing Has Consequences • Arguing "By Definition" • Where to Draw the Boundary? • Mutual Information, and Density in Thingspace • Superexponential Conceptspace, and Simple Words • 37 Ways That Words Can Be Wrong • Reductive Reference • Classical Configuration Spaces • No Logical Positivist I • Unnatural Categories • Selling Nonapples • The Nature of Logic

Original with comments: The Cluster Structure of Thingspace

Disguised Queries

Eliezer Yudkowsky, 09 February 2008 12:05AM

Followup to: The Cluster Structure of Thingspace

Imagine that you have a peculiar job in a peculiar factory: Your task is to take objects from a mysterious conveyor belt, and sort the objects into two bins. When you first arrive, Susan the Senior Sorter explains to you that blue egg-shaped objects are called "bleggs" and go in the "blegg bin", while red cubes are called "rubes" and go in the "rube bin".

Once you start working, you notice that bleggs and rubes differ in ways besides color and shape. Bleggs have fur on their surface, while rubes are smooth. Bleggs flex slightly to the touch; rubes are hard. Bleggs are opaque; the rube's surface slightly translucent.

Soon after you begin working, you encounter a blegg shaded an unusually dark blue - in fact, on closer examination, the color proves to be purple, halfway between red and blue.

Yet wait! Why are you calling this object a "blegg"? A "blegg" was originally defined as blue and egg-shaped - the qualification of blueness appears in the very name "blegg", in fact. This object is not blue. One of the necessary qualifications is missing; you should call this a "purple egg-shaped object", not a "blegg".

But it so happens that, in addition to being purple and egg-shaped, the object is also furred, flexible, and opaque. So when you saw the object, you thought, "Oh, a strangely colored blegg." It certainly isn't a rube... right?

Still, you aren't quite sure what to do next. So you call over Susan the Senior Sorter.

"Oh, yes, it's a blegg," Susan says, "you can put it in the blegg bin."
You start to toss the purple blegg into the blegg bin, but pause for a moment. "Susan," you say, "how do you know this is a blegg?"
Susan looks at you oddly. "Isn't it obvious? This object may be purple, but it's still egg-shaped, furred, flexible, and opaque, like all the other bleggs. You've got to expect a few color defects. Or is this one of those philosophical conundrums, like 'How do you know the world wasn't created five minutes ago complete with false memories?' In a philosophical sense I'm not absolutely certain that this is a blegg, but it seems like a good guess."
"No, I mean..." You pause, searching for words. "Why is there a blegg bin and a rube bin? What's the difference between bleggs and rubes?"
"Bleggs are blue and egg-shaped, rubes are red and cube-shaped," Susan says patiently. "You got the standard orientation lecture, right?"
"Why do bleggs and rubes need to be sorted?"
"Er... because otherwise they'd be all mixed up?" says Susan. "Because nobody will pay us to sit around all day and not sort bleggs and rubes?"
"Who originally determined that the first blue egg-shaped object was a 'blegg', and how did they determine that?"
Susan shrugs. "I suppose you could just as easily call the red cube-shaped objects 'bleggs' and the blue egg-shaped objects 'rubes', but it seems easier to remember this way."
You think for a moment. "Suppose a completely mixed-up object came off the conveyor. Like, an orange sphere-shaped furred translucent object with writhing green tentacles. How could I tell whether it was a blegg or a rube?"
"Wow, no one's ever found an object that mixed up," says Susan, "but I guess we'd take it to the sorting scanner."
"How does the sorting scanner work?" you inquire. "X-rays? Magnetic resonance imaging? Fast neutron transmission spectroscopy?"
"I'm told it works by Bayes's Rule, but I don't quite understand how," says Susan. "I like to say it, though. Bayes Bayes Bayes Bayes Bayes."
"What does the sorting scanner tell you?"
"It tells you whether to put the object into the blegg bin or the rube bin. That's why it's called a sorting scanner."
At this point you fall silent.
"Incidentally," Susan says casually, "it may interest you to know that bleggs contain small nuggets of vanadium ore, and rubes contain shreds of palladium, both of which are useful industrially."
"Susan, you are pure evil."
"Thank you."

So now it seems we've discovered the heart and essence of bleggness: a blegg is an object that contains a nugget of vanadium ore. Surface characteristics, like blue color and furredness, do not determine whether an object is a blegg; surface characteristics only matter because they help you infer whether an object is a blegg, that is, whether the object contains vanadium.

Containing vanadium is a necessary and sufficient definition: all bleggs contain vanadium and everything that contains vanadium is a blegg: "blegg" is just a shorthand way of saying "vanadium-containing object." Right?

Not so fast, says Susan: Around 98% of bleggs contain vanadium, but 2% contain palladium instead. To be precise (Susan continues) around 98% of blue egg-shaped furred flexible opaque objects contain vanadium. For unusual bleggs, it may be a different percentage: 95% of purple bleggs contain vanadium, 92% of hard bleggs contain vanadium, etc.

Now suppose you find a blue egg-shaped furred flexible opaque object, an ordinary blegg in every visible way, and just for kicks you take it to the sorting scanner, and the scanner says "palladium" - this is one of the rare 2%. Is it a blegg?

At first you might answer that, since you intend to throw this object in the rube bin, you might as well call it a "rube". However, it turns out that almost all bleggs, if you switch off the lights, glow faintly in the dark; while almost all rubes do not glow in the dark. And the percentage of bleggs that glow in the dark is not significantly different for blue egg-shaped furred flexible opaque objects that contain palladium, instead of vanadium. Thus, if you want to guess whether the object glows like a blegg, or remains dark like a rube, you should guess that it glows like a blegg.

So is the object really a blegg or a rube?

On one hand, you'll throw the object in the rube bin no matter what else you learn. On the other hand, if there are any unknown characteristics of the object you need to infer, you'll infer them as if the object were a blegg, not a rube - group it into the similarity cluster of blue egg-shaped furred flexible opaque things, and not the similarity cluster of red cube-shaped smooth hard translucent things.

The question "Is this object a blegg?" may stand in for different queries on different occasions.

If it weren't standing in for some query, you'd have no reason to care.

Is atheism a "religion"? Is transhumanism a "cult"? People who argue that atheism is a religion "because it states beliefs about God" are really trying to argue (I think) that the reasoning methods used in atheism are on a par with the reasoning methods used in religion, or that atheism is no safer than religion in terms of the probability of causally engendering violence, etc... What's really at stake is an atheist's claim of substantial difference and superiority relative to religion, which the religious person is trying to reject by denying the difference rather than the superiority(!)

But that's not the a priori irrational part: The a priori irrational part is where, in the course of the argument, someone pulls out a dictionary and looks up the definition of "atheism" or "religion". (And yes, it's just as silly whether an atheist or religionist does it.) How could a dictionary possibly decide whether an empirical cluster of atheists is really substantially different from an empirical cluster of theologians? How can reality vary with the meaning of a word? The points in thingspace don't move around when we redraw a boundary.

But people often don't realize that their argument about where to draw a definitional boundary, is really a dispute over whether to infer a characteristic shared by most things inside an empirical cluster...

Hence the phrase, "disguised query".

Sequence: A Human's Guide to Words

The Cluster Structure of Thingspace

Neural Categories

Referenced by: Neural Categories • How An Algorithm Feels From Inside • Feel the Meaning • The Argument from Common Usage • Arguing "By Definition" • Searching for Bayes-Structure • 37 Ways That Words Can Be Wrong • Dissolving the Question • Invisible Frameworks • Unnatural Categories

Original with comments: Disguised Queries

Neural Categories

Eliezer Yudkowsky, 10 February 2008 12:33AM

Followup to: Disguised Queries

In Disguised Queries, I talked about a classification task of "bleggs" and "rubes". The typical blegg is blue, egg-shaped, furred, flexible, opaque, glows in the dark, and contains vanadium. The typical rube is red, cube-shaped, smooth, hard, translucent, unglowing, and contains palladium. For the sake of simplicity, let us forget the characteristics of flexibility/hardness and opaqueness/translucency. This leaves five dimensions in thingspace: Color, shape, texture, luminance, and interior.

Suppose I want to create an Artificial Neural Network (ANN) to predict unobserved blegg characteristics from observed blegg characteristics. And suppose I'm fairly naive about ANNs: I've read excited popular science books about how neural networks are distributed, emergent, and parallel just like the human brain!! but I can't derive the differential equations for gradient descent in a non-recurrent multilayer network with sigmoid units (which is actually a lot easier than it sounds).

Then I might design a neural network that looks something like this:

Network 1 is for classifying bleggs and rubes. But since "blegg" is an unfamiliar and synthetic concept, I've also included a similar Network 1b for distinguishing humans from Space Monsters, with input from Aristotle ("All men are mortal") and Plato's Academy ("A featherless biped with broad nails").

A neural network needs a learning rule. The obvious idea is that when two nodes are often active at the same time, we should strengthen the connection between them - this is one of the first rules ever proposed for training a neural network, known as Hebb's Rule.

Thus, if you often saw things that were both blue and furred - thus simultaneously activating the "color" node in the + state and the "texture" node in the + state - the connection would strengthen between color and texture, so that + colors activated + textures, and vice versa. If you saw things that were blue and egg-shaped and vanadium-containing, that would strengthen positive mutual connections between color and shape and interior.

Let's say you've already seen plenty of bleggs and rubes come off the conveyor belt. But now you see something that's furred, egg-shaped, and - gasp! - reddish purple (which we'll model as a "color" activation level of -2/3). You haven't yet tested the luminance, or the interior. What to predict, what to predict?

What happens then is that the activation levels in Network 1 bounce around a bit. Positive activation flows luminance from shape, negative activation flows to interior from color, negative activation flows from interior to luminance... Of course all these messages are passed in parallel!! and asynchronously!! just like the human brain...

Finally Network 1 settles into a stable state, which has high positive activation for "luminance" and "interior". The network may be said to "expect" (though it has not yet seen) that the object will glow in the dark, and that it contains vanadium.

And lo, Network 1 exhibits this behavior even though there's no explicit node that says whether the object is a blegg or not. The judgment is implicit in the whole network!! Bleggness is an attractor!! which arises as the result of emergent behavior!! from the distributed!! learning rule.

Now in real life, this kind of network design - however faddish it may sound - runs into all sorts of problems. Recurrent networks don't always settle right away: They can oscillate, or exhibit chaotic behavior, or just take a very long time to settle down. This is a Bad Thing when you see something big and yellow and striped, and you have to wait five minutes for your distributed neural network to settle into the "tiger" attractor. Asynchronous and parallel it may be, but it's not real-time.

And there are other problems, like double-counting the evidence when messages bounce back and forth: If you suspect that an object glows in the dark, your suspicion will activate belief that the object contains vanadium, which in turn will activate belief that the object glows in the dark.

Plus if you try to scale up the Network 1 design, it requires O(N^2) connections, where N is the total number of observables.

So what might be a more realistic neural network design?

In this network, a wave of activation converges on the central node from any clamped (observed) nodes, and then surges back out again to any unclamped (unobserved) nodes. Which means we can compute the answer in one step, rather than waiting for the network to settle - an important requirement in biology when the neurons only run at 20Hz. And the network architecture scales as O(N), rather than O(N^2).

Admittedly, there are some things you can notice more easily with the first network architecture than the second. Network 1 has a direct connection between every two nodes. So if red objects never glow in the dark, but red furred objects usually have the other blegg characteristics like egg-shape and vanadium, Network 1 can easily represent this: it just takes a very strong direct negative connection from color to luminance, but more powerful positive connections from texture to all other nodes except luminance.

Nor is this a "special exception" to the general rule that bleggs glow - remember, in Network 1, there is no unit that represents blegg-ness; blegg-ness emerges as an attractor in the distributed network.

So yes, those N^2 connections were buying us something. But not very much. Network 1 is not more useful on most real-world problems, where you rarely find an animal stuck halfway between being a cat and a dog.

(There are also facts that you can't easily represent in Network 1 or Network 2. Let's say sea-blue color and spheroid shape, when found together, always indicate the presence of palladium; but when found individually, without the other, they are each very strong evidence for vanadium. This is hard to represent, in either architecture, without extra nodes. Both Network 1 and Network 2 embody implicit assumptions about what kind of environmental structure is likely to exist; the ability to read this off is what separates the adults from the babes, in machine learning.)

Make no mistake: Neither Network 1, nor Network 2, are biologically realistic. But it still seems like a fair guess that however the brain really works, it is in some sense closer to Network 2 than Network 1. Fast, cheap, scalable, works well to distinguish dogs and cats: natural selection goes for that sort of thing like water running down a fitness landscape.

It seems like an ordinary enough task to classify objects as either bleggs or rubes, tossing them into the appropriate bin. But would you notice if sea-blue objects never glowed in the dark?

Maybe, if someone presented you with twenty objects that were alike only in being sea-blue, and then switched off the light, and none of the objects glowed. If you got hit over the head with it, in other words. Perhaps by presenting you with all these sea-blue objects in a group, your brain forms a new subcategory, and can detect the "doesn't glow" characteristic within that subcategory. But you probably wouldn't notice if the sea-blue objects were scattered among a hundred other bleggs and rubes. It wouldn't be easy or intuitive to notice, the way that distinguishing cats and dogs is easy and intuitive.

Or: "Socrates is human, all humans are mortal, therefore Socrates is mortal." How did Aristotle know that Socrates was human? Well, Socrates had no feathers, and broad nails, and walked upright, and spoke Greek, and, well, was generally shaped like a human and acted like one. So the brain decides, once and for all, that Socrates is human; and from there, infers that Socrates is mortal like all other humans thus yet observed. It doesn't seem easy or intuitive to ask how much wearing clothes, as opposed to using language, is associated with mortality. Just, "things that wear clothes and use language are human" and "humans are mortal".

Are there biases associated with trying to classify things into categories once and for all? Of course there are. See e.g. Cultish Countercultishness.

To be continued...

Sequence: A Human's Guide to Words

Disguised Queries

How An Algorithm Feels From Inside

Referenced by: How An Algorithm Feels From Inside • Feel the Meaning • Categorizing Has Consequences • Searching for Bayes-Structure • 37 Ways That Words Can Be Wrong • Dissolving the Question

Original with comments: Neural Categories

How An Algorithm Feels From Inside

Eliezer Yudkowsky, 11 February 2008 02:35AM

Followup to: Neural Categories

"If a tree falls in the forest, and no one hears it, does it make a sound?" I remember seeing an actual argument get started on this subject - a fully naive argument that went nowhere near Berkeleyan subjectivism. Just:

"It makes a sound, just like any other falling tree!"
"But how can there be a sound that no one hears?"

The standard rationalist view would be that the first person is speaking as if "sound" means acoustic vibrations in the air; the second person is speaking as if "sound" means an auditory experience in a brain. If you ask "Are there acoustic vibrations?" or "Are there auditory experiences?", the answer is at once obvious. And so the argument is really about the definition of the word "sound".

I think the standard analysis is essentially correct. So let's accept that as a premise, and ask: Why do people get into such an argument? What's the underlying psychology?

A key idea of the heuristics and biases program is that mistakes are often more revealing of cognition than correct answers. Getting into a heated dispute about whether, if a tree falls in a deserted forest, it makes a sound, is traditionally considered a mistake.

So what kind of mind design corresponds to that error?

In Disguised Queries I introduced the blegg/rube classification task, in which Susan the Senior Sorter explains that your job is to sort objects coming off a conveyor belt, putting the blue eggs or "bleggs" into one bin, and the red cubes or "rubes" into the rube bin. This, it turns out, is because bleggs contain small nuggets of vanadium ore, and rubes contain small shreds of palladium, both of which are useful industrially.

Except that around 2% of blue egg-shaped objects contain palladium instead. So if you find a blue egg-shaped thing that contains palladium, should you call it a "rube" instead? You're going to put it in the rube bin - why not call it a "rube"?

But when you switch off the light, nearly all bleggs glow faintly in the dark. And blue egg-shaped objects that contain palladium are just as likely to glow in the dark as any other blue egg-shaped object.

So if you find a blue egg-shaped object that contains palladium, and you ask "Is it a blegg?", the answer depends on what you have to do with the answer: If you ask "Which bin does the object go in?", then you choose as if the object is a rube. But if you ask "If I turn off the light, will it glow?", you predict as if the object is a blegg. In one case, the question "Is it a blegg?" stands in for the disguised query, "Which bin does it go in?". In the other case, the question "Is it a blegg?" stands in for the disguised query, "Will it glow in the dark?"

Now suppose that you have an object that is blue and egg-shaped and contains palladium; and you have already observed that it is furred, flexible, opaque, and glows in the dark.

This answers every query, observes every observable introduced. There's nothing left for a disguised query to stand for.

So why might someone feel an impulse to go on arguing whether the object is really a blegg?

This diagram from Neural Categories shows two different neural networks that might be used to answer questions about bleggs and rubes. Network 1 has a number of disadvantages - such as potentially oscillating/chaotic behavior, or requiring O(N^2) connections - but Network 1's structure does have one major advantage over Network 2: Every unit in the network corresponds to a testable query. If you observe every observable, clamping every value, there are no units in the network left over.

Network 2, however, is a far better candidate for being something vaguely like how the human brain works: It's fast, cheap, scalable - and has an extra dangling unit in the center, whose activation can still vary, even after we've observed every single one of the surrounding nodes.

Which is to say that even after you know whether an object is blue or red, egg or cube, furred or smooth, bright or dark, and whether it contains vanadium or palladium, it feels like there's a leftover, unanswered question: But is it really a blegg?

Usually, in our daily experience, acoustic vibrations and auditory experience go together. But a tree falling in a deserted forest unbundles this common association. And even after you know that the falling tree creates acoustic vibrations but not auditory experience, it feels like there's a leftover question: Did it make a sound?

We know where Pluto is, and where it's going; we know Pluto's shape, and Pluto's mass - but is it a planet?

Now remember: When you look at Network 2, as I've laid it out here, you're seeing the algorithm from the outside. People don't think to themselves, "Should the central unit fire, or not?" any more than you think "Should neuron #12,234,320,242 in my visual cortex fire, or not?"

It takes a deliberate effort to visualize your brain from the outside - and then you still don't see your actual brain; you imagine what you think is there, hopefully based on science, but regardless, you don't have any direct access to neural network structures from introspection. That's why the ancient Greeks didn't invent computational neuroscience.

When you look at Network 2, you are seeing from the outside; but the way that neural network structure feels from the inside, if you yourself are a brain running that algorithm, is that even after you know every characteristic of the object, you still find yourself wondering: "But is it a blegg, or not?"

This is a great gap to cross, and I've seen it stop people in their tracks - albeit this is the first time I've tried explaining it at this length. Because we don't instinctively see our intuitions as "intuitions", we just see them as the world. When you look at a green cup, you don't think of yourself as seeing a picture reconstructed in your visual cortex - although that is what you are seeing - you just see a green cup. You think, "Why, look, this cup is green," not, "The picture in my visual cortex of this cup is green."

And in the same way, when people argue over whether the falling tree makes a sound, or whether Pluto is a planet, they don't see themselves as arguing over whether a categorization should be active in their neural networks. It seems like either the tree makes a sound, or not.

We know where Pluto is, and where it's going; we know Pluto's shape, and Pluto's mass - but is it a planet? And yes, there were people who said this was a fight over definitions - but even that is a Network 2 sort of perspective, because you're arguing about how the central unit ought to be wired up. If you were a mind constructed along the lines of Network 1, you wouldn't say "It depends on how you define 'planet'," you would just say, "Given that we know Pluto's orbit and shape and mass, there is no question left to ask." Or, rather, that's how it would feel - it would feel like there was no question left - if you were a mind constructed along the lines of Network 1.

Before you can question your intuitions, you have to realize that what your mind's eye is looking at is an intuition - some cognitive algorithm, as seen from the inside - rather than a direct perception of the Way Things Really Are.

People cling to their intuitions, I think, not so much because they believe their cognitive algorithms are perfectly reliable, but because they can't see their intuitions as the way their cognitive algorithms happen to look from the inside.

And so everything you try to say about how the native cognitive algorithm goes astray, ends up being contrasted to their direct perception of the Way Things Really Are - and discarded as obviously wrong.

Sequence: A Human's Guide to Words

Neural Categories

Disputing Definitions

Referenced by: Disputing Definitions • Feel the Meaning • Fallacies of Compression • Searching for Bayes-Structure • Variable Question Fallacies • 37 Ways That Words Can Be Wrong • Dissolving the Question • Righting a Wrong Question • Mind Projection Fallacy • Qualitatively Confused • Reductionism • Angry Atoms • Heat vs. Motion • Relative Configuration Space • Timeless Physics • Why Quantum? • Possibility and Could-ness • Heading Toward Morality • Probability is Subjectively Objective • Setting Up Metaethics • The Meaning of Right • Extenuating Circumstances

Original with comments: How An Algorithm Feels From Inside

Disputing Definitions

Eliezer Yudkowsky, 12 February 2008 12:15AM

Followup to: How An Algorithm Feels From Inside

I have watched more than one conversation - even conversations supposedly about cognitive science - go the route of disputing over definitions. Taking the classic example to be "If a tree falls in a forest, and no one hears it, does it make a sound?", the dispute often follows a course like this:

If a tree falls in the forest, and no one hears it, does it make a sound?

Albert: "Of course it does. What kind of silly question is that? Every time I've listened to a tree fall, it made a sound, so I'll guess that other trees falling also make sounds. I don't believe the world changes around when I'm not looking."

Barry: "Wait a minute. If no one hears it, how can it be a sound?"

In this example, Barry is arguing with Albert because of a genuinely different intuition about what constitutes a sound. But there's more than one way the Standard Dispute can start. Barry could have a motive for rejecting Albert's conclusion. Or Barry could be a skeptic who, upon hearing Albert's argument, reflexively scrutinized it for possible logical flaws; and then, on finding a counterargument, automatically accepted it without applying a second layer of search for a counter-counterargument; thereby arguing himself into the opposite position. This doesn't require that Barry's prior intuition - the intuition Barry would have had, if we'd asked him before Albert spoke - have differed from Albert's.

Well, if Barry didn't have a differing intuition before, he sure has one now.

Albert: "What do you mean, there's no sound? The tree's roots snap, the trunk comes crashing down and hits the ground. This generates vibrations that travel through the ground and the air. That's where the energy of the fall goes, into heat and sound. Are you saying that if people leave the forest, the tree violates conservation of energy?"

Barry: "But no one hears anything. If there are no humans in the forest, or, for the sake of argument, anything else with a complex nervous system capable of 'hearing', then no one hears a sound."

Albert and Barry recruit arguments that feel like support for their respective positions, describing in more detail the thoughts that caused their "sound"-detectors to fire or stay silent. But so far the conversation has still focused on the forest, rather than definitions. And note that they don't actually disagree on anything that happens in the forest.

Albert: "This is the dumbest argument I've ever been in. You're a niddlewicking fallumphing pickleplumber."

Barry: "Yeah? Well, you look like your face caught on fire and someone put it out with a shovel."

Insult has been proffered and accepted; now neither party can back down without losing face. Technically, this isn't part of the argument, as rationalists account such things; but it's such an important part of the Standard Dispute that I'm including it anyway.

Albert: "The tree produces acoustic vibrations. By definition, that is a sound."

Barry: "No one hears anything. By definition, that is not a sound."

The argument starts shifting to focus on definitions. Whenever you feel tempted to say the words "by definition" in an argument that is not literally about pure mathematics, remember that anything which is true "by definition" is true in all possible worlds, and so observing its truth can never constrain which world you live in.

Albert: "My computer's microphone can record a sound without anyone being around to hear it, store it as a file, and it's called a 'sound file'. And what's stored in the file is the pattern of vibrations in air, not the pattern of neural firings in anyone's brain. 'Sound' means a pattern of vibrations."

Albert deploys an argument that feels like support for the word "sound" having a particular meaning. This is a different kind of question from whether acoustic vibrations take place in a forest - but the shift usually passes unnoticed.

Barry: "Oh, yeah? Let's just see if the dictionary agrees with you."

There's a lot of things I could be curious about in the falling-tree scenario. I could go into the forest and look at trees, or learn how to derive the wave equation for changes of air pressure, or examine the anatomy of an ear, or study the neuroanatomy of the auditory cortex. Instead of doing any of these things, I am to consult a dictionary, apparently. Why? Are the editors of the dictionary expert botanists, expert physicists, expert neuroscientists? Looking in an encyclopedia might make sense, but why a dictionary?

Albert: "Hah! Definition 2c in Merriam-Webster: 'Sound: Mechanical radiant energy that is transmitted by longitudinal pressure waves in a material medium (as air).'"

Barry: "Hah! Definition 2b in Merriam-Webster: 'Sound: The sensation perceived by the sense of hearing.'"

Albert and Barry, chorus: "Consarned dictionary! This doesn't help at all!"

Dictionary editors are historians of usage, not legislators of language. Dictionary editors find words in current usage, then write down the words next to (a small part of) what people seem to mean by them. If there's more than one usage, the editors write down more than one definition.

Albert: "Look, suppose that I left a microphone in the forest and recorded the pattern of the acoustic vibrations of the tree falling. If I played that back to someone, they'd call it a 'sound'! That's the common usage! Don't go around making up your own wacky definitions!"

Barry: "One, I can define a word any way I like so long as I use it consistently. Two, the meaning I gave was in the dictionary. Three, who gave you the right to decide what is or isn't common usage?"

There's quite a lot of rationality errors in the Standard Dispute. Some of them I've already covered, and some of them I've yet to cover; likewise the remedies.

But for now, I would just like to point out - in a mournful sort of way - that Albert and Barry seem to agree on virtually every question of what is actually going on inside the forest, and yet it doesn't seem to generate any feeling of agreement.

Arguing about definitions is a garden path; people wouldn't go down the path if they saw at the outset where it led. If you asked Albert (Barry) why he's still arguing, he'd probably say something like: "Barry (Albert) is trying to sneak in his own definition of 'sound', the scurvey scoundrel, to support his ridiculous point; and I'm here to defend the standard definition."

But suppose I went back in time to before the start of the argument:

(Eliezer appears from nowhere in a peculiar conveyance that looks just like the time machine from the original 'The Time Machine' movie.)

Barry: "Gosh! A time traveler!"

Eliezer: "I am a traveler from the future! Hear my words! I have traveled far into the past - around fifteen minutes -"

Albert: "Fifteen minutes?"

Eliezer: "- to bring you this message!"

(There is a pause of mixed confusion and expectancy.)

Eliezer: "Do you think that 'sound' should be defined to require both acoustic vibrations (pressure waves in air) and also auditory experiences (someone to listen to the sound), or should 'sound' be defined as meaning only acoustic vibrations, or only auditory experience?"

Barry: "You went back in time to ask us that?"

Eliezer: "My purposes are my own! Answer!"

Albert: "Well... I don't see why it would matter. You can pick any definition so long as you use it consistently."

Barry: "Flip a coin. Er, flip a coin twice."

Eliezer: "Personally I'd say that if the issue arises, both sides should switch to describing the event in unambiguous lower-level constituents, like acoustic vibrations or auditory experiences. Or each side could designate a new word, like 'alberzle' and 'bargulum', to use for what they respectively used to call 'sound'; and then both sides could use the new words consistently. That way neither side has to back down or lose face, but they can still communicate. And of course you should try to keep track, at all times, of some testable proposition that the argument is actually about. Does that sound right to you?"

Albert: "I guess..."

Barry: "Why are we talking about this?"

Eliezer: "To preserve your friendship against a contingency you will, now, never know. For the future has already changed!"

(Eliezer and the machine vanish in a puff of smoke.)

Barry: "Where were we again?"

Albert: "Oh, yeah: If a tree falls in the forest, and no one hears it, does it make a sound?"

Barry: "It makes an alberzle but not a bargulum. What's the next question?"

This remedy doesn't destroy every dispute over categorizations. But it destroys a substantial fraction.

Sequence: A Human's Guide to Words

How An Algorithm Feels From Inside

Feel the Meaning

Referenced by: Feel the Meaning • The Argument from Common Usage • Variable Question Fallacies • 37 Ways That Words Can Be Wrong • Dissolving the Question • Wrong Questions • Mind Projection Fallacy • 2-Place and 1-Place Words • Unnatural Categories • Whence Your Abstractions? • What Do We Mean By "Rationality"?

Original with comments: Disputing Definitions

Feel the Meaning

Eliezer Yudkowsky, 13 February 2008 01:01AM

Followup to: Disputing Definitions

When I hear someone say, "Oh, look, a butterfly," the spoken phonemes "butterfly" enter my ear and vibrate on my ear drum, being transmitted to the cochlea, tickling auditory nerves that transmit activation spikes to the auditory cortex, where phoneme processing begins, along with recognition of words, and reconstruction of syntax (a by no means serial process), and all manner of other complications.

But at the end of the day, or rather, at the end of the second, I am primed to look where my friend is pointing and see a visual pattern that I will recognize as a butterfly; and I would be quite surprised to see a wolf instead.

My friend looks at a butterfly, his throat vibrates and lips move, the pressure waves travel invisibly through the air, my ear hears and my nerves transduce and my brain reconstructs, and lo and behold, I know what my friend is looking at. Isn't that marvelous? If we didn't know about the pressure waves in the air, it would be a tremendous discovery in all the newspapers: Humans are telepathic! Human brains can transfer thoughts to each other!

Well, we are telepathic, in fact; but magic isn't exciting when it's merely real, and all your friends can do it too.

Think telepathy is simple? Try building a computer that will be telepathic with you. Telepathy, or "language", or whatever you want to call our partial thought transfer ability, is more complicated than it looks.

But it would be quite inconvenient to go around thinking, "Now I shall partially transduce some features of my thoughts into a linear sequence of phonemes which will invoke similar thoughts in my conversational partner..."

So the brain hides the complexity - or rather, never represents it in the first place - which leads people to think some peculiar thoughts about words.

As I remarked earlier, when a large yellow striped object leaps at me, I think "Yikes! A tiger!" not "Hm... objects with the properties of largeness, yellowness, and stripedness have previously often possessed the properties 'hungry' and 'dangerous', and therefore, although it is not logically necessary, auughhhh CRUNCH CRUNCH GULP."

Similarly, when someone shouts "Yikes! A tiger!", natural selection would not favor an organism that thought, "Hm... I have just heard the syllables 'Tie' and 'Grr' which my fellow tribe members associate with their internal analogues of my own tiger concept, and which they are more likely to utter if they see an object they categorize as aiiieeee CRUNCH CRUNCH help it's got my arm CRUNCH GULP".

Considering this as a design constraint on the human cognitive architecture, you wouldn't want any extra steps between when your auditory cortex recognizes the syllables "tiger", and when the tiger concept gets activated.

Going back to the parable of bleggs and rubes, and the centralized network that categorizes quickly and cheaply, you might visualize a direct connection running from the unit that recognizes the syllable "blegg", to the unit at the center of the blegg network. The central unit, the blegg concept, gets activated almost as soon as you hear Susan the Senior Sorter say "Blegg!"

Or, for purposes of talking - which also shouldn't take eons - as soon as you see a blue egg-shaped thing and the central blegg unit fires, you holler "Blegg!" to Susan.

And what that algorithm feels like from inside is that the label, and the concept, are very nearly identified; the meaning feels like an intrinsic property of the word itself.

The cognoscenti will recognize this as yet another case of E. T. Jaynes's "Mind Projection Fallacy". It feels like a word has a meaning, as a property of the word itself; just like how redness is a property of a red apple, or mysteriousness is a property of a mysterious phenomenon.

Indeed, on most occasions, the brain will not distinguish at all between the word and the meaning - only bothering to separate the two while learning a new language, perhaps. And even then, you'll see Susan pointing to a blue egg-shaped thing and saying "Blegg!", and you'll think, I wonder what "blegg" means, and not, I wonder what mental category Susan associates to the auditory label "blegg".

Consider, in this light, the part of the Standard Dispute of Definitions where the two parties argue about what the word "sound" really means - the same way they might argue whether a particular apple is really red or green:

Albert: "My computer's microphone can record a sound without anyone being around to hear it, store it as a file, and it's called a 'sound file'. And what's stored in the file is the pattern of vibrations in air, not the pattern of neural firings in anyone's brain. 'Sound' means a pattern of vibrations."

Barry: "Oh, yeah? Let's just see if the dictionary agrees with you."

Albert feels intuitively that the word "sound" has a meaning and that the meaning is acoustic vibrations. Just as Albert feels that a tree falling in the forest makes a sound (rather than causing an event that matches the sound category).

Barry likewise feels that:

sound.meaning == auditory experiences
forest.sound == false

Rather than:

myBrain.FindConcept("sound") == concept_AuditoryExperience
concept_AuditoryExperience.match(forest) == false

Which is closer to what's really going on; but humans have not evolved to know this, anymore than humans instinctively know the brain is made of neurons.

Albert and Barry's conflicting intuitions provide the fuel for continuing the argument in the phase of arguing over what the word "sound" means - which feels like arguing over a fact like any other fact, like arguing over whether the sky is blue or green.

You may not even notice that anything has gone astray, until you try to perform the rationalist ritual of stating a testable experiment whose result depends on the facts you're so heatedly disputing...

Sequence: A Human's Guide to Words

Disputing Definitions

The Argument from Common Usage

Referenced by: The Argument from Common Usage • Where to Draw the Boundary? • Searching for Bayes-Structure • 37 Ways That Words Can Be Wrong • Dissolving the Question • Timeless Beauty • The Ultimate Source • Heading Toward Morality • Probability is Subjectively Objective • Humans in Funny Suits • Three Fallacies of Teleology

Original with comments: Feel the Meaning

The Argument from Common Usage

Eliezer Yudkowsky, 13 February 2008 04:24PM

Followup to: Feel the Meaning

Part of the Standard Definitional Dispute runs as follows:

Albert: "Look, suppose that I left a microphone in the forest and recorded the pattern of the acoustic vibrations of the tree falling. If I played that back to someone, they'd call it a 'sound'! That's the common usage! Don't go around making up your own wacky definitions!"

Barry: "One, I can define a word any way I like so long as I use it consistently. Two, the meaning I gave was in the dictionary. Three, who gave you the right to decide what is or isn't common usage?"

Not all definitional disputes progress as far as recognizing the notion of common usage. More often, I think, someone picks up a dictionary because they believe that words have meanings, and the dictionary faithfully records what this meaning is. Some people even seem to believe that the dictionary determines the meaning - that the dictionary editors are the Legislators of Language. Maybe because back in elementary school, their authority-teacher said that they had to obey the dictionary, that it was a mandatory rule rather than an optional one?

Dictionary editors read what other people write, and record what the words seem to mean; they are historians. The Oxford English Dictionary may be comprehensive, but never authoritative.

But surely there is a social imperative to use words in a commonly understood way? Does not our human telepathy, our valuable power of language, rely on mutual coordination to work? Perhaps we should voluntarily treat dictionary editors as supreme arbiters - even if they prefer to think of themselves as historians - in order to maintain the quiet cooperation on which all speech depends.

The phrase "authoritative dictionary" is almost never used correctly, an example of proper usage being the Authoritative Dictionary of IEEE Standards. The IEEE is a body of voting members who have a professional need for exact agreement on terms and definitions, and so the Authoritative Dictionary of IEEE Standards is actual, negotiated legislation, which exerts whatever authority one regards as residing in the IEEE.

In everyday life, shared language usually does not arise from a deliberate agreement, as of the IEEE. It's more a matter of infection, as words are invented and diffuse through the culture. (A "meme", one might say, following Richard Dawkins thirty years ago - but you already know what I mean, and if not, you can look it up on Google, and then you too will have been infected.)

Yet as the example of the IEEE shows, agreement on language can also be a cooperatively established public good. If you and I wish to undergo an exchange of thoughts via language, the human telepathy, then it is in our mutual interest that we use the same word for similar concepts - preferably, concepts similar to the limit of resolution in our brain's representation thereof - even though we have no obvious mutual interest in using any particular word for a concept.

We have no obvious mutual interest in using the word "oto" to mean sound, or "sound" to mean oto; but we have a mutual interest in using the same word, whichever word it happens to be. (Preferably, words we use frequently should be short, but let's not get into information theory just yet.)

But, while we have a mutual interest, it is not strictly necessary that you and I use the similar labels internally; it is only convenient. If I know that, to you, "oto" means sound - that is, you associate "oto" to a concept very similar to the one I associate to "sound" - then I can say "Paper crumpling makes a crackling oto." It requires extra thought, but I can do it if I want.

Similarly, if you say "What is the walking-stick of a bowling ball dropping on the floor?" and I know which concept you associate with the syllables "walking-stick", then I can figure out what you mean. It may require some thought, and give me pause, because I ordinarily associate "walking-stick" with a different concept. But I can do it just fine.

When humans really want to communicate with each other, we're hard to stop! If we're stuck on a deserted island with no common language, we'll take up sticks and draw pictures in sand.

Albert's appeal to the Argument from Common Usage assumes that agreement on language is a cooperatively established public good. Yet Albert assumes this for the sole purpose of rhetorically accusing Barry of breaking the agreement, and endangering the public good. Now the falling-tree argument has gone all the way from botany to semantics to politics; and so Barry responds by challenging Albert for the authority to define the word.

A rationalist, with the discipline of hugging the query active, would notice that the conversation had gone rather far astray.

Oh, dear reader, is it all really necessary? Albert knows what Barry means by "sound". Barry knows what Albert means by "sound". Both Albert and Barry have access to words, such as "acoustic vibrations" or "auditory experience", which they already associate to the same concepts, and which can describe events in the forest without ambiguity. If they were stuck on a deserted island, trying to communicate with each other, their work would be done.

When both sides know what the other side wants to say, and both sides accuse the other side of defecting from "common usage", then whatever it is they are about, it is clearly not working out a way to communicate with each other. But this is the whole benefit that common usage provides in the first place.

Why would you argue about the meaning of a word, two sides trying to wrest it back and forth? If it's just a namespace conflict that has gotten blown out of proportion, and nothing more is at stake, then the two sides need merely generate two new words and use them consistently.

Yet often categorizations function as hidden inferences and disguised queries. Is atheism a "religion"? If someone is arguing that the reasoning methods used in atheism are on a par with the reasoning methods used in Judaism, or that atheism is on a par with Islam in terms of causally engendering violence, then they have a clear argumentative stake in lumping it all together into an indistinct gray blur of "faith".

Or consider the fight to blend together blacks and whites as "people". This would not be a time to generate two words - what's at stake is exactly the idea that you shouldn't draw a moral distinction.

But once any empirical proposition is at stake, or any moral proposition, you can no longer appeal to common usage.

If the question is how to cluster together similar things for purposes of inference, empirical predictions will depend on the answer; which means that definitions can be wrong. A conflict of predictions cannot be settled by an opinion poll.

If you want to know whether atheism should be clustered with supernaturalist religions for purposes of some particular empirical inference, the dictionary can't answer you.

If you want to know whether blacks are people, the dictionary can't answer you.

If everyone believes that the red light in the sky is Mars the God of War, the dictionary will define "Mars" as the God of War. If everyone believes that fire is the release of phlogiston, the dictionary will define "fire" as the release of phlogiston.

There is an art to using words; even when definitions are not literally true or false, they are often wiser or more foolish. Dictionaries are mere histories of past usage; if you treat them as supreme arbiters of meaning, it binds you to the wisdom of the past, forbidding you to do better.

Though do take care to ensure (if you must depart from the wisdom of the past) that people can figure out what you're trying to swim.

Sequence: A Human's Guide to Words

Feel the Meaning

Empty Labels

Referenced by: Empty Labels • Where to Draw the Boundary? • Entropy, and Short Codes • 37 Ways That Words Can Be Wrong • Reductionism • What Do We Mean By "Rationality"? • Conversation Halters

Original with comments: The Argument from Common Usage

Empty Labels

Eliezer Yudkowsky, 14 February 2008 11:50PM

Followup to: The Argument from Common Usage

Consider (yet again) the Aristotelian idea of categories. Let's say that there's some object with properties A, B, C, D, and E, or at least it looks E-ish.

Fred: "You mean that thing over there is blue, round, fuzzy, and -"
Me: "In Aristotelian logic, it's not supposed to make a difference what the properties are, or what I call them. That's why I'm just using the letters."

Next, I invent the Aristotelian category "zawa", which describes those objects, all those objects, and only those objects, which have properties A, C, and D.

Me: "Object 1 is zawa, B, and E."
Fred: "And it's blue - I mean, A - too, right?"
Me: "That's implied when I say it's zawa."
Fred: "Still, I'd like you to say it explicitly."
Me: "Okay. Object 1 is A, B, zawa, and E."

Then I add another word, "yokie", which describes all and only objects that are B and E; and the word "xippo", which describes all and only objects which are E but not D.

Me: "Object 1 is zawa and yokie, but not xippo."
Fred: "Wait, is it luminescent? I mean, is it E?"
Me: "Yes. That is the only possibility on the information given."
Fred: "I'd rather you spelled it out."
Me: "Fine: Object 1 is A, zawa, B, yokie, C, D, E, and not xippo."
Fred: "Amazing! You can tell all that just by looking?"

Impressive, isn't it? Let's invent even more new words: "Bolo" is A, C, and yokie; "mun" is A, C, and xippo; and "merlacdonian" is bolo and mun.

Pointlessly confusing? I think so too. Let's replace the labels with the definitions:

"Zawa, B, and E" becomes [A, C, D], B, E
"Bolo and A" becomes [A, C, [B, E]], A
"Merlacdonian" becomes [A, C, [B, E]], [A, C, [E, ~D]]

And the thing to remember about the Aristotelian idea of categories is that [A, C, D] is the entire information of "zawa". It's not just that I can vary the label, but that I can get along just fine without any label at all - the rules for Aristotelian classes work purely on structures like [A, C, D]. To call one of these structures "zawa", or attach any other label to it, is a human convenience (or inconvenience) which makes not the slightest difference to the Aristotelian rules.

Let's say that "human" is to be defined as a mortal featherless biped. Then the classic syllogism would have the form:

All [mortal, ~feathers, bipedal] are mortal.
Socrates is a [mortal, ~feathers, bipedal].
Therefore, Socrates is mortal.

The feat of reasoning looks a lot less impressive now, doesn't it?

Here the illusion of inference comes from the labels, which conceal the premises, and pretend to novelty in the conclusion. Replacing labels with definitions reveals the illusion, making visible the tautology's empirical unhelpfulness. You can never say that Socrates is a [mortal, ~feathers, biped] until you have observed him to be mortal.

There's an idea, which you may have noticed I hate, that "you can define a word any way you like". This idea came from the Aristotelian notion of categories; since, if you follow the Aristotelian rules exactly and without flaw - which humans never do; Aristotle knew perfectly well that Socrates was human, even though that wasn't justified under his rules - but, if some imaginary nonhuman entity were to follow the rules exactly, they would never arrive at a contradiction. They wouldn't arrive at much of anything: they couldn't say that Socrates is a [mortal, ~feathers, biped] until they observed him to be mortal.

But it's not so much that labels are arbitrary in the Aristotelian system, as that the Aristotelian system works fine without any labels at all - it cranks out exactly the same stream of tautologies, they just look a lot less impressive. The labels are only there to create the illusion of inference.

So if you're going to have an Aristotelian proverb at all, the proverb should be, not "I can define a word any way I like," nor even, "Defining a word never has any consequences," but rather, "Definitions don't need words."

Sequence: A Human's Guide to Words

The Argument from Common Usage

Taboo Your Words

Referenced by: Taboo Your Words • Mutual Information, and Density in Thingspace • 37 Ways That Words Can Be Wrong

Original with comments: Empty Labels

Classic Sichuan in Millbrae, Thu Feb 21, 7pm

Eliezer Yudkowsky, 15 February 2008 12:21AM

Followup to: Bay Area Bayesians Unite, OB Meetup

The Bay Area Overcoming Bias meetup will take place in the Classic Sichuan restaurant, 148 El Camino Real, Millbrae, CA 94031. 15 people said they would "Definitely" attend and an additional 27 said "Maybe". Oh, and Robin Hanson will be there too.

Dinner is scheduled for 7:00pm, on Thursday, February 21st, 2008. I'll show up at 6:30pm, though, just to cut people some antislack if it's easier for them to arrive earlier.

If you're arriving via the BART/Caltrain station, just walk up from the Southbound Caltrain side and turn right onto El Camino, walk a few meters, and you're there.

If driving, I'd suggest taking the exit from 101 onto Millbrae Ave - the exit from 280 onto Millbrae surprisingly goes down a winding mountain road before arriving at downtown. Doesn't mean you have to take 101 the whole way there, but I definitely recommend the 101 exit.

For parking, I would suggest parking near the BART/Caltrain station. From 101 onto Millbrae Ave., turn right onto El Camino, almost immediately pass the Peter's Cafe parking lot and then turn right onto a small street toward the train station. The first parking lot you see on your right is reserved for Peter's Cafe, but immediately after that (still on your right) is some city parking that looked mostly empty when I visited last Thursday. El Camino itself was parked up, though. If all else fails, you should be able to park in the train station lots and pay a small fee. Then walk up to El Camino and turn right, as before.

Classic Sichuan has vegetarian dishes. They also have a reputation for their spicy food being spicy, so watch out! I ate there to check quality, and while I'm generally a cultural barbarian, I didn't detect any problems with the food I was served. Depending on how many people actually show up, we may overflow their small private room, but hopefully we won't overflow the restaurant.

My cellphone number is (866) 983-5697. That's toll-free, 866-YUDKOWS.

Long live the Bayesian Conspiracy! See you there!

Original with comments: Classic Sichuan in Millbrae, Thu Feb 21, 7pm

Taboo Your Words

Eliezer Yudkowsky, 15 February 2008 10:53PM

Followup to: Empty Labels

In the game Taboo (by Hasbro), the objective is for a player to have their partner guess a word written on a card, without using that word or five additional words listed on the card. For example, you might have to get your partner to say "baseball" without using the words "sport", "bat", "hit", "pitch", "base" or of course "baseball".

The existence of this game surprised me, when I discovered it. Why wouldn't you just say "An artificial group conflict in which you use a long wooden cylinder to whack a thrown spheroid, and then run between four safe positions"?

But then, by the time I discovered the game, I'd already been practicing it for years - albeit with a different purpose.

Yesterday we saw how replacing terms with definitions could reveal the empirical unproductivity of the classical Aristotelian syllogism:

All [mortal, ~feathers, biped] are mortal;
Socrates is a [mortal, ~feathers, biped];
Therefore Socrates is mortal.

But the principle applies much more broadly:

Albert: "A tree falling in a deserted forest makes a sound."
Barry: "A tree falling in a deserted forest does not make a sound."

Clearly, since one says "sound" and one says "~sound", we must have a contradiction, right? But suppose that they both dereference their pointers before speaking:

Albert: "A tree falling in a deserted forest matches [membership test: this event generates acoustic vibrations]."
Barry: "A tree falling in a deserted forest does not match [membership test: this event generates auditory experiences]."

Now there is no longer an apparent collision - all they had to do was prohibit themselves from using the word sound. If "acoustic vibrations" came into dispute, we would just play Taboo again and say "pressure waves in a material medium"; if necessary we would play Taboo again on the word "wave" and replace it with the wave equation. (Play Taboo on "auditory experience" and you get "That form of sensory processing, within the human brain, which takes as input a linear time series of frequency mixes.")

But suppose, on the other hand, that Albert and Barry were to have the argument:

Albert: "Socrates matches the concept [membership test: this person will die after drinking hemlock]."
Barry: "Socrates matches the concept [membership test: this person will not die after drinking hemlock]."

Now Albert and Barry have a substantive clash of expectations; a difference in what they anticipate seeing after Socrates drinks hemlock. But they might not notice this, if they happened to use the same word "human" for their different concepts.

You get a very different picture of what people agree or disagree about, depending on whether you take a label's-eye-view (Albert says "sound" and Barry says "not sound", so they must disagree) or taking the test's-eye-view (Albert's membership test is acoustic vibrations, Barry's is auditory experience).

Get together a pack of soi-disant futurists and ask them if they believe we'll have Artificial Intelligence in thirty years, and I would guess that at least half of them will say yes. If you leave it at that, they'll shake hands and congratulate themselves on their consensus. But make the term "Artificial Intelligence" taboo, and ask them to describe what they expect to see, without ever using words like "computers" or "think", and you might find quite a conflict of expectations hiding under that featureless standard word. Likewise that other term. And see also Shane Legg's compilation of 71 definitions of "intelligence".

The illusion of unity across religions can be dispelled by making the term "God" taboo, and asking them to say what it is they believe in; or making the word "faith" taboo, and asking them why they believe it. Though mostly they won't be able to answer at all, because it is mostly profession in the first place, and you cannot cognitively zoom in on an audio recording.

When you find yourself in philosophical difficulties, the first line of defense is not to define your problematic terms, but to see whether you can think without using those terms at all. Or any of their short synonyms. And be careful not to let yourself invent a new word to use instead. Describe outward observables and interior mechanisms; don't use a single handle, whatever that handle may be.

Albert says that people have "free will". Barry says that people don't have "free will". Well, that will certainly generate an apparent conflict. Most philosophers would advise Albert and Barry to try to define exactly what they mean by "free will", on which topic they will certainly be able to discourse at great length. I would advise Albert and Barry to describe what it is that they think people do, or do not have, without using the phrase "free will" at all. (If you want to try this at home, you should also avoid the words "choose", "act", "decide", "determined", "responsible", or any of their synonyms.)

This is one of the nonstandard tools in my toolbox, and in my humble opinion, it works way way better than the standard one. It also requires more effort to use; you get what you pay for.

Sequence: A Human's Guide to Words

Empty Labels

Replace the Symbol with the Substance

Referenced by: Replace the Symbol with the Substance • Fallacies of Compression • Where to Draw the Boundary? • Variable Question Fallacies • 37 Ways That Words Can Be Wrong • Possibility and Could-ness • Heading Toward Morality • 2-Place and 1-Place Words • Setting Up Metaethics • The Meaning of Right • "Arbitrary" • Dreams of AI Design • Raised in Technophilia • What Do We Mean By "Rationality"?

Original with comments: Taboo Your Words

Replace the Symbol with the Substance

Eliezer Yudkowsky, 16 February 2008 06:12PM

Continuation of: Taboo Your Words
Followup to: Original Seeing, Lost Purposes

What does it take to - as in yesterday's example - see a "baseball game" as "An artificial group conflict in which you use a long wooden cylinder to whack a thrown spheroid, and then run between four safe positions"? What does it take to play the rationalist version of Taboo, in which the goal is not to find a synonym that isn't on the card, but to find a way of describing without the standard concept-handle?

You have to visualize. You have to make your mind's eye see the details, as though looking for the first time. You have to perform an Original Seeing.

Is that a "bat"? No, it's a long, round, tapering, wooden rod, narrowing at one end so that a human can grasp and swing it.

Is that a "ball"? No, it's a leather-covered spheroid with a symmetrical stitching pattern, hard but not metal-hard, which someone can grasp and throw, or strike with the wooden rod, or catch.

Are those "bases"? No, they're fixed positions on a game field, that players try to run to as quickly as possible because of their safety within the game's artificial rules.

The chief obstacle to performing an original seeing is that your mind already has a nice neat summary, a nice little easy-to-use concept handle. Like the word "baseball", or "bat", or "base". It takes an effort to stop your mind from sliding down the familiar path, the easy path, the path of least resistance, where the small featureless word rushes in and obliterates the details you're trying to see. A word itself can have the destructive force of cliche; a word itself can carry the poison of a cached thought.

Playing the game of Taboo - being able to describe without using the standard pointer/label/handle - is one of the fundamental rationalist capacities. It occupies the same primordial level as the habit of constantly asking "Why?" or "What does this belief make me anticipate?"

The art is closely related to:

- Pragmatism, because seeing in this way often gives you a much closer connection to anticipated experience, rather than propositional belief;

- Reductionism, because seeing in this way often forces you to drop down to a lower level of organization, look at the parts instead of your eye skipping over the whole;

- Hugging the query, because words often distract you from the question you really want to ask;

- Avoiding cached thoughts, which will rush in using standard words, so you can block them by tabooing standard words;

- The writer's rule of "Show, don't tell!", which has power among rationalists;

- And not losing sight of your original purpose.

How could tabooing a word help you keep your purpose?

From Lost Purposes:

As you read this, some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper.

Why are you going to "school"? To get an "education" ending in a "degree". Blank out the forbidden words and all their obvious synonyms, visualize the actual details, and you're much more likely to notice that "school" currently seems to consist of sitting next to bored teenagers listening to material you already know, that a "degree" is a piece of paper with some writing on it, and that "education" is forgetting the material as soon as you're tested on it.

Leaky generalizations often manifest through categorizations: People who actually learn in classrooms are categorized as "getting an education", so "getting an education" must be good; but then anyone who actually shows up at a college will also match against the concept "getting an education", whether or not they learn.

Students who understand math will do well on tests, but if you require schools to produce good test scores, they'll spend all their time teaching to the test. A mental category, that imperfectly matches your goal, can produce the same kind of incentive failure internally. You want to learn, so you need an "education"; and then as long as you're getting anything that matches against the category "education", you may not notice whether you're learning or not. Or you'll notice, but you won't realize you've lost sight of your original purpose, because you're "getting an education" and that's how you mentally described your goal.

To categorize is to throw away information. If you're told that a falling tree makes a "sound", you don't know what the actual sound is; you haven't actually heard the tree falling. If a coin lands "heads", you don't know its radial orientation. A blue egg-shaped thing may be a "blegg", but what if the exact egg shape varies, or the exact shade of blue? You want to use categories to throw away irrelevant information, to sift gold from dust, but often the standard categorization ends up throwing out relevant information too. And when you end up in that sort of mental trouble, the first and most obvious solution is to play Taboo.

For example: "Play Taboo" is itself a leaky generalization. Hasbro's version is not the rationalist version; they only list five additional banned words on the card, and that's not nearly enough coverage to exclude thinking in familiar old words. What rationalists do would count as playing Taboo - it would match against the "play Taboo" concept - but not everything that counts as playing Taboo works to force original seeing. If you just think "play Taboo to force original seeing", you'll start thinking that anything that counts as playing Taboo must count as original seeing.

The rationalist version isn't a game, which means that you can't win by trying to be clever and stretching the rules. You have to play Taboo with a voluntary handicap: Stop yourself from using synonyms that aren't on the card. You also have to stop yourself from inventing a new simple word or phrase that functions as an equivalent mental handle to the old one. You are trying to zoom in on your map, not rename the cities; dereference the pointer, not allocate a new pointer; see the events as they happen, not rewrite the cliche in a different wording.

By visualizing the problem in more detail, you can see the lost purpose: Exactly what do you do when you "play Taboo"? What purpose does each and every part serve?

If you see your activities and situation originally, you will be able to originally see your goals as well. If you can look with fresh eyes, as though for the first time, you will see yourself doing things that you would never dream of doing if they were not habits.

Purpose is lost whenever the substance (learning, knowledge, health) is displaced by the symbol (a degree, a test score, medical care). To heal a lost purpose, or a lossy categorization, you must do the reverse:

Replace the symbol with the substance; replace the signifier with the signified; replace the property with the membership test; replace the word with the meaning; replace the label with the concept; replace the summary with the details; replace the proxy question with the real question; dereference the pointer; drop into a lower level of organization; mentally simulate the process instead of naming it; zoom in on your map.

"The Simple Truth" was generated by an exercise of this discipline to describe "truth" on a lower level of organization, without invoking terms like "accurate", "correct", "represent", "reflect", "semantic", "believe", "knowledge", "map", or "real". (And remember that the goal is not really to play Taboo - the word "true" appears in the text, but not to define truth. It would get a buzzer in Hasbro's game, but we're not actually playing that game. Ask yourself whether the document fulfilled its purpose, not whether it followed the rules.)

Bayes's Rule itself describes "evidence" in pure math, without using words like "implies", "means", "supports", "proves", or "justifies". Set out to define such philosophical terms, and you'll just go in circles.

And then there's the most important word of all to Taboo. I've often warned that you should be careful not to overuse it, or even avoid the concept in certain cases. Now you know the real reason why. It's not a bad subject to think about. But your true understanding is measured by your ability to describe what you're doing and why, without using that word or any of its synonyms.

Sequence: A Human's Guide to Words

Taboo Your Words

Fallacies of Compression

Referenced by: Fallacies of Compression • Sneaking in Connotations • Arguing "By Definition" • Where to Draw the Boundary? • Mutual Information, and Density in Thingspace • 37 Ways That Words Can Be Wrong • Dissolving the Question • Wrong Questions • Decoherent Essences • Timeless Beauty • Heading Toward Morality • Setting Up Metaethics • The Meaning of Right • "Arbitrary" • Dreams of AI Design • ESR's comments on some EY:OB/LW posts • "Outside View!" as Conversation-Halter

Original with comments: Replace the Symbol with the Substance

Fallacies of Compression

Eliezer Yudkowsky, 17 February 2008 06:51PM

Followup to: Replace the Symbol with the Substance

"The map is not the territory," as the saying goes. The only life-size, atomically detailed, 100% accurate map of California is California. But California has important regularities, such as the shape of its highways, that can be described using vastly less information - not to mention vastly less physical material - than it would take to describe every atom within the state borders. Hence the other saying: "The map is not the territory, but you can't fold up the territory and put it in your glove compartment."

A paper map of California, at a scale of 10 kilometers to 1 centimeter (a million to one), doesn't have room to show the distinct position of two fallen leaves lying a centimeter apart on the sidewalk. Even if the map tried to show the leaves, the leaves would appear as the same point on the map; or rather the map would need a feature size of 10 nanometers, which is a finer resolution than most book printers handle, not to mention human eyes.

Reality is very large - just the part we can see is billions of lightyears across. But your map of reality is written on a few pounds of neurons, folded up to fit inside your skull. I don't mean to be insulting, but your skull is tiny, comparatively speaking.

Inevitably, then, certain things that are distinct in reality, will be compressed into the same point on your map.

But what this feels like from inside is not that you say, "Oh, look, I'm compressing two things into one point on my map." What it feels like from inside is that there is just one thing, and you are seeing it.

A sufficiently young child, or a sufficiently ancient Greek philosopher, would not know that there were such things as "acoustic vibrations" or "auditory experiences". There would just be a single thing that happened when a tree fell; a single event called "sound".

To realize that there are two distinct events, underlying one point on your map, is an essentially scientific challenge - a big, difficult scientific challenge.

Sometimes fallacies of compression result from confusing two known things under the same label - you know about acoustic vibrations, and you know about auditory processing in brains, but you call them both "sound" and so confuse yourself. But the more dangerous fallacy of compression arises from having no idea whatsoever that two distinct entities even exist. There is just one mental folder in the filing system, labeled "sound", and everything thought about "sound" drops into that one folder. It's not that there are two folders with the same label; there's just a single folder. By default, the map is compressed; why would the brain create two mental buckets where one would serve?

Or think of a mystery novel in which the detective's critical insight is that one of the suspects has an identical twin. In the course of the detective's ordinary work, his job is just to observe that Carol is wearing red, that she has black hair, that her sandals are leather - but all these are facts about Carol. It's easy enough to question an individual fact, like WearsRed(Carol) or BlackHair(Carol). Maybe BlackHair(Carol) is false. Maybe Carol dyes her hair. Maybe BrownHair(Carol). But it takes a subtler detective to wonder if the Carol in WearsRed(Carol) and BlackHair(Carol) - the Carol file into which his observations drop - should be split into two files. Maybe there are two Carols, so that the Carol who wore red is not the same woman as the Carol who had black hair.

Here it is the very act of creating two different buckets that is the stroke of genius insight. 'Tis easier to question one's facts than one's ontology.

The map of reality contained in a human brain, unlike a paper map of California, can expand dynamically when we write down more detailed descriptions. But what this feels like from inside is not so much zooming in on a map, as fissioning an indivisible atom - taking one thing (it felt like one thing) and splitting it into two or more things.

Often this manifests in the creation of new words, like "acoustic vibrations" and "auditory experiences" instead of just "sound". Something about creating the new name seems to allocate the new bucket. The detective is liable to start calling one of his suspects "Carol-2" or "the Other Carol" almost as soon as he realizes that there are two of them.

But expanding the map isn't always as simple as generating new city names. It is a stroke of scientific insight to realize that such things as acoustic vibrations, or auditory experiences, even exist.

The obvious modern-day illustration would be words like "intelligence" or "consciousness". Every now and then one sees a press release claiming that a research has "explained consciousness" because a team of neurologists investigated a 40Hz electrical rhythm that might have something to do with cross-modality binding of sensory information, or because they investigated the reticular activating system that keeps humans awake. That's an extreme example, and the usual failures are more subtle, but they are of the same kind. The part of "consciousness" that people find most interesting is reflectivity, self-awareness, realizing that the person I see in the mirror is "me"; that and the hard problem of subjective experience as distinguished by Chalmers. We also label "conscious" the state of being awake, rather than asleep, in our daily cycle. But they are all different concepts going under the same name, and the underlying phenomena are different scientific puzzles. You can explain being awake without explaining reflectivity or subjectivity.

Fallacies of compression also underlie the bait-and-switch technique in philosophy - you argue about "consciousness" under one definition (like the ability to think about thinking) and then apply the conclusions to "consciousness" under a different definition (like subjectivity). Of course it may be that the two are the same thing, but if so, genuinely understanding this fact would require first a conceptual split and then a genius stroke of reunification.

Expanding your map is (I say again) a scientific challenge: part of the art of science, the skill of inquiring into the world. (And of course you cannot solve a scientific challenge by appealing to dictionaries, nor master a complex skill of inquiry by saying "I can define a word any way I like".) Where you see a single confusing thing, with protean and self-contradictory attributes, it is a good guess that your map is cramming too much into one point - you need to pry it apart and allocate some new buckets. This is not like defining the single thing you see, but it does often follow from figuring out how to talk about the thing without using a single mental handle.

So the skill of prying apart the map is linked to the rationalist version of Taboo, and to the wise use of words; because words often represent the points on our map, the labels under which we file our propositions and the buckets into which we drop our information. Avoiding a single word, or allocating new ones, is often part of the skill of expanding the map.

Sequence: A Human's Guide to Words

Replace the Symbol with the Substance

Categorizing Has Consequences

Referenced by: Categorizing Has Consequences • Where to Draw the Boundary? • Variable Question Fallacies • 37 Ways That Words Can Be Wrong • The Quotation is not the Referent • The Generalized Anti-Zombie Principle • Why Quantum? • Probability is Subjectively Objective • Morality as Fixed Computation • Three Fallacies of Teleology • Moore's Paradox

Original with comments: Fallacies of Compression

Categorizing Has Consequences

Eliezer Yudkowsky, 19 February 2008 01:40AM

Followup to: Fallacies of Compression

Among the many genetic variations and mutations you carry in your genome, there are a very few alleles you probably know - including those determining your blood type: the presence or absence of the A, B, and + antigens. If you receive a blood transfusion containing an antigen you don't have, it will trigger an allergic reaction. It was Karl Landsteiner's discovery of this fact, and how to test for compatible blood types, that made it possible to transfuse blood without killing the patient. (1930 Nobel Prize in Medicine.) Also, if a mother with blood type A (for example) bears a child with blood type A+, the mother may acquire an allergic reaction to the + antigen; if she has another child with blood type A+, the child will be in danger, unless the mother takes an allergic suppressant during pregnancy. Thus people learn their blood types before they marry.

Oh, and also: people with blood type A are earnest and creative, while people with blood type B are wild and cheerful. People with type O are agreeable and sociable, while people with type AB are cool and controlled. (You would think that O would be the absence of A and B, while AB would just be A plus B, but no...) All this, according to the Japanese blood type theory of personality. It would seem that blood type plays the role in Japan that astrological signs play in the West, right down to blood type horoscopes in the daily newspaper.

This fad is especially odd because blood types have never been mysterious, not in Japan and not anywhere. We only know blood types even exist thanks to Karl Landsteiner. No mystic witch doctor, no venerable sorcerer, ever said a word about blood types; there are no ancient, dusty scrolls to shroud the error in the aura of antiquity. If the medical profession claimed tomorrow that it had all been a colossal hoax, we layfolk would not have one scrap of evidence from our unaided senses to contradict them.

There's never been a war between blood types. There's never even been a political conflict between blood types. The stereotypes must have arisen strictly from the mere existence of the labels.

Now, someone is bound to point out that this is a story of categorizing humans. Does the same thing happen if you categorize plants, or rocks, or office furniture? I can't recall reading about such an experiment, but of course, that doesn't mean one hasn't been done. (I'd expect the chief difficulty of doing such an experiment would be finding a protocol that didn't mislead the subjects into thinking that, since the label was given you, it must be significant somehow.) So while I don't mean to update on imaginary evidence, I would predict a positive result for the experiment: I would expect them to find that mere labeling had power over all things, at least in the human imagination.

You can see this in terms of similarity clusters: once you draw a boundary around a group, the mind starts trying to harvest similarities from the group. And unfortunately the human pattern-detectors seem to operate in such overdrive that we see patterns whether they're there or not; a weakly negative correlation can be mistaken for a strong positive one with a bit of selective memory.

You can see this in terms of neural algorithms: creating a name for a set of things is like allocating a subnetwork to find patterns in them.

You can see this in terms of a compression fallacy: things given the same name end up dumped into the same mental bucket, blurring them together into the same point on the map.

Or you can see this in terms of the boundless human ability to make stuff up out of thin air and believe it because no one can prove it's wrong. As soon as you name the category, you can start making up stuff about it. The named thing doesn't have to be perceptible; it doesn't have to exist; it doesn't even have to be coherent.

And no, it's not just Japan: Here in the West, a blood-type-based diet book called Eat Right 4 Your Type was a bestseller.

Any way you look at it, drawing a boundary in thingspace is not a neutral act. Maybe a more cleanly designed, more purely Bayesian AI could ponder an arbitrary class and not be influenced by it. But you, a human, do not have that option. Categories are not static things in the context of a human brain; as soon as you actually think of them, they exert force on your mind. One more reason not to believe you can define a word any way you like.

Sequence: A Human's Guide to Words

Fallacies of Compression

Sneaking in Connotations

Referenced by: Sneaking in Connotations • 37 Ways That Words Can Be Wrong • Surface Analogies and Deep Causes • The Nature of Logic

Original with comments: Categorizing Has Consequences

Sneaking in Connotations

Eliezer Yudkowsky, 19 February 2008 07:41PM

Followup to: Categorizing Has Consequences

Yesterday, we saw that in Japan, blood types have taken the place of astrology - if your blood type is AB, for example, you're supposed to be "cool and controlled".

So suppose we decided to invent a new word, "wiggin", and defined this word to mean people with green eyes and black hair -

A green-eyed man with black hair walked into a restaurant.
"Ha," said Danny, watching from a nearby table, "did you see that? A wiggin just walked into the room. Bloody wiggins. Commit all sorts of crimes, they do."
His sister Erda sighed. "You haven't seen him commit any crimes, have you, Danny?"
"Don't need to," Danny said, producing a dictionary. "See, it says right here in the Oxford English Dictionary. 'Wiggin. (1) A person with green eyes and black hair.' He's got green eyes and black hair, he's a wiggin. You're not going to argue with the Oxford English Dictionary, are you? By definition, a green-eyed black-haired person is a wiggin."
"But you called him a wiggin," said Erda. "That's a nasty thing to say about someone you don't even know. You've got no evidence that he puts too much ketchup on his burgers, or that as a kid he used his slingshot to launch baby squirrels."
"But he is a wiggin," Danny said patiently. "He's got green eyes and black hair, right? Just you watch, as soon as his burger arrives, he's reaching for the ketchup."

The human mind passes from observed characteristics to inferred characteristics via the medium of words. In "All humans are mortal, Socrates is a human, therefore Socrates is mortal", the observed characteristics are Socrates's clothes, speech, tool use, and generally human shape; the categorization is "human"; the inferred characteristic is poisonability by hemlock.

Of course there's no hard distinction between "observed characteristics" and "inferred characteristics". If you hear someone speak, they're probably shaped like a human, all else being equal. If you see a human figure in the shadows, then ceteris paribus it can probably speak.

And yet some properties do tend to be more inferred than observed. You're more likely to decide that someone is human, and will therefore burn if exposed to open flame, than carry through the inference the other way around.

If you look in a dictionary for the definition of "human", you're more likely to find characteristics like "intelligence" and "featherless biped" - characteristics that are useful for quickly eyeballing what is and isn't a human - rather than the ten thousand connotations, from vulnerability to hemlock, to overconfidence, that we can infer from someone's being human. Why? Perhaps dictionaries are intended to let you match up labels to similarity groups, and so are designed to quickly isolate clusters in thingspace. Or perhaps the big, distinguishing characteristics are the most salient, and therefore first to pop into a dictionary editor's mind. (I'm not sure how aware dictionary editors are of what they really do.)

But the upshot is that when Danny pulls out his OED to look up "wiggin", he sees listed only the first-glance characteristics that distinguish a wiggin: Green eyes and black hair. The OED doesn't list the many minor connotations that have come to attach to this term, such as criminal proclivities, culinary peculiarities, and some unfortunate childhood activities.

How did those connotations get there in the first place? Maybe there was once a famous wiggin with those properties. Or maybe someone made stuff up at random, and wrote a series of bestselling books about it (The Wiggin, Talking to Wiggins, Raising Your Little Wiggin, Wiggins in the Bedroom). Maybe even the wiggins believe it now, and act accordingly. As soon as you call some people "wiggins", the word will begin acquiring connotations.

But remember the Parable of Hemlock: If we go by the logical class definitions, we can never class Socrates as a "human" until after we observe him to be mortal. Whenever someone pulls a dictionary, they're generally trying to sneak in a connotation, not the actual definition written down in the dictionary.

After all, if the only meaning of the word "wiggin" is "green-eyed black-haired person", then why not just call those people "green-eyed black-haired people"? And if you're wondering whether someone is a ketchup-reacher, why not ask directly, "Is he a ketchup-reacher?" rather than "Is he a wiggin?" (Note substitution of substance for symbol.)

Oh, but arguing the real question would require work. You'd have to actually watch the wiggin to see if he reached for the ketchup. Or maybe see if you can find statistics on how many green-eyed black-haired people actually like ketchup. At any rate, you wouldn't be able to do it sitting in your living room with your eyes closed. And people are lazy. They'd rather argue "by definition", especially since they think "you can define a word any way you like".

But of course the real reason they care whether someone is a "wiggin" is a connotation - a feeling that comes along with the word - that isn't in the definition they claim to use.

Imagine Danny saying, "Look, he's got green eyes and black hair. He's a wiggin! It says so right there in the dictionary! - therefore, he's got black hair. Argue with that, if you can!"

Doesn't have much of a triumphant ring to it, does it? If the real point of the argument actually was contained in the dictionary definition - if the argument genuinely was logically valid - then the argument would feel empty; it would either say nothing new, or beg the question.

It's only the attempt to smuggle in connotations not explicitly listed in the definition, that makes anyone feel they can score a point that way.

Sequence: A Human's Guide to Words

Categorizing Has Consequences

Arguing "By Definition"

Referenced by: Arguing "By Definition" • Mutual Information, and Density in Thingspace • 37 Ways That Words Can Be Wrong

Original with comments: Sneaking in Connotations

Arguing "By Definition"

Eliezer Yudkowsky, 20 February 2008 11:37PM

Followup to: Sneaking in Connotations

"This plucked chicken has two legs and no feathers - therefore, by definition, it is a human!"

When people argue definitions, they usually start with some visible, known, or at least widely believed set of characteristics; then pull out a dictionary, and point out that these characteristics fit the dictionary definition; and so conclude, "Therefore, by definition, atheism is a religion!"

But visible, known, widely believed characteristics are rarely the real point of a dispute. Just the fact that someone thinks Socrates's two legs are evident enough to make a good premise for the argument, "Therefore, by definition, Socrates is human!" indicates that bipedalism probably isn't really what's at stake - or the listener would reply, "Whaddaya mean Socrates is bipedal? That's what we're arguing about in the first place!"

Now there is an important sense in which we can legitimately move from evident characteristics to not-so-evident ones. You can, legitimately, see that Socrates is human-shaped, and predict his vulnerability to hemlock. But this probabilistic inference does not rely on dictionary definitions or common usage; it relies on the universe containing empirical clusters of similar things.

This cluster structure is not going to change depending on how you define your words. Even if you look up the dictionary definition of "human" and it says "all featherless bipeds except Socrates", that isn't going to change the actual degree to which Socrates is similar to the rest of us featherless bipeds.

When you are arguing correctly from cluster structure, you'll say something like, "Socrates has two arms, two feet, a nose and tongue, speaks fluent Greek, uses tools, and in every aspect I've been able to observe him, seems to have every major and minor property that characterizes Homo sapiens; so I'm going to guess that he has human DNA, human biochemistry, and is vulnerable to hemlock just like all other Homo sapiens in whom hemlock has been clinically tested for lethality."

And suppose I reply, "But I saw Socrates out in the fields with some herbologists; I think they were trying to prepare an antidote. Therefore I don't expect Socrates to keel over after he drinks the hemlock - he will be an exception to the general behavior of objects in his cluster: they did not take an antidote, and he did."

Now there's not much point in arguing over whether Socrates is "human" or not. The conversation has to move to a more detailed level, poke around inside the details that make up the "human" category - talk about human biochemistry, and specifically, the neurotoxic effects of coniine.

If you go on insisting, "But Socrates is a human and humans, by definition, are mortal!" then what you're really trying to do is blur out everything you know about Socrates except the fact of his humanity - insist that the only correct prediction is the one you would make if you knew nothing about Socrates except that he was human.

Which is like insisting that a coin is 50% likely to be showing heads or tails, because it is a "fair coin", after you've actually looked at the coin and it's showing heads. It's like insisting that Frodo has ten fingers, because most hobbits have ten fingers, after you've already looked at his hands and seen nine fingers. Naturally this is illegal under Bayesian probability theory: You can't just refuse to condition on new evidence.

And you can't just keep one categorization and make estimates based on that, while deliberately throwing out everything else you know.

Not every piece of new evidence makes a significant difference, of course. If I see that Socrates has nine fingers, this isn't going to noticeably change my estimate of his vulnerability to hemlock, because I'll expect that the way Socrates lost his finger didn't change the rest of his biochemistry. And this is true, whether or not the dictionary's definition says that human beings have ten fingers. The legal inference is based on the cluster structure of the environment, and the causal structure of biology; not what the dictionary editor writes down, nor even "common usage".

Now ordinarily, when you're doing this right - in a legitimate way - you just say, "The coniine alkaloid found in hemlock produces muscular paralysis in humans, resulting in death by asphyxiation." Or more simply, "Humans are vulnerable to hemlock." That's how it's usually said in a legitimate argument.

When would someone feel the need to strengthen the argument with the emphatic phrase "by definition"? (I.e. "Humans are vulnerable to hemlock by definition!") Why, when the inferred characteristic has been called into doubt - Socrates has been seen consulting herbologists - and so the speaker feels the need to tighten the vise of logic.

So when you see "by definition" used like this, it usually means: "Forget what you've heard about Socrates consulting herbologists - humans, by definition, are mortal!"

People feel the need to squeeze the argument onto a single course by saying "Any P, by definition, has property Q!", on exactly those occasions when they see, and prefer to dismiss out of hand, additional arguments that call into doubt the default inference based on clustering.

So too with the argument "X, by definition, is a Y!" E.g., "Atheists believe that God doesn't exist; therefore atheists have beliefs about God, because a negative belief is still a belief; therefore atheism asserts answers to theological questions; therefore atheism is, by definition, a religion."

You wouldn't feel the need to say, "Hinduism, by definition, is a religion!" because, well, of course Hinduism is a religion. It's not just a religion "by definition", it's, like, an actual religion.

Atheism does not resemble the central members of the "religion" cluster, so if it wasn't for the fact that atheism is a religion by definition, you might go around thinking that atheism wasn't a religion. That's why you've got to crush all opposition by pointing out that "Atheism is a religion" is true by definition, because it isn't true any other way.

Which is to say: People insist that "X, by definition, is a Y!" on those occasions when they're trying to sneak in a connotation of Y that isn't directly in the definition, and X doesn't look all that much like other members of the Y cluster.

Over the last thirteen years I've been keeping track of how often this phrase is used correctly versus incorrectly - though not with literal statistics, I fear. But eyeballing suggests that using the phrase by definition, anywhere outside of math, is among the most alarming signals of flawed argument I've ever found. It's right up there with "Hitler", "God", "absolutely certain" and "can't prove that".

This heuristic of failure is not perfect - the first time I ever spotted a correct usage outside of math, it was by Richard Feynman; and since then I've spotted more. But you're probably better off just deleting the phrase "by definition" from your vocabulary - and always on any occasion where you might be tempted to say it in italics or followed with an exclamation mark. That's a bad idea by definition!

Sequence: A Human's Guide to Words

Sneaking in Connotations

Where to Draw the Boundary?

Referenced by: Where to Draw the Boundary? • 37 Ways That Words Can Be Wrong • Heading Toward Morality • The Design Space of Minds-In-General • Is Morality Given? • Complexity and Intelligence • Recognizing Intelligence • What Do We Mean By "Rationality"?

Original with comments: Arguing "By Definition"

Where to Draw the Boundary?

Eliezer Yudkowsky, 21 February 2008 07:14PM

Followup to: Arguing "By Definition"

The one comes to you and says:

Long have I pondered the meaning of the word "Art", and at last I've found what seems to me a satisfactory definition: "Art is that which is designed for the purpose of creating a reaction in an audience."

Just because there's a word "art" doesn't mean that it has a meaning, floating out there in the void, which you can discover by finding the right definition.

It feels that way, but it is not so.

Wondering how to define a word means you're looking at the problem the wrong way - searching for the mysterious essence of what is, in fact, a communication signal.

Now, there is a real challenge which a rationalist may legitimately attack, but the challenge is not to find a satisfactory definition of a word. The real challenge can be played as a single-player game, without speaking aloud. The challenge is figuring out which things are similar to each other - which things are clustered together - and sometimes, which things have a common cause.

If you define "eluctromugnetism" to include lightning, include compasses, exclude light, and include Mesmer's "animal magnetism" (what we now call hypnosis), then you will have some trouble asking "How does electromugnetism work?" You have lumped together things which do not belong together, and excluded others that would be needed to complete a set. (This example is historically plausible; Mesmer came before Faraday.)

We could say that electromugnetism is a wrong word, a boundary in thingspace that loops around and swerves through the clusters, a cut that fails to carve reality along its natural joints.

Figuring where to cut reality in order to carve along the joints - this is the problem worthy of a rationalist. It is what people should be trying to do, when they set out in search of the floating essence of a word.

And make no mistake: it is a scientific challenge to realize that you need a single word to describe breathing and fire. So do not think to consult the dictionary editors, for that is not their job.

What is "art"? But there is no essence of the word, floating in the void.

Perhaps you come to me with a long list of the things that you call "art" and "not art":

The Little Fugue in G Minor: Art.
A punch in the nose: Not art.
Escher's Relativity: Art.
A flower: Not art.
The Python programming language: Art.
A cross floating in urine: Not art.
Jack Vance's Tschai novels: Art.
Modern Art: Not art.

And you say to me: "It feels intuitive to me to draw this boundary, but I don't know why - can you find me an intension that matches this extension? Can you give me a simple description of this boundary?"

So I reply: "I think it has to do with admiration of craftsmanship: work going in and wonder coming out. What the included items have in common is the similar aesthetic emotions that they inspire, and the deliberate human effort that went into them with the intent of producing such an emotion."

Is this helpful, or is it just cheating at Taboo? I would argue that the list of which human emotions are or are not aesthetic is far more compact than the list of everything that is or isn't art. You might be able to see those emotions lighting up an fMRI scan - I say this by way of emphasizing that emotions are not ethereal.

But of course my definition of art is not the real point. The real point is that you could well dispute either the intension or the extension of my definition.

You could say, "Aesthetic emotion is not what these things have in common; what they have in common is an intent to inspire any complex emotion for the sake of inspiring it." That would be disputing my intension, my attempt to draw a curve through the data points. You would say, "Your equation may roughly fit those points, but it is not the true generating distribution."

Or you could dispute my extension by saying, "Some of these things do belong together - I can see what you're getting at - but the Python language shouldn't be on the list, and Modern Art should be." (This would mark you as a gullible philistine, but you could argue it.) Here, the presumption is that there is indeed an underlying curve that generates this apparent list of similar and dissimilar things - that there is a rhyme and reason, even though you haven't said yet where it comes from - but I have unwittingly lost the rhythm and included some data points from a different generator.

Long before you know what it is that electricity and magnetism have in common, you might still suspect - based on surface appearances - that "animal magnetism" does not belong on the list.

Once upon a time it was thought that the word "fish" included dolphins. Now you could play the oh-so-clever arguer, and say, "The list: {Salmon, guppies, sharks, dolphins, trout} is just a list - you can't say that a list is wrong. I can prove in set theory that this list exists. So my definition of fish, which is simply this extensional list, cannot possibly be 'wrong' as you claim."

Or you could stop playing nitwit games and admit that dolphins don't belong on the fish list.

You come up with a list of things that feel similar, and take a guess at why this is so. But when you finally discover what they really have in common, it may turn out that your guess was wrong. It may even turn out that your list was wrong.

You cannot hide behind a comforting shield of correct-by-definition. Both extensional definitions and intensional definitions can be wrong, can fail to carve reality at the joints.

Categorizing is a guessing endeavor, in which you can make mistakes; so it's wise to be able to admit, from a theoretical standpoint, that your definition-guesses can be "mistaken".

Sequence: A Human's Guide to Words

Arguing "By Definition"

Entropy, and Short Codes

Referenced by: Entropy, and Short Codes • Superexponential Conceptspace, and Simple Words • 37 Ways That Words Can Be Wrong • Reductive Reference • The Outside View's Domain • Recognizing Intelligence • Lawful Creativity • The Nature of Logic • Whence Your Abstractions? • Recursive Self-Improvement

Original with comments: Where to Draw the Boundary?

Entropy, and Short Codes

Eliezer Yudkowsky, 23 February 2008 03:16AM

Followup to: Where to Draw the Boundary?

Suppose you have a system X that's equally likely to be in any of 8 possible states:

{X_1, X_2, X_3, X_4, X_5, X_6, X_7, X_8.}

There's an extraordinarily ubiquitous quantity - in physics, mathematics, and even biology - called entropy; and the entropy of X is 3 bits. This means that, on average, we'll have to ask 3 yes-or-no questions to find out X's value. For example, someone could tell us X's value using this code:

X_1: 001

X_2: 010

X_3: 011

X_4: 100

X_5: 101

X_6: 110

X_7: 111

X_8: 000

So if I asked "Is the first symbol 1?" and heard "yes", then asked "Is the second symbol 1?" and heard "no", then asked "Is the third symbol 1?" and heard "no", I would know that X was in state 4.

Now suppose that the system Y has four possible states with the following probabilities:

Y_1: 1/2 (50%)

Y_2: 1/4 (25%)

Y_3: 1/8 (12.5%)

Y_4: 1/8 (12.5%)

Then the entropy of Y would be 1.75 bits, meaning that we can find out its value by asking 1.75 yes-or-no questions.

What does it mean to talk about asking one and three-fourths of a question? Imagine that we designate the states of Y using the following code:

Y_1: 1

Y_2: 01

Y_3: 001

Y_4: 000

First you ask, "Is the first symbol 1?" If the answer is "yes", you're done: Y is in state 1. This happens half the time, so 50% of the time, it takes 1 yes-or-no question to find out Y's state.

Suppose that instead the answer is "No". Then you ask, "Is the second symbol 1?" If the answer is "yes", you're done: Y is in state 2. Y is in state 2 with probability 1/4, and each time Y is in state 2 we discover this fact using two yes-or-no questions, so 25% of the time it takes 2 questions to discover Y's state.

If the answer is "No" twice in a row, you ask "Is the third symbol 1?" If "yes", you're done and Y is in state 3; if "no", you're done and Y is in state 4. The 1/8 of the time that Y is in state 3, it takes three questions; and the 1/8 of the time that Y is in state 4, it takes three questions.

(1/2 * 1) + (1/4 * 2) + (1/8 * 3) + (1/8 * 3)
= 0.5 + 0.5 + 0.375 + 0.375
= 1.75.

The general formula for the entropy of a system S is the sum, over all Si, of -p(Si)*log_2(p(Si)).

For example, the log (base 2) of 1/8 is -3. So -(1/8 * -3) = 0.375 is the contribution of state S_4 to the total entropy: 1/8 of the time, we have to ask 3 questions.

You can't always devise a perfect code for a system, but if you have to tell someone the state of arbitrarily many copies of S in a single message, you can get arbitrarily close to a perfect code. (Google "arithmetic coding" for a simple method.)

Now, you might ask: "Why not use the code 10 for Y_4, instead of 000? Wouldn't that let us transmit messages more quickly?"

But if you use the code 10 for Y_4, then when someone answers "Yes" to the question "Is the first symbol 1?", you won't know yet whether the system state is Y_1 (1) or Y_4 (10). In fact, if you change the code this way, the whole system falls apart - because if you hear "1001", you don't know if it means "Y_4, followed by Y_2" or "Y_1, followed by Y_3."

The moral is that short words are a conserved resource.

The key to creating a good code - a code that transmits messages as compactly as possible - is to reserve short words for things that you'll need to say frequently, and use longer words for things that you won't need to say as often.

When you take this art to its limit, the length of the message you need to describe something, corresponds exactly or almost exactly to its probability. This is the Minimum Description Length or Minimum Message Length formalization of Occam's Razor.

And so even the labels that we use for words are not quite arbitrary. The sounds that we attach to our concepts can be better or worse, wiser or more foolish. Even apart from considerations of common usage!

I say all this, because the idea that "You can X any way you like" is a huge obstacle to learning how to X wisely. "It's a free country; I have a right to my own opinion" obstructs the art of finding truth. "I can define a word any way I like" obstructs the art of carving reality at its joints. And even the sensible-sounding "The labels we attach to words are arbitrary" obstructs awareness of compactness. Prosody too, for that matter - Tolkien once observed what a beautiful sound the phrase "cellar door" makes; that is the kind of awareness it takes to use language like Tolkien.

The length of words also plays a nontrivial role in the cognitive science of language:

Consider the phrases "recliner", "chair", and "furniture". Recliner is a more specific category than chair; furniture is a more general category than chair. But the vast majority of chairs have a common use - you use the same sort of motor actions to sit down in them, and you sit down in them for the same sort of purpose (to take your weight off your feet while you eat, or read, or type, or rest). Recliners do not depart from this theme. "Furniture", on the other hand, includes things like beds and tables which have different uses, and call up different motor functions, from chairs.

In the terminology of cognitive psychology, "chair" is a basic-level category.

People have a tendency to talk, and presumably think, at the basic level of categorization - to draw the boundary around "chairs", rather than around the more specific category "recliner", or the more general category "furniture". People are more likely to say "You can sit in that chair" than "You can sit in that recliner" or "You can sit in that furniture".

And it is no coincidence that the word for "chair" contains fewer syllables than either "recliner" or "furniture". Basic-level categories, in general, tend to have short names; and nouns with short names tend to refer to basic-level categories. Not a perfect rule, of course, but a definite tendency. Frequent use goes along with short words; short words go along with frequent use.

Or as Douglas Hofstadter put it, there's a reason why the English language uses "the" to mean "the" and "antidisestablishmentarianism" to mean "antidisestablishmentarianism" instead of antidisestablishmentarianism other way around.

Sequence: A Human's Guide to Words

Where to Draw the Boundary?

Mutual Information, and Density in Thingspace

Referenced by: Mutual Information, and Density in Thingspace • Superexponential Conceptspace, and Simple Words • The Second Law of Thermodynamics, and Engines of Cognition • Conditional Independence, and Naive Bayes • 37 Ways That Words Can Be Wrong • Measuring Optimization Power

Original with comments: Entropy, and Short Codes

Mutual Information, and Density in Thingspace

Eliezer Yudkowsky, 23 February 2008 07:14PM

Continuation of: Entropy, and Short Codes

Suppose you have a system X that can be in any of 8 states, which are all equally probable (relative to your current state of knowledge), and a system Y that can be in any of 4 states, all equally probable.

The entropy of X, as defined yesterday, is 3 bits; we'll need to ask 3 yes-or-no questions to find out X's exact state. The entropy of Y, as defined yesterday, is 2 bits; we have to ask 2 yes-or-no questions to find out Y's exact state. This may seem obvious since 2^3 = 8 and 2^2 = 4, so 3 questions can distinguish 8 possibilities and 2 questions can distinguish 4 possibilities; but remember that if the possibilities were not all equally likely, we could use a more clever code to discover Y's state using e.g. 1.75 questions on average. In this case, though, X's probability mass is evenly distributed over all its possible states, and likewise Y, so we can't use any clever codes.

What is the entropy of the combined system (X,Y)?

You might be tempted to answer, "It takes 3 questions to find out X, and then 2 questions to find out Y, so it takes 5 questions total to find out the state of X and Y."

But what if the two variables are entangled, so that learning the state of Y tells us something about the state of X?

In particular, let's suppose that X and Y are either both odd, or both even.

Now if we receive a 3-bit message (ask 3 questions) and learn that X is in state 5, we know that Y is in state 1 or state 3, but not state 2 or state 4. So the single additional question "Is Y in state 3?", answered "No", tells us the entire state of (X,Y): X=X_5, Y=Y_1. And we learned this with a total of 4 questions.

Conversely, if we learn that Y is in state 4 using two questions, it will take us only an additional two questions to learn whether X is in state 2, 4, 6, or 8. Again, four questions to learn the state of the joint system.

The mutual information of two variables is defined as the difference between the entropy of the joint system and the entropy of the independent systems: I(X;Y) = H(X) + H(Y) - H(X,Y).

Here there is one bit of mutual information between the two systems: Learning X tells us one bit of information about Y (cuts down the space of possibilities from 4 to 2, a factor-of-2 decrease in the volume) and learning Y tells us one bit of information about X (cuts down the possibility space from 8 to 4).

What about when probability mass is not evenly distributed? Yesterday, for example, we discussed the case in which Y had the probabilities 1/2, 1/4, 1/8, 1/8 for its four states. Let us take this to be our probability distribution over Y, considered independently - if we saw Y, without seeing anything else, this is what we'd expect to see. And suppose the variable Z has two states, 1 and 2, with probabilities 3/8 and 5/8 respectively.

Then if and only if the joint distribution of Y and Z is as follows, there is zero mutual information between Y and Z:

Z_1Y_1: 3/16

Z_1Y_2: 3/32

Z_1Y_3: 3/64

Z_1Y_3: 3/64

Z_2Y_1: 5/16

Z_2Y_2: 5/32

Z_2Y_3: 5/64

Z_2Y_3: 5/64

This distribution obeys the law:

p(Y,Z) = P(Y)P(Z)

For example, P(Z_1Y_2) = P(Z_1)P(Y_2) = 3/8 * 1/4 = 3/32.

And observe that we can recover the marginal (independent) probabilities of Y and Z just by looking at the joint distribution:

P(Y_1) = total probability of all the different ways Y_1 can happen
= P(Z_1Y_1) + P(Z_2Y_1)
= 3/16 + 5/16
= 1/2.

So, just by inspecting the joint distribution, we can determine whether the marginal variables Y and Z are independent; that is, whether the joint distribution factors into the product of the marginal distributions; whether, for all Y and Z, P(Y,Z) = P(Y)P(Z).

This last is significant because, by Bayes's Rule:

P(Yi,Zj) = P(Yi)P(Zj)
P(Yi,Zj)/P(Zj) = P(Yi)
P(Yi|Zj) = P(Yi)

In English, "After you learn Zj, your belief about Yi is just what it was before."

So when the distribution factorizes - when P(Y,Z) = P(Y)P(Z) - this is equivalent to "Learning about Y never tells us anything about Z or vice versa."

From which you might suspect, correctly, that there is no mutual information between Y and Z. Where there is no mutual information, there is no Bayesian evidence, and vice versa.

Suppose that in the distribution YZ above, we treated each possible combination of Y and Z as a separate event - so that the distribution YZ would have a total of 8 possibilities, with the probabilities shown - and then we calculated the entropy of the distribution YZ the same way we would calculate the entropy of any distribution:

3/16 log_2(3/16) + 3/32 log_2(3/32) + 3/64 log_2(3/64) + ... + 5/64 log_2(5/64)

You would end up with the same total you would get if you separately calculated the entropy of Y plus the entropy of Z. There is no mutual information between the two variables, so our uncertainty about the joint system is not any less than our uncertainty about the two systems considered separately. (I am not showing the calculations, but you are welcome to do them; and I am not showing the proof that this is true in general, but you are welcome to Google on "Shannon entropy" and "mutual information".)

What if the joint distribution doesn't factorize? For example:

Z_1Y_1: 12/64

Z_1Y_2: 8/64

Z_1Y_3: 1/64

Z_1Y_4: 3/64

Z_2Y_1: 20/64

Z_2Y_2: 8/64

Z_2Y_3: 7/64

Z_2Y_4: 5/64

If you add up the joint probabilities to get marginal probabilities, you should find that P(Y_1) = 1/2, P(Z_1) = 3/8, and so on - the marginal probabilities are the same as before.

But the joint probabilities do not always equal the product of the marginal probabilities. For example, the probability P(Z_1Y_2) equals 8/64, where P(Z_1)P(Y_2) would equal 3/8 * 1/4 = 6/64. That is, the probability of running into Z_1Y_2 together, is greater than you'd expect based on the probabilities of running into Z_1 or Y_2 separately.

Which in turn implies:

P(Z_1Y_2) > P(Z_1)P(Y_2)
P(Z_1Y_2)/P(Y_2) > P(Z_1)
P(Z_1|Y_2) > P(Z_1)

Since there's an "unusually high" probability for P(Z_1Y_2) - defined as a probability higher than the marginal probabilities would indicate by default - it follows that observing Y_2 is evidence which increases the probability of Z_1. And by a symmetrical argument, observing Z_1 must favor Y_2.

As there are at least some values of Y that tell us about Z (and vice versa) there must be mutual information between the two variables; and so you will find - I am confident, though I haven't actually checked - that calculating the entropy of YZ yields less total uncertainty than the sum of the independent entropies of Y and Z. H(Y,Z) = H(Y) + H(Z) - I(Y;Z) with all quantities necessarily nonnegative.

(I digress here to remark that the symmetry of the expression for the mutual information shows that Y must tell us as much about Z, on average, as Z tells us about Y. I leave it as an exercise to the reader to reconcile this with anything they were taught in logic class about how, if all ravens are black, being allowed to reason Raven(x)->Black(x) doesn't mean you're allowed to reason Black(x)->Raven(x). How different seem the symmetrical probability flows of the Bayesian, from the sharp lurches of logic - even though the latter is just a degenerate case of the former.)

"But," you ask, "what has all this to do with the proper use of words?"

In Empty Labels and then Replace the Symbol with the Substance, we saw the technique of replacing a word with its definition - the example being given:

All [mortal, ~feathers, bipedal] are mortal.
Socrates is a [mortal, ~feathers, bipedal].
Therefore, Socrates is mortal.

Why, then, would you even want to have a word for "human"? Why not just say "Socrates is a mortal featherless biped"?

Because it's helpful to have shorter words for things that you encounter often. If your code for describing single properties is already efficient, then there will not be an advantage to having a special word for a conjunction - like "human" for "mortal featherless biped" - unless things that are mortal and featherless and bipedal, are found more often than the marginal probabilities would lead you to expect.

In efficient codes, word length corresponds to probability - so the code for Z_1Y_2 will be just as long as the code for Z_1 plus the code for Y_2, unless P(Z_1Y_2) > P(Z_1)P(Y_2), in which case the code for the word can be shorter than the codes for its parts.

And this in turn corresponds exactly to the case where we can infer some of the properties of the thing, from seeing its other properties. It must be more likely than the default that featherless bipedal things will also be mortal.

Of course the word "human" really describes many, many more properties - when you see a human-shaped entity that talks and wears clothes, you can infer whole hosts of biochemical and anatomical and cognitive facts about it. To replace the word "human" with a description of everything we know about humans would require us to spend an inordinate amount of time talking. But this is true only because a featherless talking biped is far more likely than default to be poisonable by hemlock, or have broad nails, or be overconfident.

Having a word for a thing, rather than just listing its properties, is a more compact code precisely in those cases where we can infer some of those properties from the other properties. (With the exception perhaps of very primitive words, like "red", that we would use to send an entirely uncompressed description of our sensory experiences. But by the time you encounter a bug, or even a rock, you're dealing with nonsimple property collections, far above the primitive level.)

So having a word "wiggin" for green-eyed black-haired people, is more useful than just saying "green-eyed black-haired person", precisely when:

1. Green-eyed people are more likely than average to be black-haired (and vice versa), meaning that we can probabilistically infer green eyes from black hair or vice versa; or

2. Wiggins share other properties that can be inferred at greater-than-default probability. In this case we have to separately observe the green eyes and black hair; but then, after observing both these properties independently, we can probabilistically infer other properties (like a taste for ketchup).

One may even consider the act of defining a word as a promise to this effect. Telling someone, "I define the word 'wiggin' to mean a person with green eyes and black hair", by Gricean implication, asserts that the word "wiggin" will somehow help you make inferences / shorten your messages.

If green-eyes and black hair have no greater than default probability to be found together, nor does any other property occur at greater than default probability along with them, then the word "wiggin" is a lie: The word claims that certain people are worth distinguishing as a group, but they're not.

In this case the word "wiggin" does not help describe reality more compactly - it is not defined by someone sending the shortest message - it has no role in the simplest explanation. Equivalently, the word "wiggin" will be of no help to you in doing any Bayesian inference. Even if you do not call the word a lie, it is surely an error.

And the way to carve reality at its joints, is to draw your boundaries around concentrations of unusually high probability density in Thingspace.

Sequence: A Human's Guide to Words

Entropy, and Short Codes

Superexponential Conceptspace, and Simple Words

Referenced by: Superexponential Conceptspace, and Simple Words • The Second Law of Thermodynamics, and Engines of Cognition • Searching for Bayes-Structure • Conditional Independence, and Naive Bayes • 37 Ways That Words Can Be Wrong • Is Humanism A Religion-Substitute? • Reductive Reference

Original with comments: Mutual Information, and Density in Thingspace

Superexponential Conceptspace, and Simple Words

Eliezer Yudkowsky, 24 February 2008 11:59PM

Followup to: Mutual Information, and Density in Thingspace

Thingspace, you might think, is a rather huge space. Much larger than reality, for where reality only contains things that actually exist, Thingspace contains everything that could exist.

Actually, the way I "defined" Thingspace to have dimensions for every possible attribute - including correlated attributes like density and volume and mass - Thingspace may be too poorly defined to have anything you could call a size. But it's important to be able to visualize Thingspace anyway. Surely, no one can really understand a flock of sparrows if all they see is a cloud of flapping cawing things, rather than a cluster of points in Thingspace.

But as vast as Thingspace may be, it doesn't hold a candle to the size of Conceptspace.

"Concept", in machine learning, means a rule that includes or excludes examples. If you see the data 2:+, 3:-, 14:+, 23:-, 8:+, 9:- then you might guess that the concept was "even numbers". There is a rather large literature (as one might expect) on how to learn concepts from data... given random examples, given chosen examples... given possible errors in classification... and most importantly, given different spaces of possible rules.

Suppose, for example, that we want to learn the concept "good days on which to play tennis". The possible attributes of Days are:

Sky:    {Sunny, Cloudy, Rainy}
AirTemp: {Warm, Cold}
Humidity: {Normal, High}
Wind:    {Strong, Weak}

We're then presented with the following data, where + indicates a positive example of the concept, and - indicates a negative classification:

+ Sky: Sunny; AirTemp: Warm; Humidity: High; Wind: Strong.
- Sky: Rainy; AirTemp: Cold; Humidity: High; Wind: Strong.
+ Sky: Sunny; AirTemp: Warm; Humidity: High; Wind: Weak.

What should an algorithm infer from this?

A machine learner might represent one concept that fits this data as follows:

Sky: ?; AirTemp: Warm; Humidity: High; Wind: ?

In this format, to determine whether this concept accepts or rejects an example, we compare element-by-element: ? accepts anything, but a specific value accepts only that specific value.

So the concept above will accept only Days with AirTemp=Warm and Humidity=High, but the Sky and the Wind can take on any value. This fits both the negative and the positive classifications in the data so far - though it isn't the only concept that does so.

We can also simplify the above concept representation to {?, Warm, High, ?}.

Without going into details, the classic algorithm would be:

- Maintain the set of the most general hypotheses that fit the data - those that positively classify as many examples as possible, while still fitting the facts.

- Maintain another set of the most specific hypotheses that fit the data - those that negatively classify as many examples as possible, while still fitting the facts.

- Each time we see a new negative example, we strengthen all the most general hypotheses as little as possible, so that the new set is again as general as possible while fitting the facts.

- Each time we see a new positive example, we relax all the most specific hypotheses as little as possible, so that the new set is again as specific as possible while fitting the facts.

- We continue until we have only a single hypothesis left. This will be the answer if the target concept was in our hypothesis space at all.

In the case above, the set of most general hypotheses would be {?, Warm, ?, ?} and {Sunny, ?, ?, ?}, while the set of most specific hypotheses is the single member {Sunny, Warm, High, ?}.

Any other concept you can find that fits the data will be strictly more specific than one of the most general hypotheses, and strictly more general than the most specific hypothesis.

(For more on this, I recommend Tom Mitchell's Machine Learning, from which this example was adapted.)

Now you may notice that the format above cannot represent all possible concepts. E.g. "Play tennis when the sky is sunny or the air is warm". That fits the data, but in the concept representation defined above, there's no quadruplet of values that describes the rule.

Clearly our machine learner is not very general. Why not allow it to represent all possible concepts, so that it can learn with the greatest possible flexibility?

Days are composed of these four variables, one variable with 3 values and three variables with 2 values. So there are 3*2*2*2 = 24 possible Days that we could encounter.

The format given for representing Concepts allows us to require any of these values for a variable, or leave the variable open. So there are 4*3*3*3 = 108 concepts in that representation. For the most-general/most-specific algorithm to work, we need to start with the most specific hypothesis "no example is ever positively classified". If we add that, it makes a total of 109 concepts.

Is it suspicious that there are more possible concepts than possible Days? Surely not: After all, a concept can be viewed as a collection of Days. A concept can be viewed as the set of days that it classifies positively, or isomorphically, the set of days that it classifies negatively.

So the space of all possible concepts that classify Days is the set of all possible sets of Days, whose size is 2^24 = 16,777,216.

This complete space includes all the concepts we have discussed so far. But it also includes concepts like "Positively classify only the examples {Sunny, Warm, High, Strong} and {Sunny, Warm, High, Weak} and reject everything else" or "Negatively classify only the example {Rainy, Cold, High, Strong} and accept everything else." It includes concepts with no compact representation, just a flat list of what is and isn't allowed.

That's the problem with trying to build a "fully general" inductive learner: They can't learn concepts until they've seen every possible example in the instance space.

If we add on more attributes to Days - like the Water temperature, or the Forecast for tomorrow - then the number of possible days will grow exponentially in the number of attributes. But this isn't a problem with our restricted concept space, because you can narrow down a large space using a logarithmic number of examples.

Let's say we add the Water: {Warm, Cold} attribute to days, which will make for 48 possible Days and 325 possible concepts. Let's say that each Day we see is, usually, classified positive by around half of the currently-plausible concepts, and classified negative by the other half. Then when we learn the actual classification of the example, it will cut the space of compatible concepts in half. So it might only take 9 examples (2^9 = 512) to narrow 325 possible concepts down to one.

Even if Days had forty binary attributes, it should still only take a manageable amount of data to narrow down the possible concepts to one. 64 examples, if each example is classified positive by half the remaining concepts. Assuming, of course, that the actual rule is one we can represent at all!

If you want to think of all the possibilities, well, good luck with that. The space of all possible concepts grows superexponentially in the number of attributes.

By the time you're talking about data with forty binary attributes, the number of possible examples is past a trillion - but the number of possible concepts is past two-to-the-trillionth-power. To narrow down that superexponential concept space, you'd have to see over a trillion examples before you could say what was In, and what was Out. You'd have to see every possible example, in fact.

That's with forty binary attributes, mind you. 40 bits, or 5 bytes, to be classified simply "Yes" or "No". 40 bits implies 2^40 possible examples, and 2^(2^40) possible concepts that classify those examples as positive or negative.

So, here in the real world, where objects take more than 5 bytes to describe and a trillion examples are not available and there is noise in the training data, we only even think about highly regular concepts. A human mind - or the whole observable universe - is not nearly large enough to consider all the other hypotheses.

From this perspective, learning doesn't just rely on inductive bias, it is nearly all inductive bias - when you compare the number of concepts ruled out a priori, to those ruled out by mere evidence.

But what has this (you inquire) to do with the proper use of words?

It's the whole reason that words have intensions as well as extensions.

In yesterday's post, I concluded:

The way to carve reality at its joints, is to draw boundaries around concentrations of unusually high probability density.

I deliberately left out a key qualification in that (slightly edited) statement, because I couldn't explain it until today. A better statement would be:

The way to carve reality at its joints, is to draw simple boundaries around concentrations of unusually high probability density in Thingspace.

Otherwise you would just gerrymander Thingspace. You would create really odd noncontiguous boundaries that collected the observed examples, examples that couldn't be described in any shorter message than your observations themselves, and say: "This is what I've seen before, and what I expect to see more of in the future."

In the real world, nothing above the level of molecules repeats itself exactly. Socrates is shaped a lot like all those other humans who were vulnerable to hemlock, but he isn't shaped exactly like them. So your guess that Socrates is a "human" relies on drawing simple boundaries around the human cluster in Thingspace. Rather than, "Things shaped exactly like [5-megabyte shape specification 1] and with [lots of other characteristics], or exactly like [5-megabyte shape specification 2] and [lots of other characteristics]", ..., are human."

If you don't draw simple boundaries around your experiences, you can't do inference with them. So you try to describe "art" with intensional definitions like "that which is intended to inspire any complex emotion for the sake of inspiring it", rather than just pointing at a long list of things that are, or aren't art.

In fact, the above statement about "how to carve reality at its joints" is a bit chicken-and-eggish: You can't assess the density of actual observations, until you've already done at least a little carving. And the probability distribution comes from drawing the boundaries, not the other way around - if you already had the probability distribution, you'd have everything necessary for inference, so why would you bother drawing boundaries?

And this suggests another - yes, yet another - reason to be suspicious of the claim that "you can define a word any way you like". When you consider the superexponential size of Conceptspace, it becomes clear that singling out one particular concept for consideration is an act of no small audacity - not just for us, but for any mind of bounded computing power.

Presenting us with the word "wiggin", defined as "a black-haired green-eyed person", without some reason for raising this particular concept to the level of our deliberate attention, is rather like a detective saying: "Well, I haven't the slightest shred of support one way or the other for who could've murdered those orphans... not even an intuition, mind you... but have we considered John Q. Wiffleheim of 1234 Norkle Rd as a suspect?"

Sequence: A Human's Guide to Words

Mutual Information, and Density in Thingspace

Conditional Independence, and Naive Bayes

Referenced by: The Second Law of Thermodynamics, and Engines of Cognition • 37 Ways That Words Can Be Wrong • Is Humanism A Religion-Substitute? • Reductive Reference • Unnatural Categories • Magical Categories • Qualitative Strategies of Friendliness

Original with comments: Superexponential Conceptspace, and Simple Words

Leave a Line of Retreat

Eliezer Yudkowsky, 25 February 2008 11:57PM

"When you surround the enemy
Always allow them an escape route.
They must see that there is
An alternative to death."
-- Sun Tzu, The Art of War, Cloud Hands edition

"Don't raise the pressure, lower the wall."
-- Lois McMaster Bujold, Komarr

Last night I happened to be conversing with a nonrationalist who had somehow wandered into a local rationalists' gathering. She had just declared (a) her belief in souls and (b) that she didn't believe in cryonics because she believed the soul wouldn't stay with the frozen body. I asked, "But how do you know that?" From the confusion that flashed on her face, it was pretty clear that this question had never occurred to her. I don't say this in a bad way - she seemed like a nice person with absolutely no training in rationality, just like most of the rest of the human species. I really need to write that book.

Most of the ensuing conversation was on items already covered on Overcoming Bias - if you're really curious about something, you probably can figure out a good way to test it; try to attain accurate beliefs first and then let your emotions flow from that - that sort of thing. But the conversation reminded me of one notion I haven't covered here yet:

"Make sure," I suggested to her, "that you visualize what the world would be like if there are no souls, and what you would do about that. Don't think about all the reasons that it can't be that way, just accept it as a premise and then visualize the consequences. So that you'll think, 'Well, if there are no souls, I can just sign up for cryonics', or 'If there is no God, I can just go on being moral anyway,' rather than it being too horrifying to face. As a matter of self-respect you should try to believe the truth no matter how uncomfortable it is, like I said before; but as a matter of human nature, it helps to make a belief less uncomfortable, before you try to evaluate the evidence for it."

The principle behind the technique is simple: As Sun Tzu advises you to do with your enemies, you must do with yourself - leave yourself a line of retreat, so that you will have less trouble retreating. The prospect of losing your job, say, may seem a lot more scary when you can't even bear to think about it, than after you have calculated exactly how long your savings will last, and checked the job market in your area, and otherwise planned out exactly what to do next. Only then will you be ready to fairly assess the probability of keeping your job in the planned layoffs next month. Be a true coward, and plan out your retreat in detail - visualize every step - preferably before you first come to the battlefield.

The hope is that it takes less courage to visualize an uncomfortable state of affairs as a thought experiment, than to consider how likely it is to be true. But then after you do the former, it becomes easier to do the latter.

Remember that Bayesianism is precise - even if a scary proposition really should seem unlikely, it's still important to count up all the evidence, for and against, exactly fairly, to arrive at the rational quantitative probability. Visualizing a scary belief does not mean admitting that you think, deep down, it's probably true. You can visualize a scary belief on general principles of good mental housekeeping. "The thought you cannot think controls you more than thoughts you speak aloud" - this happens even if the unthinkable thought is false!

The leave-a-line-of-retreat technique does require a certain minimum of self-honesty to use correctly.

For a start: You must at least be able to admit to yourself which ideas scare you, and which ideas you are attached to. But this is a substantially less difficult test than fairly counting the evidence for an idea that scares you. Does it help if I say that I have occasion to use this technique myself? A rationalist does not reject all emotion, after all. There are ideas which scare me, yet I still believe to be false. There are ideas to which I know I am attached, yet I still believe to be true. But I still plan my retreats, not because I'm planning to retreat, but because planning my retreat in advance helps me think about the problem without attachment.

But greater test of self-honesty is to really accept the uncomfortable proposition as a premise, and figure out how you would really deal with it. When we're faced with an uncomfortable idea, our first impulse is naturally to think of all the reasons why it can't possibly be so. And so you will encounter a certain amount of psychological resistance in yourself, if you try to visualize exactly how the world would be, and what you would do about it, if My-Most-Precious-Belief were false, or My-Most-Feared-Belief were true.

Think of all the people who say that, without God, morality was impossible. (And yes, this topic did come up in the conversation; so I am not offering a strawman.) If theists could visualize their real reaction to believing as a fact that God did not exist, they could realize that, no, they wouldn't go around slaughtering babies. They could realize that atheists are reacting to the nonexistence of God in pretty much the way they themselves would, if they came to believe that. I say this, to show that it is a considerable challenge to visualize the way you really would react, to believing the opposite of a tightly held belief.

Plus it's always counterintuitive to realize that, yes, people do get over things. Newly minted quadriplegics are not as sad as they expect to be six months later, etc. It can be equally counterintuitive to realize that if the scary belief turned out to be true, you would come to terms with it somehow. Quadriplegics deal, and so would you.

See also the Litany of Gendlin and the Litany of Tarski. What is true is already so; owning up to it doesn't make it worse. You shouldn't be afraid to just visualize a world you fear. If that world is already actual, visualizing it won't make it worse; and if it is not actual, visualizing it will do no harm. And remember, as you visualize, that if the scary things you're imagining really are true - which they may not be! - then you would, indeed, want to believe it, and you should visualize that too; not believing wouldn't help you.

How many religious people would retain their belief in God, if they could accurately visualize that hypothetical world in which there was no God and they themselves have become atheists?

Leaving a line of retreat is a powerful technique, but it's not easy. Honest visualization doesn't take as much effort as admitting outright that God doesn't exist, but it does take an effort.

(Meta note: I'm posting this on the advice that I should break up long sequences of mathy posts with non-mathy posts. (I was actually advised to post something "fun", but I'd rather not - it feels like I have too much important material to cover in the next couple of months.) If anyone thinks that I should have, instead, gone ahead and posted the next item in the information-theory sequence rather than breaking it up; or, alternatively, thinks that this non-mathy post came as a welcome change; then I am interested in hearing from you in the comments.)

Sequence: Letting Go

No One Can Exempt You From Rationality's Laws

Crisis of Faith

Referenced by: Bind Yourself to Reality • Zombies! Zombies? • Changing Your Metaethics • Fighting a Rearguard Action Against the Truth • Singletons Rule OK

Original with comments: Leave a Line of Retreat

The Second Law of Thermodynamics, and Engines of Cognition

Eliezer Yudkowsky, 27 February 2008 12:48AM

Followup to: Superexponential Conceptspace, and Simple Words

The first law of thermodynamics, better known as Conservation of Energy, says that you can't create energy from nothing: it prohibits perpetual motion machines of the first type, which run and run indefinitely without consuming fuel or any other resource. According to our modern view of physics, energy is conserved in each individual interaction of particles. By mathematical induction, we see that no matter how large an assemblage of particles may be, it cannot produce energy from nothing - not without violating what we presently believe to be the laws of physics.

This is why the US Patent Office will summarily reject your amazingly clever proposal for an assemblage of wheels and gears that cause one spring to wind up another as the first runs down, and so continue to do work forever, according to your calculations. There's a fully general proof that at least one wheel must violate (our standard model of) the laws of physics for this to happen. So unless you can explain how one wheel violates the laws of physics, the assembly of wheels can't do it either.

A similar argument applies to a "reactionless drive", a propulsion system that violates Conservation of Momentum. In standard physics, momentum is conserved for all individual particles and their interactions; by mathematical induction, momentum is conserved for physical systems whatever their size. If you can visualize two particles knocking into each other and always coming out with the same total momentum that they started with, then you can see how scaling it up from particles to a gigantic complicated collection of gears won't change anything. Even if there's a trillion quadrillion atoms involved, 0 + 0 + ... + 0 = 0.

But Conservation of Energy, as such, cannot prohibit converting heat into work. You can, in fact, build a sealed box that converts ice cubes and stored electricity into warm water. It isn't even difficult. Energy cannot be created or destroyed: The net change in energy, from transforming (ice cubes + electricity) to (warm water), must be 0. So it couldn't violate Conservation of Energy, as such, if you did it the other way around...

Perpetual motion machines of the second type, which convert warm water into electrical current and ice cubes, are prohibited by the Second Law of Thermodynamics.

The Second Law is a bit harder to understand, as it is essentially Bayesian in nature.

Yes, really.

The essential physical law underlying the Second Law of Thermodynamics is a theorem which can be proven within the standard model of physics: In the development over time of any closed system, phase space volume is conserved.

Let's say you're holding a ball high above the ground. We can describe this state of affairs as a point in a multidimensional space, at least one of whose dimensions is "height of ball above the ground". Then, when you drop the ball, it moves, and so does the dimensionless point in phase space that describes the entire system that includes you and the ball. "Phase space", in physics-speak, means that there are dimensions for the momentum of the particles, not just their position - i.e., a system of 2 particles would have 12 dimensions, 3 dimensions for each particle's position, and 3 dimensions for each particle's momentum.

If you had a multidimensional space, each of whose dimensions described the position of a gear in a huge assemblage of gears, then as you turned the gears a single point would swoop and dart around in a rather high-dimensional phase space. Which is to say, just as you can view a great big complex machine as a single point in a very-high-dimensional space, so too, you can view the laws of physics describing the behavior of this machine over time, as describing the trajectory of its point through the phase space.

The Second Law of Thermodynamics is a consequence of a theorem which can be proven in the standard model of physics: If you take a volume of phase space, and develop it forward in time using standard physics, the total volume of the phase space is conserved.

For example:

Let there be two systems, X and Y: where X has 8 possible states, Y has 4 possible states, and the joint system (X,Y) has 32 possible states.

The development of the joint system over time can be described as a rule that maps initial points onto future points. For example, the system could start out in X_7Y_2, then develop (under some set of physical laws) into the state X_3Y_3 a minute later. Which is to say: if X started in 7, and Y started in 2, and we watched it for 1 minute, we would see X go to 3 and Y go to 3. Such are the laws of physics.

Next, let's carve out a subspace S of the joint system state. S will be the subspace bounded by X being in state 1 and Y being in states 1-4. So the total volume of S is 4 states.

And let's suppose that, under the laws of physics governing (X,Y) the states initially in S behave as follows:

X_1Y_1 -> X_2Y_1
X_1Y_2 -> X_4Y_1
X_1Y_3 -> X_6Y_1
X_1Y_4 -> X_8Y_1

That, in a nutshell, is how a refrigerator works.

The X subsystem began in a narrow region of state space - the single state 1, in fact - and Y began distributed over a wider region of space, states 1-4. By interacting with each other, Y went into a narrow region, and X ended up in a wide region; but the total phase space volume was conserved. 4 initial states mapped to 4 end states.

Clearly, so long as total phase space volume is conserved by physics over time, you can't squeeze Y harder than X expands, or vice versa - for every subsystem you squeeze into a narrower region of state space, some other subsystem has to expand into a wider region of state space.

Now let's say that we're uncertain about the joint system (X,Y), and our uncertainty is described by an equiprobable distribution over S. That is, we're pretty sure X is in state 1, but Y is equally likely to be in any of states 1-4. If we shut our eyes for a minute and then open them again, we will expect to see Y in state 1, but X might be in any of states 2-8. Actually, X can only be in some of states 2-8, but it would be too costly to think out exactly which states these might be, so we'll just say 2-8.

If you consider the Shannon entropy of our uncertainty about X and Y as individual systems, X began with 0 bits of entropy because it had a single definite state, and Y began with 2 bits of entropy because it was equally likely to be in any of 4 possible states. (There's no mutual information between X and Y.) A bit of physics occurred, and lo, the entropy of Y went to 0, but the entropy of X went to log_2(7) = 2.8 bits. So entropy was transferred from one system to another, and decreased within the Y subsystem; but due to the cost of bookkeeping, we didn't bother to track some information, and hence (from our perspective) the overall entropy increased.

If there was a physical process that mapped past states onto future states like this:

X2,Y1 -> X2,Y1
X2,Y2 -> X2,Y1
X2,Y3 -> X2,Y1
X2,Y4 -> X2,Y1

Then you could have a physical process that would actually decrease entropy, because no matter where you started out, you would end up at the same place. The laws of physics, developing over time, would compress the phase space.

But there is a theorem, Liouville's Theorem, which can be proven true of our laws of physics, which says that this never happens: phase space is conserved.

The Second Law of Thermodynamics is a corollary of Liouville's Theorem: no matter how clever your configuration of wheels and gears, you'll never be able to decrease entropy in one subsystem without increasing it somewhere else. When the phase space of one subsystem narrows, the phase space of another subsystem must widen, and the joint space keeps the same volume.

Except that what was initially a compact phase space, may develop squiggles and wiggles and convolutions; so that to draw a simple boundary around the whole mess, you must draw a much larger boundary than before - this is what gives the appearance of entropy increasing. (And in quantum systems, where different universes go different ways, entropy actually does increase in any local universe. But omit this complication for now.)

The Second Law of Thermodynamics is actually probabilistic in nature - if you ask about the probability of hot water spontaneously entering the "cold water and electricity" state, the probability does exist, it's just very small. This doesn't mean Liouville's Theorem is violated with small probability; a theorem's a theorem, after all. It means that if you're in a great big phase space volume at the start, but you don't know where, you may assess a tiny little probability of ending up in some particular phase space volume. So far as you know, with infinitesimal probability, this particular glass of hot water may be the kind that spontaneously transforms itself to electrical current and ice cubes. (Neglecting, as usual, quantum effects.)

So the Second Law really is inherently Bayesian. When it comes to any real thermodynamic system, it's a strictly lawful statement of your beliefs about the system, but only a probabilistic statement about the system itself.

"Hold on," you say. "That's not what I learned in physics class," you say. "In the lectures I heard, thermodynamics is about, you know, temperatures. Uncertainty is a subjective state of mind! The temperature of a glass of water is an objective property of the water! What does heat have to do with probability?"

Oh ye of little trust.

In one direction, the connection between heat and probability is relatively straightforward: If the only fact you know about a glass of water is its temperature, then you are much more uncertain about a hot glass of water than a cold glass of water.

Heat is the zipping around of lots of tiny molecules; the hotter they are, the faster they can go. Not all the molecules in hot water are travelling at the same speed - the "temperature" isn't a uniform speed of all the molecules, it's an average speed of the molecules, which in turn corresponds to a predictable statistical distribution of speeds - anyway, the point is that, the hotter the water, the faster the water molecules could be going, and hence, the more uncertain you are about the velocity (not just speed) of any individual molecule. When you multiply together your uncertainties about all the individual molecules, you will be exponentially more uncertain about the whole glass of water.

We take the logarithm of this exponential volume of uncertainty, and call that the entropy. So it all works out, you see.

The connection in the other direction is less obvious. Suppose there was a glass of water, about which, initially, you knew only that its temperature was 72 degrees. Then, suddenly, Saint Laplace reveals to you the exact locations and velocities of all the atoms in the water. You now know perfectly the state of the water, so, by the information-theoretic definition of entropy, its entropy is zero. Does that make its thermodynamic entropy zero? Is the water colder, because we know more about it?

Ignoring quantumness for the moment, the answer is: Yes! Yes it is!

Maxwell once asked: Why can't we take a uniformly hot gas, and partition it into two volumes A and B, and let only fast-moving molecules pass from B to A, while only slow-moving molecules are allowed to pass from A to B? If you could build a gate like this, soon you would have hot gas on the A side, and cold gas on the B side. That would be a cheap way to refrigerate food, right?

The agent who inspects each gas molecule, and decides whether to let it through, is known as "Maxwell's Demon". And the reason you can't build an efficient refrigerator this way, is that Maxwell's Demon generates entropy in the process of inspecting the gas molecules and deciding which ones to let through.

But suppose you already knew where all the gas molecules were?

Then you actually could run Maxwell's Demon and extract useful work.

So (again ignoring quantum effects for the moment), if you know the states of all the molecules in a glass of hot water, it is cold in a genuinely thermodynamic sense: you can take electricity out of it and leave behind an ice cube.

This doesn't violate Liouville's Theorem, because if Y is the water, and you are Maxwell's Demon (denoted M), the physical process behaves as:

M1,Y1 -> M1,Y1
M2,Y2 -> M2,Y1
M3,Y3 -> M3,Y1
M4,Y4 -> M4,Y1

Because Maxwell's demon knows the exact state of Y, this is mutual information between M and Y. The mutual information decreases the joint entropy of (M,Y): H(M,Y) = H(M) + H(Y) - I(M;Y). M has 2 bits of entropy, Y has two bits of entropy, and their mutual information is 2 bits, so (M,Y) has a total of 2 + 2 - 2 = 2 bits of entropy. The physical process just transforms the "coldness" (negentropy) of the mutual information to make the actual water cold - afterward, M has 2 bits of entropy, Y has 0 bits of entropy, and the mutual information is 0. Nothing wrong with that!

And don't tell me that knowledge is "subjective". Knowledge has to be represented in a brain, and that makes it as physical as anything else. For M to physically represent an accurate picture of the state of Y, M's physical state must correlate with the state of Y. You can take thermodynamic advantage of that - it's called a Szilard engine.

Or as E.T. Jaynes put it, "The old adage 'knowledge is power' is a very cogent truth, both in human relations and in thermodynamics."

And conversely, one subsystem cannot increase in mutual information with another subsystem, without (a) interacting with it and (b) doing thermodynamic work.

Otherwise you could build a Maxwell's Demon and violate the Second Law of Thermodynamics - which in turn would violate Liouville's Theorem - which is prohibited in the standard model of physics.

Which is to say: To form accurate beliefs about something, you really do have to observe it. It's a very physical, very real process: any rational mind does "work" in the thermodynamic sense, not just the sense of mental effort.

(It is sometimes said that it is erasing bits in order to prepare for the next observation that takes the thermodynamic work - but that distinction is just a matter of words and perspective; the math is unambiguous.)

(Discovering logical "truths" is a complication which I will not, for now, consider - at least in part because I am still thinking through the exact formalism myself. In thermodynamics, knowledge of logical truths does not count as negentropy; as would be expected, since a reversible computer can compute logical truths at arbitrarily low cost. All this that I have said is true of the logically omniscient: any lesser mind will necessarily be less efficient.)

"Forming accurate beliefs requires a corresponding amount of evidence" is a very cogent truth both in human relations and in thermodynamics: if blind faith actually worked as a method of investigation, you could turn warm water into electricity and ice cubes. Just build a Maxwell's Demon that has blind faith in molecule velocities.

Engines of cognition are not so different from heat engines, though they manipulate entropy in a more subtle form than burning gasoline. For example, to the extent that an engine of cognition is not perfectly efficient, it must radiate waste heat, just like a car engine or refrigerator.

"Cold rationality" is true in a sense that Hollywood scriptwriters never dreamed (and false in the sense that they did dream).

So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don't expect me to believe that a big, elaborate clever argument can do it either.

Referenced by: Perpetual Motion Beliefs • Searching for Bayes-Structure • Heat vs. Motion • GAZP vs. GLUT • Distinct Configurations • Classical Configuration Spaces • Decoherence • Which Basis Is More Fundamental? • On Being Decoherent • Decoherence is Pointless • Collapse Postulates • Science Isn't Strict Enough • Do Scientists Already Know This Stuff? • Faster Than Science • A Premature Word on AI • The Rhythm of Disagreement • Where Recursive Justification Hits Bottom • My Naturalistic Awakening • On Doing the Impossible • Dark Side Epistemology • Belief in Intelligence • Measuring Optimization Power • Recognizing Intelligence • Lawful Creativity • The Weighted Majority Algorithm • Raising the Sanity Waterline • Practical Advice Backed By Deep Theories

Original with comments: The Second Law of Thermodynamics, and Engines of Cognition

Perpetual Motion Beliefs

Eliezer Yudkowsky, 27 February 2008 08:22PM

Followup to: The Second Law of Thermodynamics, and Engines of Cognition

Yesterday's post concluded:

To form accurate beliefs about something, you really do have to observe it. It's a very physical, very real process: any rational mind does "work" in the thermodynamic sense, not just the sense of mental effort... So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don't expect me to believe that a big, elaborate clever argument can do it either.

One of the chief morals of the mathematical analogy between thermodynamics and cognition is that the constraints of probability are inescapable; probability may be a "subjective state of belief", but the laws of probability are harder than steel.

People learn under the traditional school regimen that the teacher tells you certain things, and you must believe them and recite them back; but if a mere student suggests a belief, you do not have to obey it. They map the domain of belief onto the domain of authority, and think that a certain belief is like an order that must be obeyed, but a probabilistic belief is like a mere suggestion.

They look at a lottery ticket, and say, "But you can't prove I won't win, right?" Meaning: "You may have calculated a low probability of winning, but since it is a probability, it's just a suggestion, and I am allowed to believe what I want."

Here's a little experiment: Smash an egg on the floor. The rule that says that the egg won't spontaneously reform and leap back into your hand is merely probabilistic. A suggestion, if you will. The laws of thermodynamics are probabilistic, so they can't really be laws, the way that "Thou shalt not murder" is a law... right?

So why not just ignore the suggestion? Then the egg will unscramble itself... right?

It may help to think of it this way - if you still have some lingering intuition that uncertain beliefs are not authoritative:

In reality, there may be a very small chance that the egg spontaneously reforms. But you cannot expect it to reform. You must expect it to smash. Your mandatory belief is that the egg's probability of spontaneous reformation is ~0. Probabilities are not certainties, but the laws of probability are theorems.

If you doubt this, try dropping an egg on the floor a few decillion times, ignoring the thermodynamic suggestion and expecting it to spontaneously reassemble, and see what happens. Probabilities may be subjective states of belief, but the laws governing them are stronger by far than steel.

I once knew a fellow who was convinced that his system of wheels and gears would produce reactionless thrust, and he had an Excel spreadsheet that would prove this - which of course he couldn't show us because he was still developing the system. In classical mechanics, violating Conservation of Momentum is provably impossible. So any Excel spreadsheet calculated according to the rules of classical mechanics must necessarily show that no reactionless thrust exists - unless your machine is complicated enough that you have made a mistake in the calculations.

And similarly, when half-trained or tenth-trained rationalists abandon their art and try to believe without evidence just this once, they often build vast edifices of justification, confusing themselves just enough to conceal the magical steps.

It can be quite a pain to nail down where the magic occurs - their structure of argument tends to morph and squirm away as you interrogate them. But there's always some step where a tiny probability turns into a large one - where they try to believe without evidence - where they step into the unknown, thinking, "No one can prove me wrong".

Their foot naturally lands on thin air, for there is far more thin air than ground in the realms of Possibility. Ah, but there is an (exponentially tiny) amount of ground in Possibility, and you do have an (exponentially tiny) probability of hitting it by luck, so maybe this time, your foot will land in the right place! It is merely a probability, so it must be merely a suggestion.

The exact state of a glass of boiling-hot water may be unknown to you - indeed, your ignorance of its exact state is what makes the molecules' kinetic energy "heat", rather than work waiting to be extracted like the momentum of a spinning flywheel. So the water might cool down your hand instead of heating it up, with probability ~0.

Decide to ignore the laws of thermodynamics and stick your hand in anyway, and you'll get burned.

"But you don't know that!"

I don't know it with certainty, but it is mandatory that I expect it to happen. Probabilities are not logical truths, but the laws of probability are.

"But what if I guess the state of the boiling water, and I happen to guess correctly?"

Your chance of guessing correctly by luck, is even less than the chance of the boiling water cooling your hand by luck.

"But you can't prove I won't guess correctly."

I can (indeed, must) assign extremely low probability to it.

"That's not the same as certainty, though."

Hey, maybe if you add enough wheels and gears to your argument, it'll turn warm water into electricity and ice cubes! Or, rather, you will no longer see why this couldn't be the case.

"Right! I can't see why couldn't be the case! So maybe it is!"

Another gear? That just makes your machine even less efficient. It wasn't a perpetual motion machine before, and each extra gear you add makes it even less efficient than that.

Each extra detail in your argument necessarily decreases the joint probability. The probability that you've violated the Second Law of Thermodynamics without knowing exactly how, by guessing the exact state of boiling water without evidence, so that you can stick your finger in without getting burned, is, necessarily, even less than the probability of sticking in your finger into boiling water without getting burned.

I say all this, because people really do construct these huge edifices of argument in the course of believing without evidence. One must learn to see this as analogous to all the wheels and gears that fellow added onto his reactionless drive, until he finally collected enough complications to make a mistake in his Excel spreadsheet.

Referenced by: Searching for Bayes-Structure • Righting a Wrong Question • Zombies! Zombies? • Zombie Responses • The Generalized Anti-Zombie Principle • GAZP vs. GLUT • The So-Called Heisenberg Uncertainty Principle • If Many-Worlds Had Come First • Many Worlds, One Best Guess • Possibility and Could-ness • Where Recursive Justification Hits Bottom • Contaminated by Optimism • Mirrors and Paintings • Dark Side Epistemology • The Weighted Majority Algorithm • Privileging the Hypothesis

Original with comments: Perpetual Motion Beliefs

Searching for Bayes-Structure

Eliezer Yudkowsky, 28 February 2008 10:01PM

Followup to: Perpetual Motion Beliefs

"Gnomish helms should not function. Their very construction seems to defy the nature of thaumaturgical law. In fact, they are impossible. Like most products of gnomish minds, they include a large number of bells and whistles, and very little substance. Those that work usually have a minor helm contained within, always hidden away, disguised to appear innocuous and inessential."
-- Spelljammer campaign set

We have seen that knowledge implies mutual information between a mind and its environment, and we have seen that this mutual information is negentropy in a very physical sense: If you know where molecules are and how fast they're moving, you can turn heat into work via a Maxwell's Demon / Szilard engine.

We have seen that forming true beliefs without evidence is the same sort of improbability as a hot glass of water spontaneously reorganizing into ice cubes and electricity. Rationality takes "work" in a thermodynamic sense, not just the sense of mental effort; minds have to radiate heat if they are not perfectly efficient. This cognitive work is governed by probability theory, of which thermodynamics is a special case. (Statistical mechanics is a special case of statistics.)

If you saw a machine continually spinning a wheel, apparently without being plugged into a wall outlet or any other source of power, then you would look for a hidden battery, or a nearby broadcast power source - something to explain the work being done, without violating the laws of physics.

So if a mind is arriving at true beliefs, and we assume that the second law of thermodynamics has not been violated, that mind must be doing something at least vaguely Bayesian - at least one process with a sort-of Bayesian structure somewhere - or it couldn't possibly work.

In the beginning, at time T=0, a mind has no mutual information with a subsystem S in its environment. At time T=1,the mind has 10 bits of mutual information with S. Somewhere in between, the mind must have encountered evidence - under the Bayesian definition of evidence, because all Bayesian evidence is mutual information and all mutual information is Bayesian evidence, they are just different ways of looking at it - and processed at least some of that evidence, however inefficiently, in the right direction according to Bayes on at least some occasions. The mind must have moved in harmony with the Bayes at least a little, somewhere along the line - either that or violated the second law of thermodynamics by creating mutual information from nothingness.

In fact, any part of a cognitive process that contributes usefully to truth-finding must have at least a little Bayesian structure - must harmonize with Bayes, at some point or another - must partially conform with the Bayesian flow, however noisily - despite however many disguising bells and whistles - even if this Bayesian structure is only apparent in the context of surrounding processes. Or it couldn't even help.

How philosophers pondered the nature of words! All the ink spent on the true definitions of words, and the true meaning of definitions, and the true meaning of meaning! What collections of gears and wheels they built, in their explanations! And all along, it was a disguised form of Bayesian inference!

I was actually a bit disappointed that no one in the audience jumped up and said: "Yes! Yes, that's it! Of course! It was really Bayes all along!"

But perhaps it is not quite as exciting to see something that doesn't look Bayesian on the surface, revealed as Bayes wearing a clever disguise, if: (a) you don't unravel the mystery yourself, but read about someone else doing it (Newton had more fun than most students taking calculus), and (b) you don't realize that searching for the hidden Bayes-structure is this huge, difficult, omnipresent quest, like searching for the Holy Grail.

It's a different quest for each facet of cognition, but the Grail always turns out to be the same. It has to be the right Grail, though - and the entire Grail, without any parts missing - and so each time you have to go on the quest looking for a full answer whatever form it may take, rather than trying to artificially construct vaguely hand-waving Grailish arguments. Then you always find the same Holy Grail at the end.

It was previously pointed out to me that I might be losing some of my readers with the long essays, because I hadn't "made it clear where I was going"...

...but it's not so easy to just tell people where you're going, when you're going somewhere like that.

It's not very helpful to merely know that a form of cognition is Bayesian, if you don't know how it is Bayesian. If you can't see the detailed flow of probability, you have nothing but a password - or, a bit more charitably, a hint at the form an answer would take; but certainly not an answer. That's why there's a Grand Quest for the Hidden Bayes-Structure, rather than being done when you say "Bayes!" Bayes-structure can be buried under all kinds of disguies, hidden behind thickets of wheels and gears, obscured by bells and whistles.

The way you begin to grasp the Quest for the Holy Bayes is that you learn about cognitive phenomenon XYZ, which seems really useful - and there's this bunch of philosophers who've been arguing about its true nature for centuries, and they are still arguing - and there's a bunch of AI scientists trying to make a computer do it, but they can't agree on the philosophy either -

And - Huh, that's odd! - this cognitive phenomenon didn't look anything like Bayesian on the surface, but there's this non-obvious underlying structure that has a Bayesian interpretation - but wait, there's still some useful work getting done that can't be explained in Bayesian terms - no wait, that's Bayesian too - OH MY GOD this completely different cognitive process, that also didn't look Bayesian on the surface, ALSO HAS BAYESIAN STRUCTURE - hold on, are these non-Bayesian parts even doing anything?

- Yes: Wow, those are Bayesian too!

- No: Dear heavens, what a stupid design. I could eat a bucket of amino acids and puke a better brain architecture than that.

Once this happens to you a few times, you kinda pick up the rhythm. That's what I'm talking about here, the rhythm.

Trying to talk about the rhythm is like trying to dance about architecture.

This left me in a bit of a pickle when it came to trying to explain in advance where I was going. I know from experience that if I say, "Bayes is the secret of the universe," some people may say "Yes! Bayes is the secret of the universe!"; and others will snort and say, "How narrow-minded you are; look at all these other ad-hoc but amazingly useful methods, like regularized linear regression, that I have in my toolbox."

I hoped that with a specific example in hand of "something that doesn't look all that Bayesian on the surface, but turns out to be Bayesian after all" - and an explanation of the difference between passwords and knowledge - and an explanation of the difference between tools and laws - maybe then I could convey such of the rhythm as can be understood without personally going on the quest.

Of course this is not the full Secret of the Bayesian Conspiracy, but it's all that I can convey at this point. Besides, the complete secret is known only to the Bayes Council, and if I told you, I'd have to hire you.

To see through the surface adhockery of a cognitive process, to the Bayesian structure underneath - to perceive the probability flows, and know how, not just know that, this cognition too is Bayesian - as it always is - as it always must be - to be able to sense the Force underlying all cognition - this, is the Bayes-Sight.

"...And the Queen of Kashfa sees with the Eye of the Serpent."
"I don't know that she sees with it," I said. "She's still recovering from the operation. But that's an interesting thought. If she could see with it, what might she behold?"
"The clear, cold lines of eternity, I daresay. Beneath all Shadow."
-- Roger Zelazny, Prince of Chaos

Referenced by: Conditional Independence, and Naive Bayes • To Spread Science, Keep It Secret • Zombie Responses • Science Doesn't Trust Your Rationality • A Premature Word on AI • My Naturalistic Awakening • My Bayesian Enlightenment • The Nature of Logic

Original with comments: Searching for Bayes-Structure

Conditional Independence, and Naive Bayes

Eliezer Yudkowsky, 01 March 2008 01:59AM

Followup to: Searching for Bayes-Structure

Previously I spoke of mutual information between X and Y, I(X;Y), which is the difference between the entropy of the joint probability distribution, H(X,Y) and the entropies of the marginal distributions, H(X) + H(Y).

I gave the example of a variable X, having eight states 1..8 which are all equally probable if we have not yet encountered any evidence; and a variable Y, with states 1..4, which are all equally probable if we have not yet encountered any evidence. Then if we calculate the marginal entropies H(X) and H(Y), we will find that X has 3 bits of entropy, and Y has 2 bits.

However, we also know that X and Y are both even or both odd; and this is all we know about the relation between them. So for the joint distribution (X,Y) there are only 16 possible states, all equally probable, for a joint entropy of 4 bits. This is a 1-bit entropy defect, compared to 5 bits of entropy if X and Y were independent. This entropy defect is the mutual information - the information that X tells us about Y, or vice versa, so that we are not as uncertain about one after having learned the other.

Suppose, however, that there exists a third variable Z. Z has two states, "even" and "odd", perfectly correlated to the evenness or oddness of (X,Y). In fact, we'll suppose that Z is just the question "Are X and Y even or odd?"

If we have no evidence about X and Y, then Z itself necessarily has 1 bit of entropy on the information given. There is 1 bit of mutual information between Z and X, and 1 bit of mutual information between Z and Y. And, as previously noted, 1 bit of mutual information between X and Y. So how much entropy for the whole system (X,Y,Z)? You might naively expect that

H(X,Y,Z) = H(X) + H(Y) + H(Z) - I(X;Z) - I(Z;Y) - I(X;Y)

but this turns out not to be the case.

The joint system (X,Y,Z) only has 16 possible states - since Z is just the question "Are X+Y even or odd?" - so H(X,Y,Z) = 4 bits.

But if you calculate the formula just given, you get

(3 + 2 + 1 - 1 - 1 - 1)bits = 3 bits = WRONG!

Why? Because if you have the mutual information between X and Z, and the mutual information between Z and Y, that may include some of the same mutual information that we'll calculate exists between X and Y. In this case, for example, knowing that X is even tells us that Z is even, and knowing that Z is even tells us that Y is even, but this is the same information that X would tell us about Y. We double-counted some of our knowledge, and so came up with too little entropy.

The correct formula is (I believe):

H(X,Y,Z) = H(X) + H(Y) + H(Z) - I(X;Z) - I(Z;Y) - I(X;Y | Z)

Here the last term, I(X;Y | Z), means, "the information that X tells us about Y, given that we already know Z". In this case, X doesn't tell us anything about Y, given that we already know Z, so the term comes out as zero - and the equation gives the correct answer. There, isn't that nice?

"No," you correctly reply, "for you have not told me how to calculate I(X;Y|Z), only given me a verbal argument that it ought to be zero."

We calculate I(X;Y|Z) just the way you would expect. I(X;Y) = H(X) + H(Y) - H(X,Y), so:

I(X;Y|Z) = H(X|Z) + H(Y|Z) - H(X,Y|Z)

And now, I suppose, you want to know how to calculate the conditional entropy? Well, the original formula for the entropy is:

H(S) = Sum i: p(Si)*-log_2(p(Si))

If we then learned a new fact Z_0, our remaining uncertainty about S would be:

H(S|Z_0) = Sum i: p(Si|Z_0)*-log_2(p(Si|Z_0))

So if we're going to learn a new fact Z, but we don't know which Z yet, then, on average, we expect to be around this uncertain of S afterward:

H(S|Z) = Sum j: (p(Zj) * Sum i: p(Si|Zj)*-log_2(p(Si|Zj)))

And that's how one calculates conditional entropies; from which, in turn, we can get the conditional mutual information.

There are all sorts of ancillary theorems here, like:

H(X|Y) = H(X,Y) - H(Y)

and

if I(X;Z) = 0 and I(Y;X|Z) = 0 then I(X;Y) = 0

but I'm not going to go into those.

"But," you ask, "what does this have to do with the nature of words and their hidden Bayesian structure?"

I am just so unspeakably glad that you asked that question, because I was planning to tell you whether you liked it or not. But first there are a couple more preliminaries.

You will remember - yes, you will remember - that there is a duality between mutual information and Bayesian evidence. Mutual information is positive if and only if the probability of at least some joint events P(x, y) does not equal the product of the probabilities of the separate events P(x)*P(y). This, in turn, is exactly equivalent to the condition that Bayesian evidence exists between x and y:

I(X;Y) > 0 =>
P(x,y) != P(x)*P(y)
P(x,y) / P(y) != P(x)
P(x|y) != P(x)

If you're conditioning on Z, you just adjust the whole derivation accordingly:

I(X;Y | Z) > 0 =>
P(x,y|z) != P(x|z)*P(y|z)
P(x,y|z) / P(y|z) != P(x|z)
(P(x,y,z) / P(z)) / (P(y, z) / P(z)) != P(x|z)
P(x,y,z) / P(y,z) != P(x|z)
P(x|y,z) != P(x|z)

Which last line reads "Even knowing Z, learning Y still changes our beliefs about X."

Conversely, as in our original case of Z being "even" or "odd", Z screens off X from Y - that is, if we know that Z is "even", learning that Y is in state 4 tells us nothing more about whether X is 2, 4, 6, or 8. Or if we know that Z is "odd", then learning that X is 5 tells us nothing more about whether Y is 1 or 3. Learning Z has rendered X and Y conditionally independent.

Conditional independence is a hugely important concept in probability theory - to cite just one example, without conditional independence, the universe would have no structure.

Today, though, I only intend to talk about one particular kind of conditional independence - the case of a central variable that screens off other variables surrounding it, like a central body with tentacles.

Let there be five variables U, V, W, X, Y; and moreover, suppose that for every pair of these variables, one variable is evidence about the other. If you select U and W, for example, then learning U=U_1 will tell you something you didn't know before about the probability W=W_1.

An unmanageable inferential mess? Evidence gone wild? Not necessarily.

Maybe U is "Speaks a language", V is "Two arms and ten digits", W is "Wears clothes", X is "Poisonable by hemlock", and Y is "Red blood". Now if you encounter a thing-in-the-world, that might be an apple and might be a rock, and you learn that this thing speaks Chinese, you are liable to assess a much higher probability that it wears clothes; and if you learn that the thing is not poisonable by hemlock, you will assess a somewhat lower probability that it has red blood.

Now some of these rules are stronger than others. There is the case of Fred, who is missing a finger due to a volcano accident, and the case of Barney the Baby who doesn't speak yet, and the case of Irving the IRCBot who emits sentences but has no blood. So if we learn that a certain thing is not wearing clothes, that doesn't screen off everything that its speech capability can tell us about its blood color. If the thing doesn't wear clothes but does talk, maybe it's Nude Nellie.

This makes the case more interesting than, say, five integer variables that are all odd or all even, but otherwise uncorrelated. In that case, knowing any one of the variables would screen off everything that knowing a second variable could tell us about a third variable.

But here, we have dependencies that don't go away as soon as we learn just one variable, as the case of Nude Nellie shows. So is it an unmanageable inferential inconvenience?

Fear not! for there may be some sixth variable Z, which, if we knew it, really would screen off every pair of variables from each other. There may be some variable Z - even if we have to construct Z rather than observing it directly - such that:

p(u|v,w,x,y,z) = p(u|z)
p(v|u,w,x,y,z) = p(v|z)
p(w|u,v,x,y,z) = p(w|z)
...

Perhaps, given that a thing is "human", then the probabilities of it speaking, wearing clothes, and having the standard number of fingers, are all independent. Fred may be missing a finger - but he is no more likely to be a nudist than the next person; Nude Nellie never wears clothes, but knowing this doesn't make it any less likely that she speaks; and Baby Barney doesn't talk yet, but is not missing any limbs.

This is called the "Naive Bayes" method, because it usually isn't quite true, but pretending that it's true can simplify the living daylights out of your calculations. We don't keep separate track of the influence of clothed-ness on speech capability given finger number. We just use all the information we've observed to keep track of the probability that this thingy is a human (or alternatively, something else, like a chimpanzee or robot) and then use our beliefs about the central class to predict anything we haven't seen yet, like vulnerability to hemlock.

Any observations of U, V, W, X, and Y just act as evidence for the central class variable Z, and then we use the posterior distribution on Z to make any predictions that need making about unobserved variables in U, V, W, X, and Y.

Sound familiar? It should:

As a matter of fact, if you use the right kind of neural network units, this "neural network" ends up exactly, mathematically equivalent to Naive Bayes. The central unit just needs a logistic threshold - an S-curve response - and the weights of the inputs just need to match the logarithms of the likelihood ratios, etcetera. In fact, it's a good guess that this is one of the reasons why logistic response often works so well in neural networks - it lets the algorithm sneak in a little Bayesian reasoning while the designers aren't looking.

Just because someone is presenting you with an algorithm that they call a "neural network" with buzzwords like "scruffy" and "emergent" plastered all over it, disclaiming proudly that they have no idea how the learned network works - well, don't assume that their little AI algorithm really is Beyond the Realms of Logic. For this paradigm of adhockery , if it works, will turn out to have Bayesian structure; it may even be exactly equivalent to an algorithm of the sort called "Bayesian".

Even if it doesn't look Bayesian, on the surface.

And then you just know that the Bayesians are going to start explaining exactly how the algorithm works, what underlying assumptions it reflects, which environmental regularities it exploits, where it works and where it fails, and even attaching understandable meanings to the learned network weights.

Disappointing, isn't it?

Sequence: A Human's Guide to Words

Superexponential Conceptspace, and Simple Words

Words as Mental Paintbrush Handles

Referenced by: Words as Mental Paintbrush Handles • 37 Ways That Words Can Be Wrong • Dissolving the Question • Timeless Causality

Original with comments: Conditional Independence, and Naive Bayes

Words as Mental Paintbrush Handles

Eliezer Yudkowsky, 01 March 2008 11:58PM

Followup to: Conditional Independence, and Naive Bayes

(We should be done with the mathy posts, I think, at least for now. But forgive me if, ironically, I end up resorting to Rationality Quotes for a day or two. I'm currently at the AGI-08 conference, which, as of the first session, is not nearly so bad as I feared.)

Suppose I tell you: "It's the strangest thing: The lamps in this hotel have triangular lightbulbs."

You may or may not have visualized it - if you haven't done it yet, do so now - what, in your mind's eye, does a "triangular lightbulb" look like?

In your mind's eye, did the glass have sharp edges, or smooth?

When the phrase "triangular lightbulb" first crossed my mind - no, the hotel doesn't have them - then as best as my introspection could determine, I first saw a pyramidal lightbulb with sharp edges, then (almost immediately) the edges were smoothed, and then my mind generated a loop of flourescent bulb in the shape of a smooth triangle as an alternative.

As far as I can tell, no deliberative/verbal thoughts were involved - just wordless reflex flinch away from the imaginary mental vision of sharp glass, which design problem was solved before I could even think in words.

Believe it or not, for some decades, there was a serious debate about whether people really had mental images in their mind - an actual picture of a chair somewhere - or if people just naively thought they had mental images (having been misled by "introspection", a very bad forbidden activity), while actually just having a little "chair" label, like a LISP token, active in their brain.

I am trying hard not to say anything like "How spectacularly silly," because there is always the hindsight effect to consider, but: how spectacularly silly.

This academic paradigm, I think, was mostly a deranged legacy of behaviorism, which denied the existence of thoughts in humans, and sought to explain all human phenomena as "reflex", including speech. Behaviorism probably deserves its own post at some point, as it was a perversion of rationalism; but this is not that post.

"You call it 'silly'," you inquire, "but how do you know that your brain represents visual images? Is it merely that you can close your eyes and see them?"

This question used to be harder to answer, back in the day of the controversy. If you wanted to prove the existence of mental imagery "scientifically", rather than just by introspection, you had to infer the existence of mental imagery from experiments like, e.g.: Show subjects two objects and ask them if one can be rotated into correspondence with the other. The response time is linearly proportional to the angle of rotation required. This is easy to explain if you are actually visualizing the image and continuously rotating it at a constant speed, but hard to explain if you are just checking propositional features of the image.

Today we can actually neuroimage the little pictures in the visual cortex. So, yes, your brain really does represent a detailed image of what it sees or imagines. See Stephen Kosslyn's Image and Brain: The Resolution of the Imagery Debate.

Part of the reason people get in trouble with words, is that they do not realize how much complexity lurks behind words.

Can you visualize a "green dog"? Can you visualize a "cheese apple"?

"Apple" isn't just a sequence of two syllables or five letters. That's a shadow. That's the tip of the tiger's tail.

Words, or rather the concepts behind them, are paintbrushes - you can use them to draw images in your own mind. Literally draw, if you employ concepts to make a picture in your visual cortex. And by the use of shared labels, you can reach into someone else's mind, and grasp their paintbrushes to draw pictures in their minds - sketch a little green dog in their visual cortex.

But don't think that, because you send syllables through the air, or letters through the Internet, it is the syllables or the letters that draw pictures in the visual cortex. That takes some complex instructions that wouldn't fit in the sequence of letters. "Apple" is 5 bytes, and drawing a picture of an apple from scratch would take more data than that.

"Apple" is merely the tag attached to the true and wordless apple concept, which can paint a picture in your visual cortex, or collide with "cheese", or recognize an apple when you see one, or taste its archetype in apple pie, maybe even send out the motor behavior for eating an apple...

And it's not as simple as just calling up a picture from memory. Or how would you be able to visualize combinations like a "triangular lightbulb" - imposing triangleness on lightbulbs, keeping the essence of both, even if you've never seen such a thing in your life?

Don't make the mistake the behaviorists made. There's far more to speech than sound in air. The labels are just pointers - "look in memory area 1387540". Sooner or later, when you're handed a pointer, it comes time to dereference it, and actually look in memory area 1387540.

What does a word point to?

Sequence: A Human's Guide to Words

Conditional Independence, and Naive Bayes

Variable Question Fallacies

Referenced by: Variable Question Fallacies • 37 Ways That Words Can Be Wrong • The Rhythm of Disagreement • Principles of Disagreement • Detached Lever Fallacy

Original with comments: Words as Mental Paintbrush Handles

Rationality Quotes 10

Eliezer Yudkowsky, 03 March 2008 05:48AM

"Yes, I am the last man to have walked on the moon, and that's a very dubious and disappointing honor. It's been far too long."
-- Gene Cernan

"Man, you're no smarter than me. You're just a fancier kind of stupid."
-- Spider Robinson, Distraction

"Each problem that I solved became a rule which served afterwards to solve other problems."
-- Rene Descartes, Discours de la Methode

"Faith is Hope given too much credit."
-- Matt Tuozzo

"Your Highness, I have no need of this hypothesis."
-- Pierre-Simon Laplace, to Napoleon, explaining why his works on celestial mechanics made no mention of God.

"'For, finally, one can only judge oneself by one's actions,' thought Elric. 'I have looked at what I have done, not at what I meant to do or thought I would like to do, and what I have done has, in the main, been foolish, destructive, and with little point. Yyrkoon was right to despise me and that was why I hated him so.'"
-- Michael Moorcock, Elric of Melniboné

"You will quickly find that if you are completely and self-deprecatingly truthful about how much you owe other people, the world at large will treat you like you did every bit of the invention yourself and are just being becomingly modest about your innate genius."
-- Eric S. Raymond

"The longer I live the more I see that I am never wrong about anything, and that all the pains that I have so humbly taken to verify my notions have only wasted my time."
-- George Bernard Shaw

"The trouble is that consciousness theories are very easy to dream up... Theories that explain intelligence, on the other hand, are fiendishly difficult to come by and so are profoundly useful. I don't know for sure that intelligence always produces consciousness, but I do know that if you assume it does you'll never be disappointed."
-- John K. Clark

"Intelligence is silence, truth being invisible. But what a racket I make in declaring this."
-- Ned Rorem, "Random Notes from a Diary"

Presently the mage said, speaking softly, "Do you see, Arren, how an act is not, as young men think, like a rock that one picks up and throws, and it hits or misses, and that's the end of it. When that rock is lifted the earth is lighter, the hand that bears it heavier. When it is thrown the circuits of the stars respond, and where it strikes or falls the universe is changed. On every act the balance of the whole depends. The winds and seas, the powers of water and earth and light, all that these do, and all that the beasts and green things do, is well done, and rightly done. All these act within the Equilibrium. From the hurricane and the great whale's sounding to the fall of a dry leaf and the flight of a gnat, all they do is done within the balance of the whole. But we, in so far as we have power over the world and over one another, we must learn to do what the leaf and the whale and the wind do of their own nature. We must learn to keep the balance. Having intelligence, we must not act in ignorance. Having choice, we must not act without responsibility. Who am I - though I have the power to do it - to punish and reward, playing with men's destinies?"
"But then," the boy said, frowning at the stars, "is the balance to be kept by doing nothing? Surely a man must act, even not knowing all the consequences of his act, if anything is to be done at all?"
"Never fear. It is much easier for men to act than to refrain from acting. We will continue to do good, and to do evil... But if there were a king over us all again, and he sought counsel of a mage, as in the days of old, and I were that mage, I would say to him: My lord, do nothing because it is righteous, or praiseworthy, or noble, to do so; do nothing because it seems good to do so; do only that which you must do, and which you cannot do in any other way."
-- Ursula K. LeGuin, The Farthest Shore

Original with comments: Rationality Quotes 10

Rationality Quotes 11

Eliezer Yudkowsky, 04 March 2008 05:48AM

"If we let ethical considerations get in the way of scientific hubris, then the feminists have won!"
-- Helarxe

"The trajectory to hell is paved with locally-good intentions."
-- Matt Gingell

"To a mouse, cheese is cheese; that's why mousetraps work."
-- Wendell Johnson, quoted in Language in Thought and Action

"'Ethical consideration' has come to mean reasoning from an ivory tower about abstract non-issues while people die."
-- Zeb Haradon

"I intend to live forever. So far, so good."
-- Rick Potvin

"The accessory optic system: The AOS, extensively studied in the rabbit, arises from a special class of ganglion cells, the cells of Dogiel, that are directionally selective and respond best to slow rates of movement. They project to the terminal nuclei which in turn project to the dorsal cap of Kooy of the inferior olive. The climbing fibers from the olive project to the flocculo-nodular lobe of the cerebellum from where the brain stem occulomotor centers are reached through the vestibular nuclei."
-- MIT Encyclopedia of the Cognitive Sciences, "Visual Anatomy and Physiology"

"Fight for those you have lost, and for those you don't want to lose."
-- Claymore

"Which facts are likely to reappear? The simple facts. How to recognize them? Choose those that seem simple. Either this simplicity is real or the complex elements are indistinguishable. In the first case we're likely to meet this simple fact again either alone or as an element in a complex fact. The second case too has a good chance of recurring since nature doesn't randomly construct such cases."
-- Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance

"Revolutions begin not when the first barricades are erected or even when people lose faith in the old ways of doing things, but rather when they realize that fundamental change is possible."
-- Steven Metz

"First Law of Anime Acoustics: In space, loud sounds, like explosions, are even louder because there is no air to get in the way."
"Law of Inherent Combustibility: Everything explodes. Everything."
"Law of Conservation of Firepower: Any powerful weapon capable of destroying/defeating an opponent in a single shot will invariably be reserved and used only as a last resort."
-- Laws of Anime

"On the Popo Agie in September,
I watched the water toss through the same arc,
each molecule passing through and never returning,
but the whole a permanence of chaos,
repeating to the casual glance,
various to the closer look."
-- Mick McAllister

Original with comments: Rationality Quotes 11

Variable Question Fallacies

Eliezer Yudkowsky, 05 March 2008 06:22AM

Followup to: Words as Mental Paintbrush Handles

Albert: "Every time I've listened to a tree fall, it made a sound, so I'll guess that other trees falling also make sounds. I don't believe the world changes around when I'm not looking."
Barry: "Wait a minute. If no one hears it, how can it be a sound?"

While writing the dialogue of Albert and Barry in their dispute over whether a falling tree in a deserted forest makes a sound, I sometimes found myself losing empathy with my characters. I would start to lose the gut feel of why anyone would ever argue like that, even though I'd seen it happen many times.

On these occasions, I would repeat to myself, "Either the falling tree makes a sound, or it does not!" to restore my borrowed sense of indignation.

(P or ~P) is not always a reliable heuristic, if you substitute arbitrary English sentences for P. "This sentence is false" cannot be consistently viewed as true or false. And then there's the old classic, "Have you stopped beating your wife?"

Now if you are a mathematician, and one who believes in classical (rather than intuitionistic) logic, there are ways to continue insisting that (P or ~P) is a theorem: for example, saying that "This sentence is false" is not a sentence.

But such resolutions are subtle, which suffices to demonstrate a need for subtlety. You cannot just bull ahead on every occasion with "Either it does or it doesn't!"

So does the falling tree make a sound, or not, or...?

Surely, 2 + 2 = X or it does not? Well, maybe, if it's really the same X, the same 2, and the same + and =. If X evaluates to 5 on some occasions and 4 on another, your indignation may be misplaced.

To even begin claiming that (P or ~P) ought to be a necessary truth, the symbol P must stand for exactly the same thing in both halves of the dilemma. "Either the fall makes a sound, or not!" - but if Albert::sound is not the same as Barry::sound, there is nothing paradoxical about the tree making an Albert::sound but not a Barry::sound.

(The :: idiom is something I picked up in my C++ days for avoiding namespace collisions. If you've got two different packages that define a class Sound, you can write Package1::Sound to specify which Sound you mean. The idiom is not widely known, I think; which is a pity, because I often wish I could use it in writing.)

The variability may be subtle: Albert and Barry may carefully verify that it is the same tree, in the same forest, and the same occasion of falling, just to ensure that they really do have a substantive disagreement about exactly the same event. And then forget to check that they are matching this event against exactly the same concept.

Think about the grocery store that you visit most often: Is it on the left side of the street, or the right? But of course there is no "the left side" of the street, only your left side, as you travel along it from some particular direction. Many of the words we use are really functions of implicit variables supplied by context.

It's actually one heck of a pain, requiring one heck of a lot of work, to handle this kind of problem in an Artificial Intelligence program intended to parse language - the phenomenon going by the name of "speaker deixis".

"Martin told Bob the building was on his left." But "left" is a function-word that evaluates with a speaker-dependent variable invisibly grabbed from the surrounding context. Whose "left" is meant, Bob's or Martin's?

The variables in a variable question fallacy often aren't neatly labeled - it's not as simple as "Say, do you think Z + 2 equals 6?"

If a namespace collision introduces two different concepts that look like "the same concept" because they have the same name - or a map compression introduces two different events that look like the same event because they don't have separate mental files - or the same function evaluates in different contexts - then reality itself becomes protean, changeable. At least that's what the algorithm feels like from inside. Your mind's eye sees the map, not the territory directly.

If you have a question with a hidden variable, that evaluates to different expressions in different contexts, it feels like reality itself is unstable - what your mind's eye sees, shifts around depending on where it looks.

This often confuses undergraduates (and postmodernist professors) who discover a sentence with more than one interpretation; they think they have discovered an unstable portion of reality.

"Oh my gosh! 'The Sun goes around the Earth' is true for Hunga Huntergatherer, but for Amara Astronomer, 'The Sun goes around the Earth' is false! There is no fixed truth!" The deconstruction of this sophomoric nitwittery is left as an exercise to the reader.

And yet, even I initially found myself writing "If X is 5 on some occasions and 4 on another, the sentence '2 + 2 = X' may have no fixed truth-value." There is not one sentence with a variable truth-value. "2 + 2 = X" has no truth-value. It is not a proposition, not yet, not as mathematicians define proposition-ness, any more than "2 + 2 =" is a proposition, or "Fred jumped over the" is a grammatical sentence.

But this fallacy tends to sneak in, even when you allegedly know better, because, well, that's how the algorithm feels from inside.

Sequence: A Human's Guide to Words

Words as Mental Paintbrush Handles

37 Ways That Words Can Be Wrong

Referenced by: 37 Ways That Words Can Be Wrong • Dissolving the Question • 2-Place and 1-Place Words

Original with comments: Variable Question Fallacies

37 Ways That Words Can Be Wrong

Eliezer Yudkowsky, 06 March 2008 05:09AM

Followup to: Just about every post in February, and some in March

Some reader is bound to declare that a better title for this post would be "37 Ways That You Can Use Words Unwisely", or "37 Ways That Suboptimal Use Of Categories Can Have Negative Side Effects On Your Cognition".

But one of the primary lessons of this gigantic list is that saying "There's no way my choice of X can be 'wrong'" is nearly always an error in practice, whatever the theory. You can always be wrong. Even when it's theoretically impossible to be wrong, you can still be wrong. There is never a Get-Out-Of-Jail-Free card for anything you do. That's life.

Besides, I can define the word "wrong" to mean anything I like - it's not like a word can be wrong.

Personally, I think it quite justified to use the word "wrong" when:

1. A word fails to connect to reality in the first place. Is Socrates a framster? Yes or no? (The Parable of the Dagger.)

2. Your argument, if it worked, could coerce reality to go a different way by choosing a different word definition. Socrates is a human, and humans, by definition, are mortal. So if you defined humans to not be mortal, would Socrates live forever? (The Parable of Hemlock.)

3. You try to establish any sort of empirical proposition as being true "by definition". Socrates is a human, and humans, by definition, are mortal. So is it a logical truth if we empirically predict that Socrates should keel over if he drinks hemlock? It seems like there are logically possible, non-self-contradictory worlds where Socrates doesn't keel over - where he's immune to hemlock by a quirk of biochemistry, say. Logical truths are true in all possible worlds, and so never tell you which possible world you live in - and anything you can establish "by definition" is a logical truth. (The Parable of Hemlock.)

4. You unconsciously slap the conventional label on something, without actually using the verbal definition you just gave. You know perfectly well that Bob is "human", even though, on your definition, you can never call Bob "human" without first observing him to be mortal. (The Parable of Hemlock.)

5. The act of labeling something with a word, disguises a challengable inductive inference you are making. If the last 11 egg-shaped objects drawn have been blue, and the last 8 cubes drawn have been red, it is a matter of induction to say this rule will hold in the future. But if you call the blue eggs "bleggs" and the red cubes "rubes", you may reach into the barrel, feel an egg shape, and think "Oh, a blegg." (Words as Hidden Inferences.)

6. You try to define a word using words, in turn defined with ever-more-abstract words, without being able to point to an example. "What is red?" "Red is a color." "What's a color?" "It's a property of a thing?" "What's a thing? What's a property?" It never occurs to you to point to a stop sign and an apple. (Extensions and Intensions.)

7. The extension doesn't match the intension. We aren't consciously aware of our identification of a red light in the sky as "Mars", which will probably happen regardless of your attempt to define "Mars" as "The God of War". (Extensions and Intensions.)

8. Your verbal definition doesn't capture more than a tiny fraction of the category's shared characteristics, but you try to reason as if it does. When the philosophers of Plato's Academy claimed that the best definition of a human was a "featherless biped", Diogenes the Cynic is said to have exhibited a plucked chicken and declared "Here is Plato's Man." The Platonists promptly changed their definition to "a featherless biped with broad nails". (Similarity Clusters.)

9. You try to treat category membership as all-or-nothing, ignoring the existence of more and less typical subclusters. Ducks and penguins are less typical birds than robins and pigeons. Interestingly, a between-groups experiment showed that subjects thought a dis ease was more likely to spread from robins to ducks on an island, than from ducks to robins. (Typicality and Asymmetrical Similarity.)

10. A verbal definition works well enough in practice to point out the intended cluster of similar things, but you nitpick exceptions. Not every human has ten fingers, or wears clothes, or uses language; but if you look for an empirical cluster of things which share these characteristics, you'll get enough information that the occasional nine-fingered human won't fool you. (The Cluster Structure of Thingspace.)

11. You ask whether something "is" or "is not" a category member but can't name the question you really want answered. What is a "man"? Is Barney the Baby Boy a "man"? The "correct" answer may depend considerably on whether the query you really want answered is "Would hemlock be a good thing to feed Barney?" or "Will Barney make a good husband?" (Disguised Queries.)

12. You treat intuitively perceived hierarchical categories like the only correct way to parse the world, without realizing that other forms of statistical inference are possible even though your brain doesn't use them. It's much easier for a human to notice whether an object is a "blegg" or "rube"; than for a human to notice that red objects never glow in the dark, but red furred objects have all the other characteristics of bleggs. Other statistical algorithms work differently. (Neural Categories.)

13. You talk about categories as if they are manna fallen from the Platonic Realm, rather than inferences implemented in a real brain. The ancient philosophers said "Socrates is a man", not, "My brain perceptually classifies Socrates as a match against the 'human' concept". (How An Algorithm Feels From Inside.)

14. You argue about a category membership even after screening off all questions that could possibly depend on a category-based inference. After you observe that an object is blue, egg-shaped, furred, flexible, opaque, luminescent, and palladium-containing, what's left to ask by arguing, "Is it a blegg?" But if your brain's categorizing neural network contains a (metaphorical) central unit corresponding to the inference of blegg-ness, it may still feel like there's a leftover question. (How An Algorithm Feels From Inside.)

15. You allow an argument to slide into being about definitions, even though it isn't what you originally wanted to argue about. If, before a dispute started about whether a tree falling in a deserted forest makes a "sound", you asked the two soon-to-be arguers whether they thought a "sound" should be defined as "acoustic vibrations" or "auditory experiences", they'd probably tell you to flip a coin. Only after the argument starts does the definition of a word become politically charged. (Disputing Definitions.)

16. You think a word has a meaning, as a property of the word itself; rather than there being a label that your brain associates to a particular concept. When someone shouts, "Yikes! A tiger!", evolution would not favor an organism that thinks, "Hm... I have just heard the syllables 'Tie' and 'Grr' which my fellow tribemembers associate with their internal analogues of my own tiger concept and which aiiieeee CRUNCH CRUNCH GULP." So the brain takes a shortcut, and it seems that the meaning of tigerness is a property of the label itself. People argue about the correct meaning of a label like "sound". (Feel the Meaning.)

17. You argue over the meanings of a word, even after all sides understand perfectly well what the other sides are trying to say. The human ability to associate labels to concepts is a tool for communication. When people want to communicate, we're hard to stop; if we have no common language, we'll draw pictures in sand. When you each understand what is in the other's mind, you are done. (The Argument From Common Usage.)

18. You pull out a dictionary in the middle of an empirical or moral argument. Dictionary editors are historians of usage, not legislators of language. If the common definition contains a problem - if "Mars" is defined as the God of War, or a "dolphin" is defined as a kind of fish, or "Negroes" are defined as a separate category from humans, the dictionary will reflect the standard mistake. (The Argument From Common Usage.)

19. You pull out a dictionary in the middle of any argument ever. Seriously, what the heck makes you think that dictionary editors are an authority on whether "atheism" is a "religion" or whatever? If you have any substantive issue whatsoever at stake, do you really think dictionary editors have access to ultimate wisdom that settles the argument? (The Argument From Common Usage.)

20. You defy common usage without a reason, making it gratuitously hard for others to understand you. Fast stand up plutonium, with bagels without handle. (The Argument From Common Usage.)

21. You use complex renamings to create the illusion of inference. Is a "human" defined as a "mortal featherless biped"? Then write: "All [mortal featherless bipeds] are mortal; Socrates is a [mortal featherless biped]; therefore, Socrates is mortal." Looks less impressive that way, doesn't it? (Empty Labels.)

22. You get into arguments that you could avoid if you just didn't use the word. If Albert and Barry aren't allowed to use the word "sound", then Albert will have to say "A tree falling in a deserted forest generates acoustic vibrations", and Barry will say "A tree falling in a deserted forest generates no auditory experiences". When a word poses a problem, the simplest solution is to eliminate the word and its synonyms. (Taboo Your Words.)

23. The existence of a neat little word prevents you from seeing the details of the thing you're trying to think about. What actually goes on in schools once you stop calling it "education"? What's a degree, once you stop calling it a "degree"? If a coin lands "heads", what's its radial orientation? What is "truth", if you can't say "accurate" or "correct" or "represent" or "reflect" or "semantic" or "believe" or "knowledge" or "map" or "real" or any other simple term? (Replace the Symbol with the Substance.)

24. You have only one word, but there are two or more different things-in-reality, so that all the facts about them get dumped into a single undifferentiated mental bucket. It's part of a detective's ordinary work to observe that Carol wore red last night, or that she has black hair; and it's part of a detective's ordinary work to wonder if maybe Carol dyes her hair. But it takes a subtler detective to wonder if there are two Carols, so that the Carol who wore red is not the same as the Carol who had black hair. (Fallacies of Compression.)

25. You see patterns where none exist, harvesting other characteristics from your definitions even when there is no similarity along that dimension. In Japan, it is thought that people of blood type A are earnest and creative, blood type Bs are wild and cheerful, blood type Os are agreeable and sociable, and blood type ABs are cool and controlled. (Categorizing Has Consequences.)

26. You try to sneak in the connotations of a word, by arguing from a definition that doesn't include the connotations. A "wiggin" is defined in the dictionary as a person with green eyes and black hair. The word "wiggin" also carries the connotation of someone who commits crimes and launches cute baby squirrels, but that part isn't in the dictionary. So you point to someone and say: "Green eyes? Black hair? See, told you he's a wiggin! Watch, next he's going to steal the silverware." (Sneaking in Connotations.)

27. You claim "X, by definition, is a Y!" On such occasions you're almost certainly trying to sneak in a connotation of Y that wasn't in your given definition. You define "human" as a "featherless biped", and point to Socrates and say, "No feathers - two legs - he must be human!" But what you really care about is something else, like mortality. If what was in dispute was Socrates's number of legs, the other fellow would just reply, "Whaddaya mean, Socrates's got two legs? That's what we're arguing about in the first place!" (Arguing "By Definition".)

28. You claim "Ps, by definition, are Qs!" If you see Socrates out in the field with some biologists, gathering herbs that might confer resistance to hemlock, there's no point in arguing "Men, by definition, are mortal!" The main time you feel the need to tighten the vise by insisting that something is true "by definition" is when there's other information that calls the default inference into doubt. (Arguing "By Definition".)

29. You try to establish membership in an empirical cluster "by definition". You wouldn't feel the need to say, "Hinduism, by definition, is a religion!" because, well, of course Hinduism is a religion. It's not just a religion "by definition", it's, like, an actual religion. Atheism does not resemble the central members of the "religion" cluster, so if it wasn't for the fact that atheism is a religion by definition, you might go around thinking that atheism wasn't a religion. That's why you've got to crush all opposition by pointing out that "Atheism is a religion" is true by definition, because it isn't true any other way. (Arguing "By Definition".)

30. Your definition draws a boundary around things that don't really belong together. You can claim, if you like, that you are defining the word "fish" to refer to salmon, guppies, sharks, dolphins, and trout, but not jellyfish or algae. You can claim, if you like, that this is merely a list, and there is no way a list can be "wrong". Or you can stop playing nitwit games and admit that you made a mistake and that dolphins don't belong on the fish list. (Where to Draw the Boundary?)

31. You use a short word for something that you won't need to describe often, or a long word for something you'll need to describe often. This can result in inefficient thinking, or even misapplications of Occam's Razor, if your mind thinks that short sentences sound "simpler". Which sounds more plausible, "God did a miracle" or "A supernatural universe-creating entity temporarily suspended the laws of physics"? (Entropy, and Short Codes.)

32. You draw your boundary around a volume of space where there is no greater-than-usual density, meaning that the associated word does not correspond to any performable Bayesian inferences. Since green-eyed people are not more likely to have black hair, or vice versa, and they don't share any other characteristics in common, why have a word for "wiggin"? (Mutual Information, and Density in Thingspace.)

33. You draw an unsimple boundary without any reason to do so. The act of defining a word to refer to all humans, except black people, seems kind of suspicious. If you don't present reasons to draw that particular boundary, trying to create an "arbitrary" word in that location is like a detective saying: "Well, I haven't the slightest shred of support one way or the other for who could've murdered those orphans... but have we considered John Q. Wiffleheim as a suspect?" (Superexponential Conceptspace, and Simple Words.)

34. You use categorization to make inferences about properties that don't have the appropriate empirical structure, namely, conditional independence given knowledge of the class, to be well-approximated by Naive Bayes. No way am I trying to summarize this one. Just read the blog post. (Conditional Independence, and Naive Bayes.)

35. You think that words are like tiny little LISP symbols in your mind, rather than words being labels that act as handles to direct complex mental paintbrushes that can paint detailed pictures in your sensory workspace. Visualize a "triangular lightbulb". What did you see? (Words as Mental Paintbrush Handles.)

36. You use a word that has different meanings in different places as though it meant the same thing on each occasion, possibly creating the illusion of something protean and shifting. "Martin told Bob the building was on his left." But "left" is a function-word that evaluates with a speaker-dependent variable grabbed from the surrounding context. Whose "left" is meant, Bob's or Martin's? (Variable Question Fallacies.)

37. You think that definitions can't be "wrong", or that "I can define a word any way I like!" This kind of attitude teaches you to indignantly defend your past actions, instead of paying attention to their consequences, or fessing up to your mistakes. (37 Ways That Suboptimal Use Of Categories Can Have Negative Side Effects On Your Cognition.)

Everything you do in the mind has an effect, and your brain races ahead unconsciously without your supervision.

Saying "Words are arbitrary; I can define a word any way I like" makes around as much sense as driving a car over thin ice with the accelerator floored and saying, "Looking at this steering wheel, I can't see why one radial angle is special - so I can turn the steering wheel any way I like."

If you're trying to go anywhere, or even just trying to survive, you had better start paying attention to the three or six dozen optimality criteria that control how you use words, definitions, categories, classes, boundaries, labels, and concepts.

Sequence: A Human's Guide to Words

Variable Question Fallacies

Referenced by: The Generalized Anti-Zombie Principle • The Ultimate Source • Heading Toward Morality • Setting Up Metaethics • Dreams of AI Design • Efficient Cross-Domain Optimization • The Nature of Logic • Conversation Halters

Original with comments: 37 Ways That Words Can Be Wrong

Gary Gygax Annihilated at 69

Eliezer Yudkowsky, 06 March 2008 08:54PM

Yesterday I heard that Gary Gygax, inventor of Dungeons and Dragons, had died at 69. And I don't understand, I truly don't, why that of all deaths should affect me the way it does.

Every day, people die; 150,000 of them, in fact. Every now and then I read the obituary of a scientist whose work I admired, and I don't feel like this. I should, of course, but I don't. I remember hearing about the death of Isaac Asimov, and more distantly, the death of Robert Heinlein (though I was 8 at the time) and that didn't affect me like this.

I never knew one single thing about Gary Gygax. I don't know if he had a wife or children. I couldn't guess his political opinions, or what he thought about the future of humanity. He was just a name on the cover of books I read until they disintegrated.

I searched on the Net and just found comments from other people feeling the same way. Stopped in their tracks by this one death, and not understanding why, and trying to come up with an explanation for their own feelings. Why him?

I never even really played D&D all that much. I played a little with David Levitt, my best friend in elementary school - I think it was how we initially met, in fact, though the memory fades into oblivion. I remember my father teaching me to play very simple D&D games, around the same time I was entering kindergarten; I remember being upset that I couldn't cast a Shield spell more than once. But mostly, I just read the rulebooks.

There are people who played D&D with their friends, every week or every day, until late at night, in modules that Gary Gygax designed. I understand why they feel sad. But all I did, mostly, was read the rulebooks to myself. Why do I feel the same way?

Did D&D help teach me that when the world is in danger, you are expected to save it? Did Gary Gygax teach me to form new worlds in my imagination, or to fantasize about more interesting powers than money? Is there something about mentally simulating D&D's rules that taught me how to think? Is it just the sheer amount of total time my imagination spent in those worlds?

I truly don't know. I truly don't know why I feel this way about Gary Gygax's death. I don't know why I feel this compulsion to write about it, to tell someone. I don't think I would have predicted this sadness, if you'd asked me one day before the event.

It tells me something I didn't know before, about how D&D must have helped to shape my childhood, and make me what I am.

And if you think that's amusing, honi soit qui mal y pense.

The online obituaries invariably contain comments along the line of, "Now Gygax gets to explore the Seven Heavens" or "God has new competition as a designer of worlds."

As an atheist, reading these comments just makes it worse, reminds me of the truth.

There are certain ways, in the D&D universe, to permanently destroy a soul - annihilate it, so that it can never be raised or resurrected. You destroy the soul while it's hiding inside an amulet of life protection, or travel to the Outer Planes and destroy the soul in the afterlife of its home plane. Roger M. Wilcox once wrote a story, a rather silly story, that in the midst of silliness included a funeral ceremony for a paladin whose soul had been destroyed:

Josephus took a deep breath. "It is normally at this point in a eulogy where I console the friends and family of the departed by reminding them that although the deceased is no longer with us, he still smiles down on us from Heaven, and that those of us who loved him will see him again when they themselves pass on. Once in a great while, when I agree to perform a funeral for someone who was of a different alignment, I will have to change this part of the eulogy since his soul will have gone to a plane other than Heaven. It always makes it that much more poignant for me to know that neither I nor the mourners in my congregation will see him again when our lives end, but at least we have the reassurance that the departed soul still exists somewhere and that he may be smiling down upon us from Arcadia, or Elysium, or Nirvana, or whichever version of Heaven his alignment allows; and that, the power of his priesthood permitting, he may even be raised back to life or reincarnated in a new body someday."

The cleric's voice quavered. "But this time, I cannot say even this. For I know that Ringman's soul is not in Heaven where it rightfully belongs, nor in any other plane in the multiverse. He will never come back, he cannot see us, and none of us will ever see him again. This funeral is a true goodbye, which he will never hear."

Goodbye, Gary.

Original with comments: Gary Gygax Annihilated at 69

Dissolving the Question

Eliezer Yudkowsky, 08 March 2008 03:17AM

Followup to: How an Algorithm Feels From the Inside, Feel the Meaning, Replace the Symbol with the Substance

"If a tree falls in the forest, but no one hears it, does it make a sound?"

I didn't answer that question. I didn't pick a position, "Yes!" or "No!", and defend it. Instead I went off and deconstructed the human algorithm for processing words, even going so far as to sketch an illustration of a neural network. At the end, I hope, there was no question left - not even the feeling of a question.

Many philosophers - particularly amateur philosophers, and ancient philosophers - share a dangerous instinct: If you give them a question, they try to answer it.

Like, say, "Do we have free will?"

The dangerous instinct of philosophy is to marshal the arguments in favor, and marshal the arguments against, and weigh them up, and publish them in a prestigious journal of philosophy, and so finally conclude: "Yes, we must have free will," or "No, we cannot possibly have free will."

Some philosophers are wise enough to recall the warning that most philosophical disputes are really disputes over the meaning of a word, or confusions generated by using different meanings for the same word in different places. So they try to define very precisely what they mean by "free will", and then ask again, "Do we have free will? Yes or no?"

A philosopher wiser yet, may suspect that the confusion about "free will" shows the notion itself is flawed. So they pursue the Traditional Rationalist course: They argue that "free will" is inherently self-contradictory, or meaningless because it has no testable consequences. And then they publish these devastating observations in a prestigious philosophy journal.

But proving that you are confused may not make you feel any less confused. Proving that a question is meaningless may not help you any more than answering it.

The philosopher's instinct is to find the most defensible position, publish it, and move on. But the "naive" view, the instinctive view, is a fact about human psychology. You can prove that free will is impossible until the Sun goes cold, but this leaves an unexplained fact of cognitive science: If free will doesn't exist, what goes on inside the head of a human being who thinks it does? This is not a rhetorical question!

It is a fact about human psychology that people think they have free will. Finding a more defensible philosophical position doesn't change, or explain, that psychological fact. Philosophy may lead you to reject the concept, but rejecting a concept is not the same as understanding the cognitive algorithms behind it.

You could look at the Standard Dispute over "If a tree falls in the forest, and no one hears it, does it make a sound?", and you could do the Traditional Rationalist thing: Observe that the two don't disagree on any point of anticipated experience, and triumphantly declare the argument pointless. That happens to be correct in this particular case; but, as a question of cognitive science, why did the arguers make that mistake in the first place?

The key idea of the heuristics and biases program is that the mistakes we make, often reveal far more about our underlying cognitive algorithms than our correct answers. So (I asked myself, once upon a time) what kind of mind design corresponds to the mistake of arguing about trees falling in deserted forests?

The cognitive algorithms we use, are the way the world feels. And these cognitive algorithms may not have a one-to-one correspondence with reality - not even macroscopic reality, to say nothing of the true quarks. There can be things in the mind that cut skew to the world.

For example, there can be a dangling unit in the center of a neural network, which does not correspond to any real thing, or any real property of any real thing, existent anywhere in the real world. This dangling unit is often useful as a shortcut in computation, which is why we have them. (Metaphorically speaking. Human neurobiology is surely far more complex.)

This dangling unit feels like an unresolved question, even after every answerable query is answered. No matter how much anyone proves to you that no difference of anticipated experience depends on the question, you're left wondering: "But does the falling tree really make a sound, or not?"

But once you understand in detail how your brain generates the feeling of the question - once you realize that your feeling of an unanswered question, corresponds to an illusory central unit wanting to know whether it should fire, even after all the edge units are clamped at known values - or better yet, you understand the technical workings of Naive Bayes - then you're done. Then there's no lingering feeling of confusion, no vague sense of dissatisfaction.

If there is any lingering feeling of a remaining unanswered question, or of having been fast-talked into something, then this is a sign that you have not dissolved the question. A vague dissatisfaction should be as much warning as a shout. Really dissolving the question doesn't leave anything behind.

A triumphant thundering refutation of free will, an absolutely unarguable proof that free will cannot exist, feels very satisfying - a grand cheer for the home team. And so you may not notice that - as a point of cognitive science - you do not have a full and satisfactory descriptive explanation of how each intuitive sensation arises, point by point.

You may not even want to admit your ignorance, of this point of cognitive science, because that would feel like a score against Your Team. In the midst of smashing all foolish beliefs of free will, it would seem like a concession to the opposing side to concede that you've left anything unexplained.

And so, perhaps, you'll come up with a just-so evolutionary-psychological argument that hunter-gatherers who believed in free will, were more likely to take a positive outlook on life, and so outreproduce other hunter-gatherers - to give one example of a completely bogus explanation. If you say this, you are arguing that the brain generates an illusion of free will - but you are not explaining how. You are trying to dismiss the opposition by deconstructing its motives - but in the story you tell, the illusion of free will is a brute fact. You have not taken the illusion apart to see the wheels and gears.

Imagine that in the Standard Dispute about a tree falling in a deserted forest, you first prove that no difference of anticipation exists, and then go on to hypothesize, "But perhaps people who said that arguments were meaningless were viewed as having conceded, and so lost social status, so now we have an instinct to argue about the meanings of words." That's arguing that or explaining why a confusion exists. Now look at the neural network structure in Feel the Meaning. That's explaining how, disassembling the confusion into smaller pieces which are not themselves confusing. See the difference?

Coming up with good hypotheses about cognitive algorithms (or even hypotheses that hold together for half a second) is a good deal harder than just refuting a philosophical confusion. Indeed, it is an entirely different art. Bear this in mind, and you should feel less embarrassed to say, "I know that what you say can't possibly be true, and I can prove it. But I cannot write out a flowchart which shows how your brain makes the mistake, so I'm not done yet, and will continue investigating."

I say all this, because it sometimes seems to me that at least 20% of the real-world effectiveness of a skilled rationalist comes from not stopping too early. If you keep asking questions, you'll get to your destination eventually. If you decide too early that you've found an answer, you won't.

The challenge, above all, is to notice when you are confused - even if it just feels like a little tiny bit of confusion - and even if there's someone standing across from you, insisting that humans have free will, and smirking at you, and the fact that you don't know exactly how the cognitive algorithms work, has nothing to do with the searing folly of their position...

But when you can lay out the cognitive algorithm in sufficient detail that you can walk through the thought process, step by step, and describe how each intuitive perception arises - decompose the confusion into smaller pieces not themselves confusing - then you're done.

So be warned that you may believe you're done, when all you have is a mere triumphant refutation of a mistake.

But when you're really done, you'll know you're done. Dissolving the question is an unmistakable feeling - once you experience it, and, having experienced it, resolve not to be fooled again. Those who dream do not know they dream, but when you wake you know you are awake.

Which is to say: When you're done, you'll know you're done, but unfortunately the reverse implication does not hold.

So here's your homework problem: What kind of cognitive algorithm, as felt from the inside, would generate the observed debate about "free will"?

Your assignment is not to argue about whether people have free will, or not.

Your assignment is not to argue that free will is compatible with determinism, or not.

Your assignment is not to argue that the question is ill-posed, or that the concept is self-contradictory, or that it has no testable consequences.

You are not asked to invent an evolutionary explanation of how people who believed in free will would have reproduced; nor an account of how the concept of free will seems suspiciously congruent with bias X. Such are mere attempts to explain why people believe in "free will", not explain how.

Your homework assignment is to write a stack trace of the internal algorithms of the human mind as they produce the intuitions that power the whole damn philosophical argument.

This is one of the first real challenges I tried as an aspiring rationalist, once upon a time. One of the easier conundrums, relatively speaking. May it serve you likewise.

Sequence: Reductionism

Wrong Questions

Referenced by: Wrong Questions • Righting a Wrong Question • Reductionism • Explaining vs. Explaining Away • Zombies! Zombies? • The Conscious Sorites Paradox • Timeless Physics • Timeless Identity • Thou Art Physics • Possibility and Could-ness • Heading Toward Morality • Surface Analogies and Deep Causes • The Bedrock of Fairness • Setting Up Metaethics • The Meaning of Right • A Prodigy of Refutation • Nonperson Predicates

Original with comments: Dissolving the Question

Wrong Questions

Eliezer Yudkowsky, 08 March 2008 05:11PM

Followup to: Dissolving the Question, Mysterious Answers to Mysterious Questions

Where the mind cuts against reality's grain, it generates wrong questions - questions that cannot possibly be answered on their own terms, but only dissolved by understanding the cognitive algorithm that generates the perception of a question.

One good cue that you're dealing with a "wrong question" is when you cannot even imagine any concrete, specific state of how-the-world-is that would answer the question. When it doesn't even seem possible to answer the question.

Take the Standard Definitional Dispute, for example, about the tree falling in a deserted forest. Is there any way-the-world-could-be - any state of affairs - that corresponds to the word "sound" really meaning only acoustic vibrations, or really meaning only auditory experiences?

("Why, yes," says the one, "it is the state of affairs where 'sound' means acoustic vibrations." So Taboo the word 'means', and 'represents', and all similar synonyms, and describe again: How can the world be, what state of affairs, would make one side right, and the other side wrong?)

Or if that seems too easy, take free will: What concrete state of affairs, whether in deterministic physics, or in physics with a dice-rolling random component, could ever correspond to having free will?

And if that seems too easy, then ask "Why does anything exist at all?", and then tell me what a satisfactory answer to that question would even look like.

And no, I don't know the answer to that last one. But I can guess one thing, based on my previous experience with unanswerable questions. The answer will not consist of some grand triumphant First Cause. The question will go away as a result of some insight into how my mental algorithms run skew to reality, after which I will understand how the question itself was wrong from the beginning - how the question itself assumed the fallacy, contained the skew.

Mystery exists in the mind, not in reality. If I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon itself. All the more so, if it seems like no possible answer can exist: Confusion exists in the map, not in the territory. Unanswerable questions do not mark places where magic enters the universe. They mark places where your mind runs skew to reality.

Such questions must be dissolved. Bad things happen when you try to answer them. It inevitably generates the worst sort of Mysterious Answer to a Mysterious Question: The one where you come up with seemingly strong arguments for your Mysterious Answer, but the "answer" doesn't let you make any new predictions even in retrospect, and the phenomenon still possesses the same sacred inexplicability that it had at the start.

I could guess, for example, that the answer to the puzzle of the First Cause is that nothing does exist - that the whole concept of "existence" is bogus. But if you sincerely believed that, would you be any less confused? Me neither.

But the wonderful thing about unanswerable questions is that they are always solvable, at least in my experience. What went through Queen Elizabeth I's mind, first thing in the morning, as she woke up on her fortieth birthday? As I can easily imagine answers to this question, I can readily see that I may never be able to actually answer it, the true information having been lost in time.

On the other hand, "Why does anything exist at all?" seems so absolutely impossible that I can infer that I am just confused, one way or another, and the truth probably isn't all that complicated in an absolute sense, and once the confusion goes away I'll be able to see it.

This may seem counterintuitive if you've never solved an unanswerable question, but I assure you that it is how these things work.

Coming tomorrow: A simple trick for handling "wrong questions".

Sequence: Reductionism

Dissolving the Question

Righting a Wrong Question

Referenced by: Righting a Wrong Question • To Spread Science, Keep It Secret • Angry Atoms • Zombies! Zombies? • Zombie Responses • Timeless Physics • Thou Art Physics • Heading Toward Morality • Where Recursive Justification Hits Bottom • Setting Up Metaethics • Moral Error and Moral Disagreement • Three Fallacies of Teleology • Nonperson Predicates • Nonsentient Optimizers • Changing Emotions

Original with comments: Wrong Questions

Righting a Wrong Question

Eliezer Yudkowsky, 09 March 2008 01:00PM

Followup to: How an Algorithm Feels from the Inside, Dissolving the Question, Wrong Questions

When you are faced with an unanswerable question - a question to which it seems impossible to even imagine an answer - there is a simple trick which can turn the question solvable.

Compare:

- "Why do I have free will?"

- "Why do I think I have free will?"

The nice thing about the second question is that it is guaranteed to have a real answer, whether or not there is any such thing as free will. Asking "Why do I have free will?" or "Do I have free will?" sends you off thinking about tiny details of the laws of physics, so distant from the macroscopic level that you couldn't begin to see them with the naked eye. And you're asking "Why is X the case?" where X may not be coherent, let alone the case.

"Why do I think I have free will?", in contrast, is guaranteed answerable. You do, in fact, believe you have free will. This belief seems far more solid and graspable than the ephemerality of free will. And there is, in fact, some nice solid chain of cognitive cause and effect leading up to this belief.

If you've already outgrown free will, choose one of these substitutes:

- "Why does time move forward instead of backward?" versus "Why do I think time moves forward instead of backward?"

- "Why was I born as myself rather than someone else?" versus "Why do I think I was born as myself rather than someone else?"

- "Why am I conscious?" versus "Why do I think I'm conscious?"

- "Why does reality exist?" versus "Why do I think reality exists?"

The beauty of this method is that it works whether or not the question is confused. As I type this, I am wearing socks. I could ask "Why am I wearing socks?" or "Why do I believe I'm wearing socks?" Let's say I ask the second question. Tracing back the chain of causality, I find:

- I believe I'm wearing socks, because I can see socks on my feet.

- I see socks on my feet, because my retina is sending sock signals to my visual cortex.

- My retina is sending sock signals, because sock-shaped light is impinging on my retina.

- Sock-shaped light impinges on my retina, because it reflects from the socks I'm wearing.

- It reflects from the socks I'm wearing, because I'm wearing socks.

- I'm wearing socks because I put them on.

- I put socks on because I believed that otherwise my feet would get cold.

- &c.

Tracing back the chain of causality, step by step, I discover that my belief that I'm wearing socks is fully explained by the fact that I'm wearing socks. This is right and proper, as you cannot gain information about something without interacting with it.

On the other hand, if I see a mirage of a lake in a desert, the correct causal explanation of my vision does not involve the fact of any actual lake in the desert. In this case, my belief in the lake is not just explained, but explained away.

But either way, the belief itself is a real phenomenon taking place in the real universe - psychological events are events - and its causal history can be traced back.

"Why is there a lake in the middle of the desert?" may fail if there is no lake to be explained. But "Why do I perceive a lake in the middle of the desert?" always has a causal explanation, one way or the other.

Perhaps someone will see an opportunity to be clever, and say: "Okay. I believe in free will because I have free will. There, I'm done." Of course it's not that easy.

My perception of socks on my feet, is an event in the visual cortex. The workings of the visual cortex can be investigated by cognitive science, should they be confusing.

My retina receiving light is not a mystical sensing procedure, a magical sock detector that lights in the presence of socks for no explicable reason; there are mechanisms that can be understood in terms of biology. The photons entering the retina can be understood in terms of optics. The shoe's surface reflectance can be understood in terms of electromagnetism and chemistry. My feet getting cold can be understood in terms of thermodynamics.

So it's not as easy as saying, "I believe I have free will because I have it - there, I'm done!" You have to be able to break the causal chain into smaller steps, and explain the steps in terms of elements not themselves confusing.

The mechanical interaction of my retina with my socks is quite clear, and can be described in terms of non-confusing components like photons and electrons. Where's the free-will-sensor in your brain, and how does it detect the presence or absence of free will? How does the sensor interact with the sensed event, and what are the mechanical details of the interaction?

If your belief does derive from valid observation of a real phenomenon, we will eventually reach that fact, if we start tracing the causal chain backward from your belief.

If what you are really seeing is your own confusion, tracing back the chain of causality will find an algorithm that runs skew to reality.

Either way, the question is guaranteed to have an answer. You even have a nice, concrete place to begin tracing - your belief, sitting there solidly in your mind.

Cognitive science may not seem so lofty and glorious as metaphysics. But at least questions of cognitive science are solvable. Finding an answer may not be easy, but at least an answer exists.

Oh, and also: the idea that cognitive science is not so lofty and glorious as metaphysics is simply wrong. Some readers are beginning to notice this, I hope.

Sequence: Reductionism

Wrong Questions

Mind Projection Fallacy

Referenced by: Explaining vs. Explaining Away • Possibility and Could-ness • The Ultimate Source • Passing the Recursive Buck • Setting Up Metaethics • The Meaning of Right • "Arbitrary" • Nonperson Predicates • Continuous Improvement

Original with comments: Righting a Wrong Question

Mind Projection Fallacy

Eliezer Yudkowsky, 11 March 2008 12:29AM

Followup to: How an Algorithm Feels From Inside

In the dawn days of science fiction, alien invaders would occasionally kidnap a girl in a torn dress and carry her off for intended ravishing, as lovingly depicted on many ancient magazine covers. Oddly enough, the aliens never go after men in torn shirts.

Would a non-humanoid alien, with a different evolutionary history and evolutionary psychology, sexually desire a human female? It seems rather unlikely. To put it mildly.

People don't make mistakes like that by deliberately reasoning: "All possible minds are likely to be wired pretty much the same way, therefore a bug-eyed monster will find human females attractive." Probably the artist did not even think to ask whether an alien perceives human females as attractive. Instead, a human female in a torn dress is sexy - inherently so, as an intrinsic property.

They who went astray did not think about the alien's evolutionary history; they focused on the woman's torn dress. If the dress were not torn, the woman would be less sexy; the alien monster doesn't enter into it.

Apparently we instinctively represent Sexiness as a direct attribute of the Woman object, Woman.sexiness, like Woman.height or Woman.weight.

If your brain uses that data structure, or something metaphorically similar to it, then from the inside it feels like sexiness is an inherent property of the woman, not a property of the alien looking at the woman. Since the woman is attractive, the alien monster will be attracted to her - isn't that logical?

E. T. Jaynes used the term Mind Projection Fallacy to denote the error of projecting your own mind's properties into the external world. Jaynes, as a late grand master of the Bayesian Conspiracy, was most concerned with the mistreatment of probabilities as inherent properties of objects, rather than states of partial knowledge in some particular mind. More about this shortly.

But the Mind Projection Fallacy generalizes as an error. It is in the argument over the real meaning of the word sound, and in the magazine cover of the monster carrying off a woman in the torn dress, and Kant's declaration that space by its very nature is flat, and Hume's definition of a priori ideas as those "discoverable by the mere operation of thought, without dependence on what is anywhere existent in the universe"...

(Incidentally, I once read an SF story about a human male who entered into a sexual relationship with a sentient alien plant of appropriately squishy fronds; discovered that it was an androecious (male) plant; agonized about this for a bit; and finally decided that it didn't really matter at that point. And in Foglio and Pollotta's Illegal Aliens, the humans land on a planet inhabited by sentient insects, and see a movie advertisement showing a human carrying off a bug in a delicate chiffon dress. Just thought I'd mention that.)

Sequence: Reductionism

Righting a Wrong Question

Probability is in the Mind

Referenced by: Probability is in the Mind • The Quotation is not the Referent • Qualitatively Confused • Reductionism • Explaining vs. Explaining Away • Joy in the Merely Real • Hand vs. Fingers • Quantum Explanations • Configurations and Amplitude • The So-Called Heisenberg Uncertainty Principle • Decoherent Essences • Bell's Theorem: No EPR "Reality" • Mach's Principle: Anti-Epiphenomenal Physics • Relative Configuration Space • Timeless Physics • Thou Art Physics • Possibility and Could-ness • Grasping Slippery Things • Heading Toward Morality • No Universally Compelling Arguments • 2-Place and 1-Place Words • Setting Up Metaethics • The Meaning of Right • Humans in Funny Suits • Anthropomorphic Optimism • Inseparably Right; or, Joy in the Merely Good • Magical Categories • Three Fallacies of Teleology • Dreams of Friendliness • Excluding the Supernatural • Which Parts Are "Me"? • Chaotic Inversion • Artificial Mysterious Intelligence • Value is Fragile • Raising the Sanity Waterline

Original with comments: Mind Projection Fallacy

Probability is in the Mind

Eliezer Yudkowsky, 12 March 2008 04:08AM

Followup to: The Mind Projection Fallacy

Yesterday I spoke of the Mind Projection Fallacy, giving the example of the alien monster who carries off a girl in a torn dress for intended ravishing - a mistake which I imputed to the artist's tendency to think that a woman's sexiness is a property of the woman herself, woman.sexiness, rather than something that exists in the mind of an observer, and probably wouldn't exist in an alien mind.

The term "Mind Projection Fallacy" was coined by the late great Bayesian Master, E. T. Jaynes, as part of his long and hard-fought battle against the accursèd frequentists. Jaynes was of the opinion that probabilities were in the mind, not in the environment - that probabilities express ignorance, states of partial information; and if I am ignorant of a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon.

I cannot do justice to this ancient war in a few words - but the classic example of the argument runs thus:

You have a coin.
The coin is biased.
You don't know which way it's biased or how much it's biased. Someone just told you, "The coin is biased" and that's all they said.
This is all the information you have, and the only information you have.

You draw the coin forth, flip it, and slap it down.

Now - before you remove your hand and look at the result - are you willing to say that you assign a 0.5 probability to the coin having come up heads?

The frequentist says, "No. Saying 'probability 0.5' means that the coin has an inherent propensity to come up heads as often as tails, so that if we flipped the coin infinitely many times, the ratio of heads to tails would approach 1:1. But we know that the coin is biased, so it can have any probability of coming up heads except 0.5."

The Bayesian says, "Uncertainty exists in the map, not in the territory. In the real world, the coin has either come up heads, or come up tails. Any talk of 'probability' must refer to the information that I have about the coin - my state of partial ignorance and partial knowledge - not just the coin itself. Furthermore, I have all sorts of theorems showing that if I don't treat my partial knowledge a certain way, I'll make stupid bets. If I've got to plan, I'll plan for a 50/50 state of uncertainty, where I don't weigh outcomes conditional on heads any more heavily in my mind than outcomes conditional on tails. You can call that number whatever you like, but it has to obey the probability laws on pain of stupidity. So I don't have the slightest hesitation about calling my outcome-weighting a probability."

I side with the Bayesians. You may have noticed that about me.

Even before a fair coin is tossed, the notion that it has an inherent 50% probability of coming up heads may be just plain wrong. Maybe you're holding the coin in such a way that it's just about guaranteed to come up heads, or tails, given the force at which you flip it, and the air currents around you. But, if you don't know which way the coin is biased on this one occasion, so what?

I believe there was a lawsuit where someone alleged that the draft lottery was unfair, because the slips with names on them were not being mixed thoroughly enough; and the judge replied, "To whom is it unfair?"

To make the coinflip experiment repeatable, as frequentists are wont to demand, we could build an automated coinflipper, and verify that the results were 50% heads and 50% tails. But maybe a robot with extra-sensitive eyes and a good grasp of physics, watching the autoflipper prepare to flip, could predict the coin's fall in advance - not with certainty, but with 90% accuracy. Then what would the real probability be?

There is no "real probability". The robot has one state of partial information. You have a different state of partial information. The coin itself has no mind, and doesn't assign a probability to anything; it just flips into the air, rotates a few times, bounces off some air molecules, and lands either heads or tails.

So that is the Bayesian view of things, and I would now like to point out a couple of classic brainteasers that derive their brain-teasing ability from the tendency to think of probabilities as inherent properties of objects.

Let's take the old classic: You meet a mathematician on the street, and she happens to mention that she has given birth to two children on two separate occasions. You ask: "Is at least one of your children a boy?" The mathematician says, "Yes, he is."

What is the probability that she has two boys? If you assume that the prior probability of a child being a boy is 1/2, then the probability that she has two boys, on the information given, is 1/3. The prior probabilities were: 1/4 two boys, 1/2 one boy one girl, 1/4 two girls. The mathematician's "Yes" response has probability ~1 in the first two cases, and probability ~0 in the third. Renormalizing leaves us with a 1/3 probability of two boys, and a 2/3 probability of one boy one girl.

But suppose that instead you had asked, "Is your eldest child a boy?" and the mathematician had answered "Yes." Then the probability of the mathematician having two boys would be 1/2. Since the eldest child is a boy, and the younger child can be anything it pleases.

Likewise if you'd asked "Is your youngest child a boy?" The probability of their being both boys would, again, be 1/2.

Now, if at least one child is a boy, it must be either the oldest child who is a boy, or the youngest child who is a boy. So how can the answer in the first case be different from the answer in the latter two?

Or here's a very similar problem: Let's say I have four cards, the ace of hearts, the ace of spades, the two of hearts, and the two of spades. I draw two cards at random. You ask me, "Are you holding at least one ace?" and I reply "Yes." What is the probability that I am holding a pair of aces? It is 1/5. There are six possible combinations of two cards, with equal prior probability, and you have just eliminated the possibility that I am holding a pair of twos. Of the five remaining combinations, only one combination is a pair of aces. So 1/5.

Now suppose that instead you asked me, "Are you holding the ace of spades?" If I reply "Yes", the probability that the other card is the ace of hearts is 1/3. (You know I'm holding the ace of spades, and there are three possibilities for the other card, only one of which is the ace of hearts.) Likewise, if you ask me "Are you holding the ace of hearts?" and I reply "Yes", the probability I'm holding a pair of aces is 1/3.

But then how can it be that if you ask me, "Are you holding at least one ace?" and I say "Yes", the probability I have a pair is 1/5? Either I must be holding the ace of spades or the ace of hearts, as you know; and either way, the probability that I'm holding a pair of aces is 1/3.

How can this be? Have I miscalculated one or more of these probabilities?

If you want to figure it out for yourself, do so now, because I'm about to reveal...

That all stated calculations are correct.

As for the paradox, there isn't one. The appearance of paradox comes from thinking that the probabilities must be properties of the cards themselves. The ace I'm holding has to be either hearts or spades; but that doesn't mean that your knowledge about my cards must be the same as if you knew I was holding hearts, or knew I was holding spades.

It may help to think of Bayes's Theorem:

P(H|E) = P(E|H)P(H) / P(E)

That last term, where you divide by P(E), is the part where you throw out all the possibilities that have been eliminated, and renormalize your probabilities over what remains.

Now let's say that you ask me, "Are you holding at least one ace?" Before I answer, your probability that I say "Yes" should be 5/6.

But if you ask me "Are you holding the ace of spades?", your prior probability that I say "Yes" is just 1/2.

So right away you can see that you're learning something very different in the two cases. You're going to be eliminating some different possibilities, and renormalizing using a different P(E). If you learn two different items of evidence, you shouldn't be surprised at ending up in two different states of partial information.

Similarly, if I ask the mathematician, "Is at least one of your two children a boy?" I expect to hear "Yes" with probability 3/4, but if I ask "Is your eldest child a boy?" I expect to hear "Yes" with probability 1/2. So it shouldn't be surprising that I end up in a different state of partial knowledge, depending on which of the two questions I ask.

The only reason for seeing a "paradox" is thinking as though the probability of holding a pair of aces is a property of cards that have at least one ace, or a property of cards that happen to contain the ace of spades. In which case, it would be paradoxical for card-sets containing at least one ace to have an inherent pair-probability of 1/5, while card-sets containing the ace of spades had an inherent pair-probability of 1/3, and card-sets containing the ace of hearts had an inherent pair-probability of 1/3.

Similarly, if you think a 1/3 probability of being both boys is an inherent property of child-sets that include at least one boy, then that is not consistent with child-sets of which the eldest is male having an inherent probability of 1/2 of being both boys, and child-sets of which the youngest is male having an inherent 1/2 probability of being both boys. It would be like saying, "All green apples weigh a pound, and all red apples weigh a pound, and all apples that are green or red weigh half a pound."

That's what happens when you start thinking as if probabilities are in things, rather than probabilities being states of partial information about things.

Probabilities express uncertainty, and it is only agents who can be uncertain. A blank map does not correspond to a blank territory. Ignorance is in the mind.

Sequence: Reductionism

Mind Projection Fallacy

The Quotation is not the Referent

Sequence: Quantum Physics

Reductionism

Referenced by: The Quotation is not the Referent • Qualitatively Confused • Joy in the Merely Real • Configurations and Amplitude • Bell's Theorem: No EPR "Reality" • Quantum Non-Realism • Collapse Postulates • The Quantum Physics Sequence • Possibility and Could-ness • Heading Toward Morality • Probability is Subjectively Objective • Setting Up Metaethics • Raising the Sanity Waterline • Frequentist Statistics are Frequently Subjective

Original with comments: Probability is in the Mind

The Quotation is not the Referent

Eliezer Yudkowsky, 13 March 2008 12:53AM

Followup to: The Mind Projection Fallacy, Probability is in the Mind

In classical logic, the operational definition of identity is that whenever 'A=B' is a theorem, you can substitute 'A' for 'B' in any theorem where B appears. For example, if (2 + 2) = 4 is a theorem, and ((2 + 2) + 3) = 7 is a theorem, then (4 + 3) = 7 is a theorem.

This leads to a problem which is usually phrased in the following terms: The morning star and the evening star happen to be the same object, the planet Venus. Suppose John knows that the morning star and evening star are the same object. Mary, however, believes that the morning star is the god Lucifer, but the evening star is the god Venus. John believes Mary believes that the morning star is Lucifer. Must John therefore (by substitution) believe that Mary believes that the evening star is Lucifer?

Or here's an even simpler version of the problem. 2 + 2 = 4 is true; it is a theorem that (((2 + 2) = 4) = TRUE). Fermat's Last Theorem is also true. So: I believe 2 + 2 = 4 => I believe TRUE => I believe Fermat's Last Theorem.

Yes, I know this seems obviously wrong. But imagine someone writing a logical reasoning program using the principle "equal terms can always be substituted", and this happening to them. Now imagine them writing a paper about how to prevent it from happening. Now imagine someone else disagreeing with their solution. The argument is still going on.

P'rsnally, I would say that John is committing a type error, like trying to subtract 5 grams from 20 meters. "The morning star" is not the same type as the morning star, let alone the same thing. Beliefs are not planets.

morning star = evening star
"morning star" ≠ "evening star"

The problem, in my view, stems from the failure to enforce the type distinction between beliefs and things. The original error was writing an AI that stores its beliefs about Mary's beliefs about "the morning star" using the same representation as in its beliefs about the morning star.

If Mary believes the "morning star" is Lucifer, that doesn't mean Mary believes the "evening star" is Lucifer, because "morning star" ≠ "evening star". The whole paradox stems from the failure to use quote marks in appropriate places.

You may recall that this is not the first time I've talked about enforcing type discipline - the last time was when I spoke about the error of confusing expected utilities with utilities. It is immensely helpful, when one is first learning physics, to learn to keep track of one's units - it may seem like a bother to keep writing down 'cm' and 'kg' and so on, until you notice that (a) your answer seems to be the wrong order of magnitude and (b) it is expressed in seconds per square gram.

Similarly, beliefs are different things than planets. If we're talking about human beliefs, at least, then: Beliefs live in brains, planets live in space. Beliefs weigh a few micrograms, planets weigh a lot more. Planets are larger than beliefs... but you get the idea.

Merely putting quote marks around "morning star" seems insufficient to prevent people from confusing it with the morning star, due to the visual similarity of the text. So perhaps a better way to enforce type discipline would be with a visibly different encoding:

morning star = evening star
13.15.18.14.9.14.7.0.19.20.1.18 ≠ 5.22.5.14.9.14.7.0.19.20.1.18

Studying mathematical logic may also help you learn to distinguish the quote and the referent. In mathematical logic, |- P (P is a theorem) and |- []'P' (it is provable that there exists an encoded proof of the encoded sentence P in some encoded proof system) are very distinct propositions. If you drop a level of quotation in mathematical logic, it's like dropping a metric unit in physics - you can derive visibly ridiculous results, like "The speed of light is 299,792,458 meters long."

Alfred Tarski once tried to define the meaning of 'true' using an infinite family of sentences:

("Snow is white" is true) if and only (snow is white)
("Weasels are green" is true) if and only if (weasels are green)
...

When sentences like these start seeming meaningful, you'll know that you've started to distinguish between encoded sentences and states of the outside world.

Similarly, the notion of truth is quite different from the notion of reality. Saying "true" compares a belief to reality. Reality itself does not need to be compared to any beliefs in order to be real. Remember this the next time someone claims that nothing is true.

Sequence: Reductionism

Probability is in the Mind

Qualitatively Confused

Referenced by: Qualitatively Confused • Heat vs. Motion • Probability is Subjectively Objective • Morality as Fixed Computation • Trying to Try

Original with comments: The Quotation is not the Referent

Penguicon & Blook

Eliezer Yudkowsky, 13 March 2008 05:32PM

One million cumulative daily visits! Woot n' stuff. Also we're in the top 5,000 of all blogs on Technorati, and one of the top 10 econblogs by Technorati rank.

Seems like a good time to mention that I'll be appearing at Penguicon, a combination open-source/science-fiction convention in Troy, MI, Apr 18-20, as a Nifty. I'll be doing an intro to Bayesian reasoning that you probably don't need if you're reading this, possibly a panel on the Virtues of a Rationalist, some stuff on human intelligence upgrades, and definitely "The Ethics of Ending the World" with Aaron Diaz (Dresden Codak).

After the jump, you can see some proposed cover art for the blook.

For the benefit of the humor impaired: Yes, this is a joke. Erin, my girlfriend, Photoshopped this when she heard I was planning to do a book.

This is all taking longer than I expected - as expected - but I do think I'm getting there.

My current serious strategy for the blook is as follows:

1. Finish all the important serial material on rationality - the posts that have to be done in month-long sequences. That's probably at least another two months.

2. Maybe spend another month or two doing large transhumanist sequences, either on the Singularity Institute blog (currently fairly defunct) or here on Overcoming Bias if the readers really want that. My self-imposed deadline here is August 2008.

3. Switch to writing shorter posts on topics that can be considered independently given the already-written background material - maybe cut back to a Sat-Sun-Tue-Thu schedule. Don't worry, this won't happen anytime soon.

4. On days when I don't post: spend my time compiling collections of related Overcoming Bias posts into medium-sized ebooks of 50 pages / 20,000 words or thereabouts, broken up into short blog-post-sized sections for easy reading. (This is good because it can be done incrementally, and I tend to bog down when I try to do anything book-sized all at once.) Leave a comment if you have any suggestions on ebook-writing software or ways to get the book design done cheaply.

5. Publish the ebooks incrementally on Wowio, as a compromise between "information wants to be free" and "authors want to eat". (Email me if you happen to work at/with Wowio, because I'm interested in testing the waters on this sometime soon.) Publish the ebooks with a cheap nominal charge for readers outside the US, since Wowio doesn't work outside the US yet.

6. Produce a giant expensive 500-page hardcover dead-tree compendium of all the ebooks at Lulu or somewhere similar.

7. Once all the fully detailed material exists somewhere and I can summarize it with a clean conscience: pick the easiest and most favorably received topics for a short, popular book; produce an outline and a couple of starting chapters; and begin looking for an agent who believes that the book can be a New York Times bestseller, to find a publisher who believes that the book can be a bestseller and who'll invest a corresponding amount marketing it.

If you've got more experience in the publishing industry and you see some reason that any of this won't work, i.e., "No one will talk to you if you've ever done anything with Wowio or Lulu" or "Today's readers don't want short popular books, they want 500-page tomes" or something like that, please email me or comment.

Original with comments: Penguicon & Blook

Qualitatively Confused

Eliezer Yudkowsky, 14 March 2008 05:01PM

Followup to: Probability is in the Mind, The Quotation is not the Referent

I suggest that a primary cause of confusion about the distinction between "belief", "truth", and "reality" is qualitative thinking about beliefs.

Consider the archetypal postmodernist attempt to be clever:

"The Sun goes around the Earth" is true for Hunga Huntergatherer, but "The Earth goes around the Sun" is true for Amara Astronomer! Different societies have different truths!

No, different societies have different beliefs. Belief is of a different type than truth; it's like comparing apples and probabilities.

Ah, but there's no difference between the way you use the word 'belief' and the way you use the word 'truth'! Whether you say, "I believe 'snow is white'", or you say, "'Snow is white' is true", you're expressing exactly the same opinion.

No, these sentences mean quite different things, which is how I can conceive of the possibility that my beliefs are false.

Oh, you claim to conceive it, but you never believe it. As Wittgenstein said, "If there were a verb meaning 'to believe falsely', it would not have any significant first person, present indicative."

And that's what I mean by putting my finger on qualitative reasoning as the source of the problem. The dichotomy between belief and disbelief, being binary, is confusingly similar to the dichotomy between truth and untruth.

So let's use quantitative reasoning instead. Suppose that I assign a 70% probability to the proposition that snow is white. It follows that I think there's around a 70% chance that the sentence "snow is white" will turn out to be true. If the sentence "snow is white" is true, is my 70% probability assignment to the proposition, also "true"? Well, it's more true than it would have been if I'd assigned 60% probability, but not so true as if I'd assigned 80% probability.

When talking about the correspondence between a probability assignment and reality, a better word than "truth" would be "accuracy". "Accuracy" sounds more quantitative, like an archer shooting an arrow: how close did your probability assignment strike to the center of the target?

To make a long story short, it turns out that there's a very natural way of scoring the accuracy of a probability assignment, as compared to reality: just take the logarithm of the probability assigned to the real state of affairs.

So if snow is white, my belief "70%: 'snow is white'" will score -0.51 bits: Log_2(0.7) = -0.51.

But what if snow is not white, as I have conceded a 30% probability is the case? If "snow is white" is false, my belief "30% probability: 'snow is not white'" will score -1.73 bits. Note that -1.73 < -0.51, so I have done worse.

About how accurate do I think my own beliefs are? Well, my expectation over the score is 70% * -0.51 + 30% * -1.73 = -0.88 bits. If snow is white, then my beliefs will be more accurate than I expected; and if snow is not white, my beliefs will be less accurate than I expected; but in neither case will my belief be exactly as accurate as I expected on average.

All this should not be confused with the statement "I assign 70% credence that 'snow is white'." I may well believe that proposition with probability ~1 - be quite certain that this is in fact my belief. If so I'll expect my meta-belief "~1: 'I assign 70% credence that "snow is white"'" to score ~0 bits of accuracy, which is as good as it gets.

Just because I am uncertain about snow, does not mean I am uncertain about my quoted probabilistic beliefs. Snow is out there, my beliefs are inside me. I may be a great deal less uncertain about how uncertain I am about snow, than I am uncertain about snow. (Though beliefs about beliefs are not always accurate.)

Contrast this probabilistic situation to the qualitative reasoning where I just believe that snow is white, and believe that I believe that snow is white, and believe "'snow is white' is true", and believe "my belief '"snow is white" is true' is correct", etc. Since all the quantities involved are 1, it's easy to mix them up.

Yet the nice distinctions of quantitative reasoning will be short-circuited if you start thinking "'"snow is white" with 70% probability' is true", which is a type error. It is a true fact about you, that you believe "70% probability: 'snow is white'"; but that does not mean the probability assignment itself can possibly be "true". The belief scores either -0.51 bits or -1.73 bits of accuracy, depending on the actual state of reality.

The cognoscenti will recognize "'"snow is white" with 70% probability' is true" as the mistake of thinking that probabilities are inherent properties of things.

From the inside, our beliefs about the world look like the world, and our beliefs about our beliefs look like beliefs. When you see the world, you are experiencing a belief from the inside. When you notice yourself believing something, you are experiencing a belief about belief from the inside. So if your internal representations of belief, and belief about belief, are dissimilar, then you are less likely to mix them up and commit the Mind Projection Fallacy - I hope.

When you think in probabilities, your beliefs, and your beliefs about your beliefs, will hopefully not be represented similarly enough that you mix up belief and accuracy, or mix up accuracy and reality. When you think in probabilities about the world, your beliefs will be represented with probabilities ∈ (0, 1). Unlike the truth-values of propositions, which are in {true, false}. As for the accuracy of your probabilistic belief, you can represent that in the range (-∞, 0). Your probabilities about your beliefs will typically be extreme. And things themselves - why, they're just red, or blue, or weighing 20 pounds, or whatever.

Thus we will be less likely, perhaps, to mix up the map with the territory.

This type distinction may also help us remember that uncertainty is a state of mind. A coin is not inherently 50% uncertain of which way it will land. The coin is not a belief processor, and does not have partial information about itself. In qualitative reasoning you can create a belief that corresponds very straightforwardly to the coin, like "The coin will land heads". This belief will be true or false depending on the coin, and there will be a transparent implication from the truth or falsity of the belief, to the facing side of the coin.

But even under qualitative reasoning, to say that the coin itself is "true" or "false" would be a severe type error. The coin is not a belief, it is a coin. The territory is not the map.

If a coin cannot be true or false, how much less can it assign a 50% probability to itself?

Sequence: Reductionism

The Quotation is not the Referent

Reductionism

Referenced by: Reductionism • Hand vs. Fingers • Reductive Reference • Zombies! Zombies? • Zombie Responses • Timeless Physics

Original with comments: Qualitatively Confused

Reductionism

Eliezer Yudkowsky, 16 March 2008 06:26AM

Followup to: How An Algorithm Feels From Inside, Mind Projection Fallacy

Almost one year ago, in April 2007, Matthew C submitted the following suggestion for an Overcoming Bias topic:

"How and why the current reigning philosophical hegemon (reductionistic materialism) is obviously correct [...], while the reigning philosophical viewpoints of all past societies and civilizations are obviously suspect -"

I remember this, because I looked at the request and deemed it legitimate, but I knew I couldn't do that topic until I'd started on the Mind Projection Fallacy sequence, which wouldn't be for a while...

But now it's time to begin addressing this question. And while I haven't yet come to the "materialism" issue, we can now start on "reductionism".

First, let it be said that I do indeed hold that "reductionism", according to the meaning I will give for that word, is obviously correct; and to perdition with any past civilizations that disagreed.

This seems like a strong statement, at least the first part of it. General Relativity seems well-supported, yet who knows but that some future physicist may overturn it?

On the other hand, we are never going back to Newtonian mechanics. The ratchet of science turns, but it does not turn in reverse. There are cases in scientific history where a theory suffered a wound or two, and then bounced back; but when a theory takes as many arrows through the chest as Newtonian mechanics, it stays dead.

"To hell with what past civilizations thought" seems safe enough, when past civilizations believed in something that has been falsified to the trash heap of history.

And reductionism is not so much a positive hypothesis, as the absence of belief - in particular, disbelief in a form of the Mind Projection Fallacy.

I once met a fellow who claimed that he had experience as a Navy gunner, and he said, "When you fire artillery shells, you've got to compute the trajectories using Newtonian mechanics. If you compute the trajectories using relativity, you'll get the wrong answer."

And I, and another person who was present, said flatly, "No." I added, "You might not be able to compute the trajectories fast enough to get the answers in time - maybe that's what you mean? But the relativistic answer will always be more accurate than the Newtonian one."

"No," he said, "I mean that relativity will give you the wrong answer, because things moving at the speed of artillery shells are governed by Newtonian mechanics, not relativity."

"If that were really true," I replied, "you could publish it in a physics journal and collect your Nobel Prize."

Standard physics uses the same fundamental theory to describe the flight of a Boeing 747 airplane, and collisions in the Relativistic Heavy Ion Collider. Nuclei and airplanes alike, according to our understanding, are obeying special relativity, quantum mechanics, and chromodynamics.

But we use entirely different models to understand the aerodynamics of a 747 and a collision between gold nuclei in the RHIC. A computer modeling the aerodynamics of a 747 may not contain a single token, a single bit of RAM, that represents a quark.

So is the 747 made of something other than quarks? No, you're just modeling it with representational elements that do not have a one-to-one correspondence with the quarks of the 747. The map is not the territory.

Why not model the 747 with a chromodynamic representation? Because then it would take a gazillion years to get any answers out of the model. Also we could not store the model on all the memory on all the computers in the world, as of 2008.

As the saying goes, "The map is not the territory, but you can't fold up the territory and put it in your glove compartment." Sometimes you need a smaller map to fit in a more cramped glove compartment - but this does not change the territory. The scale of a map is not a fact about the territory, it's a fact about the map.

If it were possible to build and run a chromodynamic model of the 747, it would yield accurate predictions. Better predictions than the aerodynamic model, in fact.

To build a fully accurate model of the 747, it is not necessary, in principle, for the model to contain explicit descriptions of things like airflow and lift. There does not have to be a single token, a single bit of RAM, that corresponds to the position of the wings. It is possible, in principle, to build an accurate model of the 747 that makes no mention of anything except elementary particle fields and fundamental forces.

"What?" cries the antireductionist. "Are you telling me the 747 doesn't really have wings? I can see the wings right there!"

The notion here is a subtle one. It's not just the notion that an object can have different descriptions at different levels.

It's the notion that "having different descriptions at different levels" is itself something you say that belongs in the realm of Talking About Maps, not the realm of Talking About Territory.

It's not that the airplane itself, the laws of physics themselves, use different descriptions at different levels - as yonder artillery gunner thought. Rather we, for our convenience, use different simplified models at different levels.

If you looked at the ultimate chromodynamic model, the one that contained only elementary particle fields and fundamental forces, that model would contain all the facts about airflow and lift and wing positions - but these facts would be implicit, rather than explicit.

You, looking at the model, and thinking about the model, would be able to figure out where the wings were. Having figured it out, there would be an explicit representation in your mind of the wing position - an explicit computational object, there in your neural RAM. In your mind.

You might, indeed, deduce all sorts of explicit descriptions of the airplane, at various levels, and even explicit rules for how your models at different levels interacted with each other to produce combined predictions -

And the way that algorithm feels from inside, is that the airplane would seem to be made up of many levels at once, interacting with each other.

The way a belief feels from inside, is that you seem to be looking straight at reality. When it actually seems that you're looking at a belief, as such, you are really experiencing a belief about belief.

So when your mind simultaneously believes explicit descriptions of many different levels, and believes explicit rules for transiting between levels, as part of an efficient combined model, it feels like you are seeing a system that is made of different level descriptions and their rules for interaction.

But this is just the brain trying to be efficiently compress an object that it cannot remotely begin to model on a fundamental level. The airplane is too large. Even a hydrogen atom would be too large. Quark-to-quark interactions are insanely intractable. You can't handle the truth.

But the way physics really works, as far as we can tell, is that there is only the most basic level - the elementary particle fields and fundamental forces. You can't handle the raw truth, but reality can handle it without the slightest simplification. (I wish I knew where Reality got its computing power.)

The laws of physics do not contain distinct additional causal entities that correspond to lift or airplane wings, the way that the mind of an engineer contains distinct additional cognitive entities that correspond to lift or airplane wings.

This, as I see it, is the thesis of reductionism. Reductionism is not a positive belief, but rather, a disbelief that the higher levels of simplified multilevel models are out there in the territory. Understanding this on a gut level dissolves the question of "How can you say the airplane doesn't really have wings, when I can see the wings right there?" The critical words are really and see.

Sequence: Reductionism

Qualitatively Confused

Explaining vs. Explaining Away

Sequence: Quantum Physics

Probability is in the Mind

Zombies! Zombies?

Referenced by: Explaining vs. Explaining Away • Bind Yourself to Reality • If You Demand Magic, Magic Won't Help • Hand vs. Fingers • Reductive Reference • The Generalized Anti-Zombie Principle • Quantum Explanations • The Born Probabilities • Timeless Physics • Timeless Beauty • Thou Art Physics • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • Quantum Mechanics and Personal Identity • Heading Toward Morality • Probability is Subjectively Objective • Setting Up Metaethics • Three Fallacies of Teleology • Dreams of AI Design • Dreams of Friendliness • Excluding the Supernatural • Entangled Truths, Contagious Lies • Mundane Magic • Raising the Sanity Waterline

Original with comments: Reductionism

Explaining vs. Explaining Away

Eliezer Yudkowsky, 17 March 2008 01:59AM

Followup to: Reductionism, Righting a Wrong Question

John Keats's Lamia (1819) surely deserves some kind of award for Most Famously Annoying Poetry:

...Do not all charms fly
At the mere touch of cold philosophy?
There was an awful rainbow once in heaven:
We know her woof, her texture; she is given
In the dull catalogue of common things.
Philosophy will clip an Angel's wings,
Conquer all mysteries by rule and line,
Empty the haunted air, and gnomed mine -
Unweave a rainbow...

My usual reply ends with the phrase: "If we cannot learn to take joy in the merely real, our lives will be empty indeed." I shall expand on that tomorrow.

Today I have a different point in mind. Let's just take the lines:

Empty the haunted air, and gnomed mine -
Unweave a rainbow...

Apparently "the mere touch of cold philosophy", i.e., the truth, has destroyed:

- Haunts in the air

- Gnomes in the mine

- Rainbows

Which calls to mind a rather different bit of verse:

One of these things
Is not like the others
One of these things
Doesn't belong

The air has been emptied of its haunts, and the mine de-gnomed - but the rainbow is still there!

In "Righting a Wrong Question", I wrote:

Tracing back the chain of causality, step by step, I discover that my belief that I'm wearing socks is fully explained by the fact that I'm wearing socks... On the other hand, if I see a mirage of a lake in the desert, the correct causal explanation of my vision does not involve the fact of any actual lake in the desert. In this case, my belief in the lake is not just explained, but explained away.

The rainbow was explained. The haunts in the air, and gnomes in the mine, were explained away.

I think this is the key distinction that anti-reductionists don't get about reductionism.

You can see this failure to get the distinction in the classic objection to reductionism:

If reductionism is correct, then even your belief in reductionism is just the mere result of the motion of molecules - why should I listen to anything you say?

The key word, in the above, is mere; a word which implies that accepting reductionism would explain away all the reasoning processes leading up to my acceptance of reductionism, the way that an optical illusion is explained away.

But you can explain how a cognitive process works without it being "mere"! My belief that I'm wearing socks is a mere result of my visual cortex reconstructing nerve impulses sent from my retina which received photons reflected off my socks... which is to say, according to scientific reductionism, my belief that I'm wearing socks is a mere result of the fact that I'm wearing socks.

What could be going on in the anti-reductionists' minds, such that they would put rainbows and belief-in-reductionism, in the same category as haunts and gnomes?

Several things are going on simultaneously. But for now let's focus on the basic idea introduced yesterday: The Mind Projection Fallacy between a multi-level map and a mono-level territory.

(I.e: There's no way you can model a 747 quark-by-quark, so you've got to use a multi-level map with explicit cognitive representations of wings, airflow, and so on. This doesn't mean there's a multi-level territory. The true laws of physics, to the best of our knowledge, are only over elementary particle fields.)

I think that when physicists say "There are no fundamental rainbows," the anti-reductionists hear, "There are no rainbows."

If you don't distinguish between the multi-level map and the mono-level territory, then when someone tries to explain to you that the rainbow is not a fundamental thing in physics, acceptance of this will feel like erasing rainbows from your multi-level map, which feels like erasing rainbows from the world.

When Science says "tigers are not elementary particles, they are made of quarks" the anti-reductionist hears this as the same sort of dismissal as "we looked in your garage for a dragon, but there was just empty air".

What scientists did to rainbows, and what scientists did to gnomes, seemingly felt the same to Keats...

In support of this sub-thesis, I deliberately used several phrasings, in my discussion of Keats's poem, that were Mind Projection Fallacious. If you didn't notice, this would seem to argue that such fallacies are customary enough to pass unremarked.

For example:

"The air has been emptied of its haunts, and the mine de-gnomed - but the rainbow is still there!"

Actually, Science emptied the model of air of belief in haunts, and emptied the map of the mine of representations of gnomes. Science did not actually - as Keats's poem itself would have it - take real Angel's wings, and destroy them with a cold touch of truth. In reality there never were any haunts in the air, or gnomes in the mine.

Another example:

"What scientists did to rainbows, and what scientists did to gnomes, seemingly felt the same to Keats."

Scientists didn't do anything to gnomes, only to "gnomes". The quotation is not the referent.

But if you commit the Mind Projection Fallacy - and by default, our beliefs just feel like the way the world is - then at time T=0, the mines (apparently) contain gnomes; at time T=1 a scientist dances across the scene, and at time T=2 the mines (apparently) are empty. Clearly, there used to be gnomes there, but the scientist killed them.

Bad scientist! No poems for you, gnomekiller!

Well, that's how it feels, if you get emotionally attached to the gnomes, and then a scientist says there aren't any gnomes. It takes a strong mind, a deep honesty, and a deliberate effort to say, at this point, "That which can be destroyed by the truth should be," and "The scientist hasn't taken the gnomes away, only taken my delusion away," and "I never held just title to my belief in gnomes in the first place; I have not been deprived of anything I rightfully owned," and "If there are gnomes, I desire to believe there are gnomes; if there are no gnomes, I desire to believe there are no gnomes; let me not become attached to beliefs I may not want," and all the other things that rationalists are supposed to say on such occasions.

But with the rainbow it is not even necessary to go that far. The rainbow is still there!

Sequence: Reductionism

Reductionism

Fake Reductionism

Referenced by: Fake Reductionism • Savanna Poets • Joy in the Merely Real • If You Demand Magic, Magic Won't Help • Hand vs. Fingers • Reductive Reference • Zombies! Zombies? • Many Worlds, One Best Guess • Thou Art Physics • Causality and Moral Responsibility • Heading Toward Morality • Probability is Subjectively Objective • The Gift We Give To Tomorrow • Math is Subjunctively Objective • Changing Your Metaethics • The Comedy of Behaviorism • "Arbitrary"

Original with comments: Explaining vs. Explaining Away

Fake Reductionism

Eliezer Yudkowsky, 17 March 2008 10:49PM

Followup to: Explaining vs. Explaining Away, Fake Explanation

There was an awful rainbow once in heaven:
We know her woof, her texture; she is given
In the dull catalogue of common things.
-- John Keats, Lamia

I am guessing - though it is only a guess - that Keats himself did not know the woof and texture of the rainbow. Not the way that Newton understood rainbows. Perhaps not even at all. Maybe Keats just read, somewhere, that Newton had explained the rainbow as "light reflected from raindrops" -

- which was actually known in the 13th century. Newton only added a refinement by showing that the light was decomposed into colored parts, rather than transformed in color. But that put rainbows back in the news headlines. And so Keats, with Charles Lamb and William Wordsworth and Benjamin Haydon, drank "Confusion to the memory of Newton" because "he destroyed the poetry of the rainbow by reducing it to a prism." That's one reason to suspect Keats didn't understand the subject too deeply.

I am guessing, though it is only a guess, that Keats could not have sketched out on paper why rainbows only appear when the Sun is behind your head, or why the rainbow is an arc of a circle.

If so, Keats had a Fake Explanation. In this case, a fake reduction. He'd been told that the rainbow had been reduced, but it had not actually been reduced in his model of the world.

This is another of those distinctions that anti-reductionists fail to get - the difference between professing the flat fact that something is reducible, and seeing it.

In this, the anti-reductionists are not too greatly to be blamed, for it is part of a general problem.

I've written before on seeming knowledge that is not knowledge, and beliefs that are not about their supposed objects but only recordings to recite back in the classroom, and words that operate as stop signs for curiosity rather than answers, and technobabble which only conveys membership in the literary genre of "science"...

There is a very great distinction between being able to see where the rainbow comes from, and playing around with prisms to confirm it, and maybe making a rainbow yourself by spraying water droplets -

- versus some dour-faced philosopher just telling you, "No, there's nothing special about the rainbow. Didn't you hear? Scientists have explained it away. Just something to do with raindrops or whatever. Nothing to be excited about."

I think this distinction probably accounts for a hell of a lot of the deadly existential emptiness that supposedly accompanies scientific reductionism.

You have to interpret the anti-reductionists' experience of "reductionism", not in terms of their actually seeing how rainbows work, not in terms of their having the critical "Aha!", but in terms of their being told that the password is "Science". The effect is just to move rainbows to a different literary genre - a literary genre they have been taught to regard as boring.

For them, the effect of hearing "Science has explained rainbows!" is to hang up a sign over rainbows saying, "This phenomenon has been labeled BORING by order of the Council of Sophisticated Literary Critics. Move along."

And that's all the sign says: only that, and nothing more.

So the literary critics have their gnomes yanked out by force; not dissolved in insight, but removed by flat order of authority. They are given no beauty to replace the hauntless air, no genuine understanding that could be interesting in its own right. Just a label saying, "Ha! You thought rainbows were pretty? You poor, unsophisticated fool. This is part of the literary genre of science, of dry and solemn incomprehensible words."

That's how anti-reductionists experience "reductionism".

Well, can't blame Keats, poor lad probably wasn't raised right.

But he dared to drink "Confusion to the memory of Newton"?

I propose "To the memory of Keats's confusion" as a toast for rationalists. Cheers.

Sequence: Reductionism

Explaining vs. Explaining Away

Savanna Poets

Referenced by: Savanna Poets • Joy in the Merely Real • Hand vs. Fingers • Angry Atoms • Dreams of AI Design • Dreams of Friendliness • The Anthropic Trilemma

Original with comments: Fake Reductionism

Savanna Poets

Eliezer Yudkowsky, 18 March 2008 06:42PM

Followup to: Explaining vs. Explaining Away

"Poets say science takes away from the beauty of the stars - mere globs of gas atoms. Nothing is "mere". I too can see the stars on a desert night, and feel them. But do I see less or more?
"The vastness of the heavens stretches my imagination - stuck on this carousel my little eye can catch one-million-year-old light. A vast pattern - of which I am a part - perhaps my stuff was belched from some forgotten star, as one is belching there. Or see them with the greater eye of Palomar, rushing all apart from some common starting point when they were perhaps all together. What is the pattern, or the meaning, or the why? It does not do harm to the mystery to know a little about it.
"For far more marvelous is the truth than any artists of the past imagined! Why do the poets of the present not speak of it?
"What men are poets who can speak of Jupiter if he were like a man, but if he is an immense spinning sphere of methane and ammonia must be silent?"
-- Richard Feynman, The Feynman Lectures on Physics, Vol I, p. 3-6 (line breaks added)

That's a real question, there on the last line - what kind of poet can write about Jupiter the god, but not Jupiter the immense sphere? Whether or not Feynman meant the question rhetorically, it has a real answer:

If Jupiter is like us, he can fall in love, and lose love, and regain love.
If Jupiter is like us, he can strive, and rise, and be cast down.
If Jupiter is like us, he can laugh or weep or dance.

If Jupiter is an immense spinning sphere of methane and ammonia, it is more difficult for the poet to make us feel.

There are poets and storytellers who say that the Great Stories are timeless, and they never change, they only ever retold. They say, with pride, that Shakespeare and Sophocles are bound by ties of craft stronger than mere centuries; that the two playwrights could have swapped times without a jolt.

Donald Brown once compiled a list of over two hundred "human universals", found in all (or a vast supermajority of) studied human cultures, from San Francisco to the !Kung of the Kalahari Desert. Marriage is on the list, and incest avoidance, and motherly love, and sibling rivalry, and music and envy and dance and storytelling and aesthetics, and ritual magic to heal the sick, and poetry in spoken lines separated by pauses -

No one who knows anything about evolutionary psychology could be expected to deny it: The strongest emotions we have are deeply engraved, blood and bone, brain and DNA.

It might take a bit of tweaking, but you probably could tell "Hamlet" sitting around a campfire on the ancestral savanna.

So one can see why John "Unweave a rainbow" Keats might feel something had been lost, on being told that the rainbow was sunlight scattered from raindrops. Raindrops don't dance.

In the Old Testament, it is written that God once destroyed the world with a flood that covered all the land, drowning all the horribly guilty men and women of the world along with their horribly guilty babies, but Noah built a gigantic wooden ark, etc., and after most of the human species was wiped out, God put rainbows in the sky as a sign that he wouldn't do it again. At least not with water.

You can see how Keats would be shocked that this beautiful story was contradicted by modern science. Especially if (as I described yesterday) Keats had no real understanding of rainbows, no "Aha!" insight that could be fascinating in its own right, to replace the drama subtracted -

Ah, but maybe Keats would be right to be disappointed even if he knew the math. The Biblical story of the rainbow is a tale of bloodthirsty murder and smiling insanity. How could anything about raindrops and refraction properly replace that? Raindrops don't scream when they die.

So science takes the romance away (says the Romantic poet), and what you are given back, never matches the drama of the original -

(that is, the original delusion)

- even if you do know the equations, because the equations are not about strong emotions.

That is the strongest rejoinder I can think of, that any Romantic poet could have said to Feynman - though I can't remember ever hearing it said.

You can guess that I don't agree with the Romantic poets. So my own stance is this:

It is not necessary for Jupiter to be like a human, because humans are like humans. If Jupiter is an immense spinning sphere of methane and ammonia, that doesn't mean that love and hate are emptied from the universe. There are still loving and hating minds in the universe. Us.

With more than six billion of us at the last count, does Jupiter really need to be on the list of potential protagonists?

It is not necessary to tell the Great Stories about planets or rainbows. They play out all over our world, every day. Every day, someone kills for revenge; every day, someone kills a friend by mistake; every day, upward of a hundred thousand people fall in love. And even if this were not so, you could write fiction about humans - not about Jupiter.

Earth is old, and has played out the same stories many times beneath the Sun. I do wonder if it might not be time for some of the Great Stories to change. For me, at least, the story called "Goodbye" has lost its charm.

The Great Stories are not timeless, because the human species is not timeless. Go far enough back in hominid evolution, and no one will understand Hamlet. Go far enough back in time, and you won't find any brains.

The Great Stories are not eternal, because the human species, Homo sapiens sapiens, is not eternal. I most sincerely doubt that we have another thousand years to go in our current form. I do not say this in sadness: I think we can do better.

I would not like to see all the Great Stories lost completely, in our future. I see very little difference between that outcome, and the Sun falling into a black hole.

But the Great Stories in their current forms have already been told, over and over. I do not think it ill if some of them should change their forms, or diversify their endings.

"And they lived happily ever after" seems worth trying at least once.

The Great Stories can and should diversify, as humankind grows up. Part of that ethic is the idea that when we find strangeness, we should respect it enough to tell its story truly. Even if it makes writing poetry a little more difficult.

If you are a good enough poet to write an ode to an immense spinning sphere of methane and ammonia, you are writing something original, about a newly discovered part of the real universe. It may not be as dramatic, or as gripping, as Hamlet. But the tale of Hamlet has already been told! If you write of Jupiter as though it were a human, then you are making our map of the universe just a little more impoverished of complexity; you are forcing Jupiter into the mold of all the stories that have already been told of Earth.

James Thomson's "A Poem Sacred to the Memory of Sir Isaac Newton", which praises the rainbow for what it really is - you can argue whether or not Thomson's poem is as gripping as John Keats's Lamia who was loved and lost. But tales of love and loss and cynicism had already been told, far away in ancient Greece, and no doubt many times before. Until we understood the rainbow as a thing different from tales of human-shaped magic, the true story of the rainbow could not be poeticized.

The border between science fiction and space opera was once drawn as follows: If you can take the plot of a story and put it back in the Old West, or the Middle Ages, without changing it, then it is not real science fiction. In real science fiction, the science is intrinsically part of the plot - you can't move the story from space to the savanna, not without losing something.

Richard Feynman asked: "What men are poets who can speak of Jupiter if he were like a man, but if he is an immense spinning sphere of methane and ammonia must be silent?"

They are savanna poets, who can only tell stories that would have made sense around a campfire ten thousand years ago. Savanna poets, who can tell only the Great Stories in their classic forms, and nothing more.

Sequence: Reductionism

Fake Reductionism

Hand vs. Fingers

Original with comments: Savanna Poets

Joy in the Merely Real

Eliezer Yudkowsky, 20 March 2008 06:18AM

Followup to: Explaining vs. Explaining Away

...Do not all charms fly
At the mere touch of cold philosophy?
There was an awful rainbow once in heaven:
We know her woof, her texture; she is given
In the dull catalogue of common things.
-- John Keats, Lamia

"Nothing is 'mere'."
-- Richard Feynman

You've got to admire that phrase, "dull catalogue of common things". What is it, exactly, that goes in this catalogue? Besides rainbows, that is?

Why, things that are mundane, of course. Things that are normal; things that are unmagical; things that are known, or knowable; things that play by the rules (or that play by any rules, which makes them boring); things that are part of the ordinary universe; things that are, in a word, real.

Now that's what I call setting yourself up for a fall.

At that rate, sooner or later you're going to be disappointed in everything - either it will turn out not to exist, or even worse, it will turn out to be real.

If we cannot take joy in things that are merely real, our lives will always be empty.

For what sin are rainbows demoted to the dull catalogue of common things? For the sin of having a scientific explanation. "We know her woof, her texture", says Keats - an interesting use of the word "we", because I suspect that Keats didn't know the explanation himself. I suspect that just being told that someone else knew was too much for him to take. I suspect that just the notion of rainbows being scientifically explicable in principle would have been too much to take. And if Keats didn't think like that, well, I know plenty of people who do.

I have already remarked that nothing is inherently mysterious - nothing that actually exists, that is. If I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon; to worship a phenomenon because it seems so wonderfully mysterious, is to worship your own ignorance; a blank map does not correspond to a blank territory, it is just somewhere we haven't visited yet, etc. etc...

Which is to say that everything - everything that actually exists - is liable to end up in "the dull catalogue of common things", sooner or later.

Your choice is either:

- Decide that things are allowed to be unmagical, knowable, scientifically explicable, in a word, real, and yet still worth caring about;

- Or go about the rest of your life suffering from existential ennui that is unresolvable.

(Self-deception might be an option for others, but not for you.)

This puts quite a different complexion on the bizarre habit indulged by those strange folk called scientists, wherein they suddenly become fascinated by pocket lint or bird droppings or rainbows, or some other ordinary thing which world-weary and sophisticated folk would never give a second glance.

You might say that scientists - at least some scientists - are those folk who are in principle capable of enjoying life in the real universe.

Sequence: Joy in the Merely Real

Joy in Discovery

Referenced by: Joy in Discovery • Bind Yourself to Reality • If You Demand Magic, Magic Won't Help • The Beauty of Settled Science • Hand vs. Fingers • Living in Many Worlds • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • Heading Toward Morality • Rebelling Within Nature • The Gift We Give To Tomorrow • Existential Angst Factory • Changing Your Metaethics • Setting Up Metaethics • Excluding the Supernatural • Mundane Magic • Raising the Sanity Waterline • The Sacred Mundane

Original with comments: Joy in the Merely Real

Joy in Discovery

Eliezer Yudkowsky, 21 March 2008 01:19AM

Followup to: Joy in the Merely Real

"Newton was the greatest genius who ever lived, and the most fortunate; for we cannot find more than once a system of the world to establish."
-- Lagrange

I have more fun discovering things for myself than reading about them in textbooks. This is right and proper, and only to be expected.

But discovering something that no one else knows - being the first to unravel the secret -

There is a story that one of the first men to realize that stars were burning by fusion - plausible attributions I've seen are to Fritz Houtermans and Hans Bethe - was walking out with his girlfriend of a night, and she made a comment on how beautiful the stars were, and he replied: "Yes, and right now, I'm the only man in the world who knows why they shine."

It is attested by numerous sources that this experience, being the first person to solve a major mystery, is a tremendous high. It's probably the closest experience you can get to taking drugs, without taking drugs - though I wouldn't know.

That can't be healthy.

Not that I'm objecting to the euphoria. It's the exclusivity clause that bothers me. Why should a discovery be worth less, just because someone else already knows the answer?

The most charitable interpretation I can put on the psychology, is that you don't struggle with a single problem for months or years if it's something you can just look up in the library. And that the tremendous high comes from having hit the problem from every angle you can manage, and having bounced; and then having analyzed the problem again, using every idea you can think of, and all the data you can get your hands on - making progress a little at a time - so that when, finally, you crack through the problem, all the dangling pieces and unresolved questions fall into place at once, like solving a dozen locked-room murder mysteries with a single clue.

And more, the understanding you get is real understanding - understanding that embraces all the clues you studied to solve the problem, when you didn't yet know the answer. Understanding that comes from asking questions day after day and worrying at them; understanding that no one else can get (no matter how much you tell them the answer) unless they spend months studying the problem in its historical context, even after it's been solved - and even then, they won't get the high of solving it all at once.

That's one possible reason why James Clerk Maxwell might have had more fun discovering Maxwell's Equations, than you had fun reading about them.

A slightly less charitable reading is that the tremendous high comes from what is termed, in the politesse of social psychology, "commitment" and "consistency" and "cognitive dissonance"; the part where we value something more highly just because it took more work to get it. The studies showing that subjective fraternity pledges to a harsher initiation, causes them to be more convinced of the value of the fraternity - identical wine in higher-priced bottles being rated as tasting better - that sort of thing.

Of course, if you just have more fun solving a puzzle than being told its answer, because you enjoy doing the cognitive work for its own sake, there's nothing wrong with that. The less charitable reading would be if charging $100 to be told the answer to a puzzle, made you think the answer was more interesting, worthwhile, important, surprising, etc. than if you got the answer for free.

(I strongly suspect that a major part of science's PR problem in the population at large is people who instinctively believe that if knowledge is given away for free, it cannot be important. If you had to undergo a fearsome initiation ritual to be told the truth about evolution, maybe people would be more satisfied with the answer.)

The really uncharitable reading is that the joy of first discovery is about status. Competition. Scarcity. Beating everyone else to the punch. It doesn't matter whether you have a 3-room house or a 4-room house, what matters is having a bigger house than the Joneses. A 2-room house would be fine, if you could only ensure that the Joneses had even less.

I don't object to competition as a matter of principle. I don't think that the game of Go is barbaric and should be suppressed, even though it's zero-sum. But if the euphoric joy of scientific discovery has to be about scarcity, that means it's only available to one person per civilization for any given truth.

If the joy of scientific discovery is one-shot per discovery, then, from a fun-theoretic perspective, Newton probably used up a substantial increment of the total Physics Fun available over the entire history of Earth-originating intelligent life. That selfish bastard explained the orbits of planets and the tides.

And really the situation is even worse than this, because in the Standard Model of physics (discovered by bastards who spoiled the puzzle for everyone else) the universe is spatially infinite, inflationarily branching, and branching via decoherence, which is at least three different ways that Reality is exponentially or infinitely large

So aliens, or alternate Newtons, or just Tegmark duplicates of Newton, may all have discovered gravity before our Newton did - if you believe that "before" means anything relative to those kinds of separations.

When that thought first occurred to me, I actually found it quite uplifting. Once I realized that someone, somewhere in the expanses of space and time, already knows the answer to any answerable question - even biology questions and history questions; there are other decoherent Earths - then I realized how silly it was to think as if the joy of discovery ought to be limited to one person. It becomes a fully inescapable source of unresolvable existential angst, and I regard that as a reductio.

The consistent solution which maintains the possibility of fun, is to stop worrying about what other people know. If you don't know the answer, it's a mystery to you. If you can raise your hand, and clench your fingers into a fist, and you've got no idea of how your brain is doing it - or even what exact muscles lay beneath your skin - you've got to consider yourself just as ignorant as a hunter-gatherer. Sure, someone else knows the answer - but back in the hunter-gatherer days, someone else in an alternate Earth, or for that matter, someone else in the future, knew what the answer was. Mystery, and the joy of finding out, is either a personal thing, or it doesn't exist at all - and I prefer to say it's personal.

The joy of assisting your civilization by telling it something it doesn't already know, does tend to be one-shot per discovery per civilization; that kind of value is conserved, as are Nobel Prizes. And the prospect of that reward may be what it takes to keep you focused on one problem for the years required to develop a really deep understanding; plus, working on a problem unknown to your civilization is a sure-fire way to avoid reading any spoilers.

But as part of my general project to undo this idea that rationalists have less fun, I want to restore the magic and mystery to every part of the world which you do not personally understand, regardless of what other knowledge may exist, far away in space and time, or even in your next-door neighbor's mind. If you don't know, it's a mystery. And now think of how many things you don't know! (If you can't think of anything, you have other problems.) Isn't the world suddenly a much more mysterious and magical and interesting place? As if you'd been transported into an alternate dimension, and had to learn all the rules from scratch?

"A friend once told me that I look at the world as if I've never seen it before. I thought, that's a nice compliment... Wait! I never have seen it before! What -- did everyone else get a preview?"
-- Ran Prieur

Sequence: Joy in the Merely Real

Joy in the Merely Real

Bind Yourself to Reality

Referenced by: The Beauty of Settled Science • To Spread Science, Keep It Secret • Living in Many Worlds • Mundane Magic • Devil's Offers • 31 Laws of Fun

Original with comments: Joy in Discovery

Bind Yourself to Reality

Eliezer Yudkowsky, 22 March 2008 05:09AM

Followup to: Joy in the Merely Real

So perhaps you're reading all this, and asking: "Yes, but what does this have to do with reductionism?"

Partially, it's a matter of leaving a line of retreat. It's not easy to take something important apart into components, when you're convinced that this removes magic from the world, unweaves the rainbow. I do plan to take certain things apart, on this blog; and I prefer not to create pointless existential anguish.

Partially, it's the crusade against Hollywood Rationality, the concept that understanding the rainbow subtracts its beauty. The rainbow is still beautiful plus you get the beauty of physics.

But even more deeply, it's one of these subtle hidden-core-of-rationality things. You know, the sort of thing where I start talking about 'the Way'. It's about binding yourself to reality.

In one of Frank Herbert's Dune books, IIRC, it is said that a Truthsayer gains their ability to detect lies in others by always speaking truth themselves, so that they form a relationship with the truth whose violation they can feel. It wouldn't work, but I still think it's one of the more beautiful thoughts in fiction. At the very least, to get close to the truth, you have to be willing to press yourself up against reality as tightly as possible, without flinching away, or sneering down.

You can see the bind-yourself-to-reality theme in "Lotteries: A Waste of Hope." Understanding that lottery tickets have negative expected utility, does not mean that you give up the hope of being rich. It means that you stop wasting that hope on lottery tickets. You put the hope into your job, your school, your startup, your eBay sideline; and if you truly have nothing worth hoping for, then maybe it's time to start looking.

It's not dreams I object to, only impossible dreams. The lottery isn't impossible, but it is an un-actionable near-impossibility. It's not that winning the lottery is extremely difficult - requires a desperate effort - but that work isn't the issue.

I say all this, to exemplify the idea of taking emotional energy that is flowing off to nowhere, and binding it into the realms of reality.

This doesn't mean setting goals that are low enough to be "realistic", i.e., easy and safe and parentally approved. Maybe this is good advice in your personal case, I don't know, but I'm not the one to say it.

What I mean is that you can invest emotional energy in rainbows even if they turn out not to be magic. The future is always absurd but it is never unreal.

The Hollywood Rationality stereotype is that "rational = emotionless"; the more reasonable you are, the more of your emotions Reason inevitably destroys. In "Feeling Rational" I contrast this against "That which can be destroyed by the truth should be" and "That which the truth nourishes should thrive". When you have arrived at your best picture of the truth, there is nothing irrational about the emotions you feel as a result of that - the emotions cannot be destroyed by truth, so they must not be irrational.

So instead of destroying emotional energies associated with bad explanations for rainbows, as the Hollywood Rationality stereotype would have it, let us redirect these emotional energies into reality - bind them to beliefs that are as true as we can make them.

Want to fly? Don't give up on flight. Give up on flying potions and build yourself an airplane.

Remember the theme of "Think Like Reality", where I talked about how when physics seems counterintuitive, you've got to accept that it's not physics that's weird, it's you?

What I'm talking about now is like that, only with emotions instead of hypotheses - binding your feelings into the real world. Not the "realistic" everyday world. I would be a howling hypocrite if I told you to shut up and do your homework. I mean the real real world, the lawful universe, that includes absurdities like Moon landings and the evolution of human intelligence. Just not any magic, anywhere, ever.

It is a Hollywood Rationality meme that "Science takes the fun out of life."

Science puts the fun back into life.

Rationality directs your emotional energies into the universe, rather than somewhere else.

Sequence: Joy in the Merely Real

Joy in Discovery

If You Demand Magic, Magic Won't Help

Referenced by: Is Humanism A Religion-Substitute? • To Spread Science, Keep It Secret

Original with comments: Bind Yourself to Reality

If You Demand Magic, Magic Won't Help

Eliezer Yudkowsky, 22 March 2008 06:10PM

Followup to: Explaining vs. Explaining Away, Joy in the Merely Real

Most witches don't believe in gods. They know that the gods exist, of course. They even deal with them occasionally. But they don't believe in them. They know them too well. It would be like believing in the postman.
-- Terry Pratchett, Witches Abroad

Once upon a time, I was pondering the philosophy of fantasy stories -

And before anyone chides me for my "failure to understand what fantasy is about", let me say this: I was raised in an SF&F household. I have been reading fantasy stories since I was five years old. I occasionally try to write fantasy stories. And I am not the sort of person who tries to write for a genre without pondering its philosophy. Where do you think story ideas come from?

Anyway:

I was pondering the philosophy of fantasy stories, and it occurred to me that if there were actually dragons in our world - if you could go down to the zoo, or even to a distant mountain, and meet a fire-breathing dragon - while nobody had ever actually seen a zebra, then our fantasy stories would contain zebras aplenty, while dragons would be unexciting.

Now that's what I call painting yourself into a corner, wot? The grass is always greener on the other side of unreality.

In one of the standard fantasy plots, a protagonist from our Earth, a sympathetic character with lousy grades or a crushing mortgage but still a good heart, suddenly finds themselves in a world where magic operates in place of science. The protagonist often goes on to practice magic, and become in due course a (superpowerful) sorcerer.

Now here's the question - and yes, it is a little unkind, but I think it needs to be asked: Presumably most readers of these novels see themselves in the protagonist's shoes, fantasizing about their own acquisition of sorcery. Wishing for magic. And, barring improbable demographics, most readers of these novels are not scientists.

Born into a world of science, they did not become scientists. What makes them think that, in a world of magic, they would act any differently?

If they don't have the scientific attitude, that nothing is "mere" - the capacity to be interested in merely real things - how will magic help them? If they actually had magic, it would be merely real, and lose the charm of unattainability. They might be excited at first, but (like the lottery winners who, six months later, aren't nearly as happy as they expected to be), the excitement would soon wear off. Probably as soon as they had to actually study spells.

Unless they can find the capacity to take joy in things that are merely real. To be just as excited by hang-gliding, as riding a dragon; to be as excited by making a light with electricity, as by making a light with magic... even if it takes a little study...

Don't get me wrong. I'm not dissing dragons. Who knows, we might even create some, one of these days.

But if you don't have the capacity to enjoy hang-gliding even though it is merely real, then as soon as dragons turn real, you're not going to be any more excited by dragons than you are by hang-gliding.

Do you think you would prefer living in the Future, to living in the present? That's a quite understandable preference. Things do seem to be getting better over time.

But don't forget that this is the Future, relative to the Dark Ages of a thousand years earlier. You have opportunities undreamt-of even by kings.

If the trend continues, the Future might be a very fine place indeed in which to live. But if you do make it to the Future, what you find, when you get there, will be another Now. If you don't have the basic capacity to enjoy being in a Now - if your emotional energy can only go into the Future, if you can only hope for a better tomorrow - then no amount of passing time can help you.

(Yes, in the Future there could be a pill that fixes the emotional problem of always looking to the Future. I don't think this invalidates my basic point, which is about what sort of pills we should want to take.)

Matthew C., commenting here on OB, seems very excited about an informally specified "theory" by Rupert Sheldrake which "explains" such non-explanation-demanding phenomena as protein folding and snowflake symmetry. But why isn't Matthew C. just as excited about, say, Special Relativity? Special Relativity is actually known to be a law, so why isn't it even more exciting? The advantage of becoming excited about a law already known to be true, is that you know your excitement will not be wasted.

If Sheldrake's theory were accepted truth taught in elementary schools, Matthew C. wouldn't care about it. Or why else is Matthew C. fascinated by that one particular law which he believes to be a law of physics, more than all the other laws?

The worst catastrophe you could visit upon the New Age community would be for their rituals to start working reliably, and for UFOs to actually appear in the skies. What would be the point of believing in aliens, if they were just there, and everyone else could see them too? In a world where psychic powers were merely real, New Agers wouldn't believe in psychic powers, any more than anyone cares enough about gravity to believe in it. (Except for scientists, of course.)

Why am I so negative about magic? Would it be wrong for magic to exist?

I'm not actually negative on magic. Remember, I occasionally try to write fantasy stories. But I'm annoyed with this psychology that, if it were born into a world where spells and potions did work, would pine away for a world where household goods were abundantly produced by assembly lines.

Part of binding yourself to reality, on an emotional as well as intellectual level, is coming to terms with the fact that you do live here. Only then can you see this, your world, and whatever opportunities it holds out for you, without wishing your sight away.

Not to put too fine a point on it, but I've found no lack of dragons to fight, or magics to master, in this world of my birth. If I were transported into one of those fantasy novels, I wouldn't be surprised to find myself studying the forbidden ultimate sorcery -

- because why should being transported into a magical world change anything? It's not where you are, it's who you are.

So remember the Litany Against Being Transported Into An Alternate Universe:

If I'm going to be happy anywhere,
Or achieve greatness anywhere,
Or learn true secrets anywhere,
Or save the world anywhere,
Or feel strongly anywhere,
Or help people anywhere,
I may as well do it in reality.

Sequence: Joy in the Merely Real

Bind Yourself to Reality

Mundane Magic

Referenced by: The Beauty of Settled Science • Is Humanism A Religion-Substitute? • Scarcity • Lawrence Watt-Evans's Fiction • Mundane Magic

Original with comments: If You Demand Magic, Magic Won't Help

New York OB Meetup (ad-hoc) on Monday, Mar 24, @6pm

Eliezer Yudkowsky, 22 March 2008 09:07PM

Correction: The giant Starbucks is at 13 Astor Place #25, not 51 Astor which is a smaller Starbucks. (Yes, there are two Starbucks a block apart, here.) The smaller Starbucks has metal steps and a ramp leading up; don't go here. The giant Starbucks, which is where we want to go, is next to Lafayette and Astor.

I (Eliezer) am in New York at the moment, and will have some time free on Monday night, March 24th, to meet any interested New York Overcoming Bias readers.

Where: The giant Starbucks at 13 Astor Place #25, New York, NY 10003
When: 6pm, this Monday (March 24th, 2008)
Who: Eliezer Yudkowsky (866-983-597), Carl Shulman

If you plan on attending, please leave a comment, so we know to expect you. If you're going to arrive later than 6pm, please note this as well.

I'll also be at Princeton on Sunday. My time is already mostly spoken for, but if you're at Princeton and desperately want to meet up, comment or email before 7am tomorrow.

Original with comments: New York OB Meetup (ad-hoc) on Monday, Mar 24, @6pm

The Beauty of Settled Science

Eliezer Yudkowsky, 24 March 2008 05:30AM

Followup to: Joy in Discovery, The Simple Math of Everything

Facts do not need to be unexplainable, to be beautiful; truths do not become less worth learning, if someone else knows them; beliefs do not become less worthwhile, if many others share them...

...and if you only care about scientific issues that are controversial, you will end up with a head stuffed full of garbage.

The media thinks that only the cutting edge of science is worth reporting on. How often do you see headlines like "General Relativity still governing planetary orbits" or "Phlogiston theory remains false"? So, by the time anything is solid science, it is no longer a breaking headline. "Newsworthy" science is often based on the thinnest of evidence and wrong half the time - if it were not on the uttermost fringes of the scientific frontier, it would not be breaking news.

Scientific controversies are problems so difficult that even people who've spent years mastering the field can still fool themselves. That's what makes for the heated arguments that attract all the media attention.

Worse, if you aren't in the field and part of the game, controversies aren't even fun.

Oh, sure, you can have the fun of picking a side in an argument. But you can get that in any football game. That's not what the fun of science is about.

Reading a well-written textbook, you get: Carefully phrased explanations for incoming students, math derived step by step (where applicable), plenty of experiments cited as illustration (where applicable), test problems on which to display your new mastery, and a reasonably good guarantee that what you're learning is actually true.

Reading press releases, you usually get: Fake explanations that convey nothing except the delusion of understanding of a result that the press release author didn't understand and that probably has a better-than-even chance of failing to replicate.

Modern science is built on discoveries, built on discoveries, built on discoveries, and so on, all the way back to people like Archimedes, who discovered facts like why boats float, that can make sense even if you don't know about other discoveries. A good place to start traveling that road is at the beginning.

Don't be embarrassed to read elementary science textbooks, either. If you want to pretend to be sophisticated, go find a play to sneer at. If you just want to have fun, remember that simplicity is at the core of scientific beauty.

And thinking you can jump right into the frontier, when you haven't learned the settled science, is like...

...like trying to climb only the top half of Mount Everest (which is the only part that interests you) by standing at the base of the mountain, bending your knees, and jumping really hard (so you can pass over the boring parts).

Now I'm not saying that you should never pay attention to scientific controversies. If 40% of oncologists think that white socks cause cancer, and the other 60% violently disagree, this is an important fact to know.

Just don't go thinking that science has to be controversial to be interesting.

Or, for that matter, that science has to be recent to be interesting. A steady diet of science news is bad for you: You are what you eat, and if you eat only science reporting on fluid situations, without a solid textbook now and then, your brain will turn to liquid.

Sequence: Joy in the Merely Real

Mundane Magic

Amazing Breakthrough Day: April 1st

Referenced by: Amazing Breakthrough Day: April 1st • Do Fandoms Need Awfulness? • The Correct Contrarian Cluster

Original with comments: The Beauty of Settled Science

Amazing Breakthrough Day: April 1st

Eliezer Yudkowsky, 25 March 2008 05:45AM

So you're thinking, "April 1st... isn't that already supposed to be April Fool's Day?"

Yes - and that will provide the ideal cover for celebrating Amazing Breakthrough Day.

As I argued in "The Beauty of Settled Science", it is a major problem that media coverage of science focuses only on breaking news. Breaking news, in science, occurs at the furthest fringes of the scientific frontier, which means that the new discovery is often:

- Controversial

- Supported by only one experiment

- Way the heck more complicated than an ordinary mortal can handle, and requiring lots of prerequisite science to understand, which is why it wasn't solved three centuries ago

- Later shown to be wrong

People never get to see the solid stuff, let alone the understandable stuff, because it isn't breaking news.

On Amazing Breakthrough Day, I propose, journalists who really care about science can report - under the protective cover of April 1st - such important but neglected science stories as:

- BOATS EXPLAINED: Centuries-Old Problem Solved By Bathtub Nudist

- YOU SHALL NOT CROSS! Königsberg Tourists' Hopes Dashed

- ARE YOUR LUNGS ON FIRE? Link Between Respiration And Combustion Gains Acceptance Among Scientists

Note that every one of these headlines are true - they describe events that did, in fact, happen. They just didn't happen yesterday.

There have been many humanly understandable amazing breakthroughs in the history of science, which can be understood without a PhD or even BSc. The operative word here is history. Think of Archimedes's "Eureka!" when he understood the relation between the water a ship displaces, and the reason the ship floats. This is far enough back in scientific history that you don't need to know 50 other discoveries to understand the theory; it can be explained in a couple of graphs; anyone can see how it's useful; and the confirming experiments can be duplicated in your own bathtub.

Modern science is built on discoveries built on discoveries built on discoveries and so on all the way back to Archimedes. Reporting science only as breaking news is like wandering into a movie 3/4ths of the way through, writing a story about "Bloody-handed man kisses girl holding gun!" and wandering back out again.

And if your editor says, "Oh, but our readers won't be interested in that -"

Then point out that Reddit and Digg don't link only to breaking news. They also link to short webpages that give good explanations of old science. Readers vote it up, and that should tell you something. Explain that if your newspaper doesn't change to look more like Reddit, you'll have to start selling drugs to make payroll. Editors love to hear that sort of thing, right?

On the Internet, a good new explanation of old science is news and it spreads like news. Why couldn't the science sections of newspapers work the same way? Why isn't a new explanation worth reporting on?

But all this is too visionary for a first step. For now, let's just see if any journalists out there pick up on Amazing Breakthrough Day, where you report on some understandable science breakthrough as though it had just occurred.

April 1st. Put it on your calendar.

Sequence: Joy in the Merely Real

The Beauty of Settled Science

Is Humanism A Religion-Substitute?

Referenced by: Brain Breakthrough! It's Made of Neurons!

Original with comments: Amazing Breakthrough Day: April 1st

Is Humanism A Religion-Substitute?

Eliezer Yudkowsky, 26 March 2008 04:18AM

Followup to: Bind Yourself to Reality

For many years before the Wright Brothers, people dreamed of flying with magic potions. There was nothing irrational about the raw desire to fly. There was nothing tainted about the wish to look down on a cloud from above. Only the "magic potions" part was irrational.

Suppose you were to put me into an fMRI scanner, and take a movie of my brain's activity levels, while I watched a space shuttle launch. (Wanting to visit space is not "realistic", but it is an essentially lawful dream - one that can be fulfilled in a lawful universe.) The fMRI might - maybe, maybe not - resemble the fMRI of a devout Christian watching a nativity scene.

Should an experimenter obtain this result, there's a lot of people out there, both Christians and some atheists, who would gloat: "Ha, ha, space travel is your religion!"

But that's drawing the wrong category boundary. It's like saying that, because some people once tried to fly by irrational means, no one should ever enjoy looking out of an airplane window on the clouds below.

If a rocket launch is what it takes to give me a feeling of aesthetic transcendence, I do not see this as a substitute for religion. That is theomorphism - the viewpoint from gloating religionists who assume that everyone who isn't religious has a hole in their mind that wants filling.

Now, to be fair to the religionists, this is not just a gloating assumption. There are atheists who have religion-shaped holes in their minds. I have seen attempts to substitute atheism or even transhumanism for religion. And the result is invariably awful. Utterly awful. Absolutely abjectly awful.

I call such efforts, "hymns to the nonexistence of God".

When someone sets out to write an atheistic hymn - "Hail, oh unintelligent universe," blah, blah, blah - the result will, without exception, suck.

Why? Because they're being imitative. Because they have no motivation for writing the hymn except a vague feeling that since churches have hymns, they ought to have one too. And, on a purely artistic level, that puts them far beneath genuine religious art that is not an imitation of anything, but an original expression of emotion.

Religious hymns were (often) written by people who felt strongly and wrote honestly and put serious effort into the prosody and imagery of their work - that's what gives their work the grace that it possesses, of artistic integrity.

So are atheists doomed to hymnlessness?

There is an acid test of attempts at post-theism. The acid test is: "If religion had never existed among the human species - if we had never made the original mistake - would this song, this art, this ritual, this way of thinking, still make sense?"

If humanity had never made the original mistake, there would be no hymns to the nonexistence of God. But there would still be marriages, so the notion of an atheistic marriage ceremony makes perfect sense - as long as you don't suddenly launch into a lecture on how God doesn't exist. Because, in a world where religion never had existed, nobody would interrupt a wedding to talk about the implausibility of a distant hypothetical concept. They'd talk about love, children, commitment, honesty, devotion, but who the heck would mention God?

And, in a human world where religion never had existed, there would still be people who got tears in their eyes watching a space shuttle launch.

Which is why, even if experiment shows that watching a shuttle launch makes "religion"-associated areas of my brain light up, associated with feelings of transcendence, I do not see that as a substitute for religion; I expect the same brain areas would light up, for the same reason, if I lived in a world where religion had never been invented.

A good "atheistic hymn" is simply a song about anything worth singing about that doesn't happen to be religious.

Also, reversed stupidity is not intelligence. The world's greatest idiot may say the Sun is shining, but that doesn't make it dark out. The point is not to create a life that resembles religion as little as possible in every surface aspect - this is the same kind of thinking that inspires hymns to the nonexistence of God. If humanity had never made the original mistake, no one would be trying to avoid things that vaguely resembled religion. Believe accurately, then feel accordingly: If space launches actually exist, and watching a rocket rise makes you want to sing, then write the song, dammit.

If I get tears in my eyes at a space shuttle launch, it doesn't mean I'm trying to fill a hole left by religion - it means that my emotional energies, my caring, are bound into the real world.

If God did speak plainly, and answer prayers reliably, God would just become one more boringly real thing, no more worth believing in than the postman. If God were real, it would destroy the inner uncertainty that brings forth outward fervor in compensation. And if everyone else believed God were real, it would destroy the specialness of being one of the elect.

If you invest your emotional energy in space travel, you don't have those vulnerabilities. I can see the Space Shuttle rise without losing the awe. Everyone else can believe that Space Shuttles are real, and it doesn't make them any less special. I haven't painted myself into the corner.

The choice between God and humanity is not just a choice of drugs. Above all, humanity actually exists.

Sequence: Joy in the Merely Real

Amazing Breakthrough Day: April 1st

Scarcity

Referenced by: To Spread Science, Keep It Secret • Decoherent Essences • Getting Nearer • The Sacred Mundane • Church vs. Taskforce

Original with comments: Is Humanism A Religion-Substitute?

Scarcity

Eliezer Yudkowsky, 27 March 2008 08:07AM

What follows is taken primarily from Robert Cialdini's Influence: The Psychology of Persuasion. I own three copies of this book, one for myself, and two for loaning to friends.

Scarcity, as that term is used in social psychology, is when things become more desirable as they appear less obtainable.

- If you put a two-year-old boy in a room with two toys, one toy in the open and the other behind a Plexiglas wall, the two-year-old will ignore the easily accessible toy and go after the apparently forbidden one. If the wall is low enough to be easily climbable, the toddler is no more likely to go after one toy than the other. (Brehm and Weintraub 1977.)

- When Dade County forbade use or possession of phosphate detergents, many Dade residents drove to nearby counties and bought huge amounts of phosphate laundry detergents. Compared to Tampa residents not affected by the regulation, Dade residents rated phosphate detergents as gentler, more effective, more powerful on stains, and even believed that phosphate detergents poured more easily. (Mazis 1975, Mazis et. al. 1973.)

Similarly, information that appears forbidden or secret, seems more important and trustworthy:

- When University of North Carolina students learned that a speech opposing coed dorms had been banned, they became more opposed to coed dorms (without even hearing the speech). (Probably in Ashmore et. al. 1971.)

- When a driver said he had liability insurance, experimental jurors awarded his victim an average of four thousand dollars more than if the driver said he had no insurance. If the judge afterward informed the jurors that information about insurance was inadmissible and must be ignored, jurors awarded an average of thirteen thousand dollars more than if the driver had no insurance. (Broeder 1959.)

- Buyers for supermarkets, told by a supplier that beef was in scarce supply, gave orders for twice as much beef as buyers told it was readily available. Buyers told that beef was in scarce supply, and furthermore, that the information about scarcity was itself scarce - that the shortage was not general knowledge - ordered six times as much beef. (Since the study was conducted in a real-world context, the information provided was in fact correct.) (Knishinsky 1982.)

The conventional theory for explaining this is "psychological reactance", social-psychology-speak for "When you tell people they can't do something, they'll just try even harder." The fundamental instincts involved appear to be preservation of status and preservation of options. We resist dominance, when any human agency tries to restrict our freedom. And when options seem to be in danger of disappearing, even from natural causes, we try to leap on the option before it's gone.

Leaping on disappearing options may be a good adaptation in a hunter-gatherer society - gather the fruits while the tree is still in bloom - but in a money-based society it can be rather costly. Cialdini (1993) reports that in one appliance store he observed, a salesperson who saw that a customer was evincing signs of interest in an appliance would approach, and sadly inform the customer that the item was out of stock, the last one having been sold only twenty minutes ago. Scarcity creating a sudden jump in desirability, the customer would often ask whether there was any chance that the salesperson could locate an unsold item in the back room, warehouse, or anywhere. "Well," says the salesperson, "that's possible, and I'm willing to check; but do I understand that this is the model you want, and if I can find it at this price, you'll take it?"

As Cialdini remarks, a chief sign of this malfunction is that you dream of possessing something, rather than using it. (Timothy Ferriss offers similar advice on planning your life: ask which ongoing experiences would make you happy, rather than which possessions or status-changes.)

But the really fundamental problem with desiring the unattainable is that as soon as you actually get it, it stops being unattainable. If we cannot take joy in the merely available, our lives will always be frustrated...

Ashmore, R. D., Ramachandra, V. and Jones, R. A. (1971.) "Censorship as an Attitude Change Induction." Paper presented at Eastern Psychological Association meeting, New York, April 1971.

Brehm, S. S. and Weintraub, M. (1977.) "Physical Barriers and Psychological Reactance: Two-year-olds' Responses to Threats to Freedom." Journal of Personality and Social Psychology, 35: 830-36.

Broeder, D. (1959.) "The University of Chicago Jury Project." Nebraska Law Review 38: 760-74.

Cialdini, R. B. (1993.) Influence: The Psychology of Persuasion: Revised Edition. Pp. 237-71. New York: Quill.

Knishinsky, A. (1982.) "The Effects of Scarcity of Material and Exclusivity of Information on Industrial Buyer Perceived Risk in Provoking a Purchase Decision." Doctoral dissertation, Arizona State University.

Mazis, M. B. (1975.) "Antipollution Measures and Psychological Reactance Theory: A Field Experiment." Journal of Personality and Social Psychology 31: 654-66.

Mazis, M. B., Settle, R. B. and Leslie, D. C. (1973.) "Elimination of Phosphate Detergents and Psychological Reactance." Journal of Marketing Research 10: 390-95.

Sequence: Joy in the Merely Real

Is Humanism A Religion-Substitute?

To Spread Science, Keep It Secret

Referenced by: To Spread Science, Keep It Secret • Prolegomena to a Theory of Fun • The Sacred Mundane

Original with comments: Scarcity

To Spread Science, Keep It Secret

Eliezer Yudkowsky, 28 March 2008 05:47AM

Followup to: Joy in Discovery, Bind Yourself to Reality, Scientific Evidence, Scarcity

Sometimes I wonder if the Pythagoreans had the right idea.

Yes, I've written about how "science" is inherently public. I've written that "science" is distinguished from merely rational knowledge by the in-principle ability to reproduce scientific experiments for yourself, to know without relying on authority. I've said that "science" should be defined as the publicly accessible knowledge of humankind. I've even suggested that future generations will regard all papers not published in an open-access journal as non-science, i.e., it can't be part of the public knowledge of humankind if you make people pay to read it.

But that's only one vision of the future. In another vision, the knowledge we now call "science" is taken out of the public domain - the books and journals hidden away, guarded by mystic cults of gurus wearing robes, requiring fearsome initiation rituals for access - so that more people will actually study it.

I mean, right now, people can study science but they don't.

"Scarcity", it's called in social psychology. What appears to be in limited supply, is more highly valued. And this effect is especially strong with information - we're much more likely to try to obtain information that we believe is secret, and to value it more when we do obtain it.

With science, I think, people assume that if the information is freely available, it must not be important. So instead people join cults that have the sense to keep their Great Truths secret. The Great Truth may actually be gibberish, but it's more satisfying than coherent science, because it's secret.

Science is the great Purloined Letter of our times, left out in the open and ignored.

Sure, scientific openness helps the scientific elite. They've already been through the initiation rituals. But for the rest of the planet, science is kept secret a hundred times more effectively by making it freely available, than if its books were guarded in vaults and you had to walk over hot coals to get access. (This being a fearsome trial indeed, since the great secrets of insulation are only available to Physicist-Initiates of the Third Level.)

If scientific knowledge were hidden in ancient vaults (rather than hidden in inconvenient pay-for-access journals), at least then people would try to get into the vaults. They'd be desperate to learn science. Especially when they saw the power that Eighth Level Physicists could wield, and were told that they weren't allowed to know the explanation.

And if you tried to start a cult around oh, say, Scientology, you'd get some degree of public interest, at first. But people would very quickly start asking uncomfortable questions like "Why haven't you given a public demonstration of your Eighth Level powers, like the Physicists?" and "How come none of the Master Mathematicians seem to want to join your cult?" and "Why should I follow your Founder when he isn't an Eighth Level anything outside his own cult?" and "Why should I study your cult first, when the Dentists of Doom can do things that are so much more impressive?"

When you look at it from that perspective, the escape of math from the Pythagorean cult starts to look like a major strategic blunder for humanity.

Now, I know what you're going to say: "But science is surrounded by fearsome initiation rituals! Plus it's inherently difficult to learn! Why doesn't that count?" Because the public thinks that science is freely available, that's why. If you're allowed to learn, it must not be important enough to learn.

It's an image problem, people taking their cues from others' attitudes. Just anyone can walk into the supermarket and buy a light bulb, and nobody looks at it with awe and reverence. The physics supposedly aren't secret (even though you don't know), and there's a one-paragraph explanation in the newspaper that sounds vaguely authoritative and convincing - essentially, no one treats the lightbulb as a sacred mystery, so neither do you.

Even the simplest little things, completely inert objects like crucifixes, can become magical if everyone looks at them like they're magic. But since you're theoretically allowed to know why the light bulb works without climbing the mountain to find the remote Monastery of Electricians, there's no need to actually bother to learn.

Now, because science does in fact have initiation rituals both social and cognitive, scientists are not wholly dissatisfied with their science. The problem is that, in the present world, very few people bother to study science in the first place. Science cannot be the true Secret Knowledge, because just anyone is allowed to know it - even though, in fact, they don't.

If the Great Secret of Natural Selection, passed down from Darwin Who Is Not Forgotten, was only ever imparted to you after you paid $2000 and went through a ceremony involving torches and robes and masks and sacrificing an ox, then when you were shown the fossils, and shown the optic cable going through the retina under a microscope, and finally told the Truth, you would say "That's the most brilliant thing ever!" and be satisfied. After that, if some other cult tried to tell you it was actually a bearded man in the sky 6000 years ago, you'd laugh like hell.

And you know, it might actually be more fun to do things that way. Especially if the initiation required you to put together some of the evidence for yourself - together, or with classmates - before you could tell your Science Sensei you were ready to advance to the next level. It wouldn't be efficient, sure, but it would be fun.

If humanity had never made the mistake - never gone down the religious path, and never learned to fear anything that smacks of religion - then maybe the Ph.D. granting ceremony would involve litanies and chanting, because, hey, that's what people like. Why take the fun out of everything?

Maybe we're just doing it wrong.

And no, I'm not seriously proposing that we try to reverse the last five hundred years of openness and classify all the science secret. At least, not at the moment. Efficiency is important for now, especially in things like medical research. I'm just explaining why it is that I won't tell anyone the Secret of how the ineffable difference between blueness and redness arises from mere atoms for less than $100,000 -

Ahem! I meant to say, I'm telling you about this vision of an alternate Earth, so that you give science equal treatment with cults. So that you don't undervalue scientific truth when you learn it, just because it doesn't seem to be protected appropriately to its value. Imagine the robes and masks. Visualize yourself creeping into the vaults and stealing the Lost Knowledge of Newton. And don't be fooled by any organization that does use robes and masks, unless they also show you the data.

People seem to have holes in their minds for Esoteric Knowledge, Deep Secrets, the Hidden Truth. And I'm not even criticizing this psychology! There are deep secret esoteric hidden truths, like quantum mechanics or Bayes-structure. We've just gotten into the habit of presenting the Hidden Truth in a very unsatisfying way, wrapped up in false mundanity.

But if the holes for secret knowledge are not filled by true beliefs, they will be filled by false beliefs. There is nothing but science to learn - the emotional energy must either be invested in reality, or wasted in total nonsense, or destroyed. For myself, I think it is better to invest the emotional energy; fun should not be needlessly cast away.

Right now, we've got the worst of both worlds. Science isn't really free, because the courses are expensive and the textbooks are expensive. But the public thinks that anyone is allowed to know, so it must not be important.

Ideally, you would want to arrange things the other way around.

Sequence: Joy in the Merely Real

Scarcity

Initiation Ceremony

Referenced by: Devil's Offers • Eutopia is Scary • Building Weirdtopia

Original with comments: To Spread Science, Keep It Secret

Initiation Ceremony

Eliezer Yudkowsky, 28 March 2008 08:40PM

The torches that lit the narrow stairwell burned intensely and in the wrong color, flame like melting gold or shattered suns.
192... 193...
Brennan's sandals clicked softly on the stone steps, snicking in sequence, like dominos very slowly falling.
227... 228...
Half a circle ahead of him, a trailing fringe of dark cloth whispered down the stairs, the robed figure itself staying just out of sight.
239... 240...
Not much longer, Brennan predicted to himself, and his guess was accurate:
Sixteen times sixteen steps was the number, and they stood before the portal of glass.
The great curved gate had been wrought with cunning, humor, and close attention to indices of refraction: it warped light, bent it, folded it, and generally abused it, so that there were hints of what was on the other side (stronger light sources, dark walls) but no possible way of seeing through - unless, of course, you had the key: the counter-door, thick for thin and thin for thick, in which case the two would cancel out.
From the robed figure beside Brennan, two hands emerged, gloved in reflective cloth to conceal skin's color. Fingers like slim mirrors grasped the handles of the warped gate - handles that Brennan had not guessed; in all that distortion, shapes could only be anticipated, not seen.
"Do you want to know?" whispered the guide; a whisper nearly as loud as an ordinary voice, but not revealing the slightest hint of gender.
Brennan paused. The answer to the question seemed suspiciously, indeed extraordinarily obvious, even for ritual.

"Yes," Brennan said finally.
The guide only regarded him silently.
"Yes, I want to know," said Brennan.
"Know what, exactly?" whispered the figure.
Brennan's face scrunched up in concentration, trying to visualize the game to its end, and hoping he hadn't blown it already; until finally he fell back on the first and last resort, which is the truth:
"It doesn't matter," said Brennan, "the answer is still yes."
The glass gate parted down the middle, and slid, with only the tiniest scraping sound, into the surrounding stone.
The revealed room was lined, wall-to-wall, with figures robed and hooded in light-absorbing cloth. The straight walls were not themselves black stone, but mirrored, tiling a square grid of dark robes out to infinity in all directions; so that it seemed as if the people of some much vaster city, or perhaps the whole human kind, watched in assembly. There was a hint of moist warmth in the air of the room, the breath of the gathered: a scent of crowds.
Brennan's guide moved to the center of the square, where burned four torches of that relentless yellow flame. Brennan followed, and when he stopped, he realized with a slight shock that all the cowled hoods were now looking directly at him. Brennan had never before in his life been the focus of such absolute attention; it was frightening, but not entirely unpleasant.
"He is here," said the guide in that strange loud whisper.
The endless grid of robed figures replied in one voice: perfectly blended, exactly synchronized, so that not a single individual could be singled out from the rest, and betrayed:
"Who is absent?"
"Jakob Bernoulli," intoned the guide, and the walls replied:
"Is dead but not forgotten."
"Abraham de Moivre,"
"Is dead but not forgotten."
"Pierre-Simon Laplace,"
"Is dead but not forgotten."
"Edwin Thompson Jaynes,"
"Is dead but not forgotten."
"They died," said the guide, "and they are lost to us; but we still have each other, and the project continues."
In the silence, the guide turned to Brennan, and stretched forth a hand, on which rested a small ring of nearly transparent material.
Brennan stepped forward to take the ring -
But the hand clenched tightly shut.
"If three-fourths of the humans in this room are women," said the guide, "and three-fourths of the women and half of the men belong to the Heresy of Virtue, and I am a Virtuist, what is the probability that I am a man?"
"Two-elevenths," Brennan said confidently.
There was a moment of absolute silence.
Then a titter of shocked laughter.
The guide's whisper came again, truly quiet this time, almost nonexistent: "It's one-sixth, actually."
Brennan's cheeks were flaming so hard that he thought his face might melt off. The instinct was very strong to run out of the room and up the stairs and flee the city and change his name and start his life over again and get it right this time.
"An honest mistake is at least honest," said the guide, louder now, "and we may know the honesty by its relinquishment. If I am a Virtuist, what is the probability that I am a man?"
"One -" Brennan started to say.
Then he stopped. Again, the horrible silence.
"Just say 'one-sixth' already," stage-whispered the figure, this time loud enough for the walls to hear; then there was more laughter, not all of it kind.
Brennan was breathing rapidly and there was sweat on his forehead. If he was wrong about this, he really was going to flee the city. "Three fourths women times three fourths Virtuists is nine sixteenths female Virtuists in this room. One fourth men times one half Virtuists is two sixteenths male Virtuists. If I have only that information and the fact that you are a Virtuist, I would then estimate odds of two to nine, or a probability of two-elevenths, that you are male. Though I do not, in fact, believe the information given is correct. For one thing, it seems too neat. For another, there are an odd number of people in this room."
The hand stretched out again, and opened.
Brennan took the ring. It looked almost invisible, in the torchlight; not glass, but some material with a refractive index very close to air. The ring was warm from the guide's hand, and felt like a tiny living thing as it embraced his finger.
The relief was so great that he nearly didn't hear the cowled figures applauding.
From the robed guide came one last whisper:
"You are now a novice of the Bayesian Conspiracy."

Image: The Bayesian Master, by Erin Devereux

Sequence: Joy in the Merely Real

To Spread Science, Keep It Secret

Awww, a Zebra

Referenced by: The Failures of Eld Science • That Alien Message • The Quantum Physics Sequence • The Ultimate Source

Original with comments: Initiation Ceremony

Hand vs. Fingers

Eliezer Yudkowsky, 30 March 2008 12:36AM

Followup to: Reductionism, Explaining vs. Explaining Away, Fake Reductionism

Back to our original topic: Reductionism, which (in case you've forgotten) is part of a sequence on the Mind Projection Fallacy. There can be emotional problems in accepting reductionism, if you think that things have to be fundamental to be fun. But this position commits us to never taking joy in anything more complicated than a quark, and so I prefer to reject it.

To review, the reductionist thesis is that we use multi-level models for computational reasons, but physical reality has only a single level. If this doesn't sound familiar, please reread "Reductionism".

Today I'd like to pose the following conundrum: When you pick up a cup of water, is it your hand that picks it up?

Most people, of course, go with the naive popular answer: "Yes."

Recently, however, scientists have made a stunning discovery: It's not your hand that holds the cup, it's actually your fingers, thumb, and palm.

Yes, I know! I was shocked too. But it seems that after scientists measured the forces exerted on the cup by each of your fingers, your thumb, and your palm, they found there was no force left over - so the force exerted by your hand must be zero.

The theme here is that, if you can see how (not just know that) a higher level reduces to a lower one, they will not seem like separate things within your map; you will be able to see how silly it is to think that your fingers could be in one place, and your hand somewhere else; you will be able to see how silly it is to argue about whether it is your hand picks up the cup, or your fingers.

The operative word is "see", as in concrete visualization. Imagining your hand causes you to imagine the fingers and thumb and palm; conversely, imagining fingers and thumb and palm causes you to identify a hand in the mental picture. Thus the high level of your map and the low level of your map will be tightly bound together in your mind.

In reality, of course, the levels are bound together even tighter than that - bound together by the tightest possible binding: physical identity. You can see this: You can see that saying (1) "hand" or (2) "fingers and thumb and palm", does not refer to different things, but different points of view.

But suppose you lack the knowledge to so tightly bind together the levels of your map. For example, you could have a "hand scanner" that showed a "hand" as a dot on a map (like an old-fashioned radar display), and similar scanners for fingers/thumbs/palms; then you would see a cluster of dots around the hand, but you would be able to imagine the hand-dot moving off from the others. So, even though the physical reality of the hand (that is, the thing the dot corresponds to) was identical with / strictly composed of the physical realities of the fingers and thumb and palm, you would not be able to see this fact; even if someone told you, or you guessed from the correspondence of the dots, you would only know the fact of reduction, not see it. You would still be able to imagine the hand dot moving around independently, even though, if the physical makeup of the sensors were held constant, it would be physically impossible for this to actually happen.

Or, at a still lower level of binding, people might just tell you "There's a hand over there, and some fingers over there" - in which case you would know little more than a Good-Old-Fashioned AI representing the situation using suggestively named LISP tokens. There wouldn't be anything obviously contradictory about asserting:

|- Inside(Room,Hand)
|- ~Inside(Room,Fingers)

because you would not possess the knowledge

|- Inside(x, Hand) -> Inside(x,Fingers)

None of this says that a hand can actually detach its existence from your fingers and crawl, ghostlike, across the room; it just says that a Good-Old-Fashioned AI with a propositional representation may not know any better. The map is not the territory.

In particular, you shouldn't draw too many conclusions from how it seems conceptually possible, in the mind of some specific conceiver, to separate the hand from its constituent elements of fingers, thumb, and palm. Conceptual possibility is not the same as logical possibility or physical possibility.

It is conceptually possible to you that 235757 is prime, because you don't know any better. But it isn't logically possible that 235757 is prime; if you were logically omniscient, 235757 would be obviously composite (and you would know the factors). That that's why we have the notion of impossible possible worlds, so that we can put probability distributions on propositions that may or may not be in fact logically impossible.

And you can imagine philosophers who criticize "eliminative fingerists" who contradict the direct facts of experience - we can feel our hand holding the cup, after all - by suggesting that "hands" don't really exist, in which case, obviously, the cup would fall down. And philosophers who suggest "appendigital bridging laws" to explain how a particular configuration of fingers, evokes a hand into existence - with the note, of course, that while our world contains those particular appendigital bridging laws, the laws could have been conceivably different, and so are not in any sense necessary facts, etc.

All of these are cases of Mind Projection Fallacy, and what I call "naive philosophical realism" - the confusion of philosophical intuitions for direct, veridical information about reality. Your inability to imagine something is just a computational fact about what your brain can or can't imagine. Another brain might work differently.

Sequence: Reductionism

Savanna Poets

Angry Atoms

Referenced by: Angry Atoms • Reductive Reference • Zombies! Zombies? • The Rhythm of Disagreement • Principles of Disagreement • Thou Art Physics • Causality and Moral Responsibility • The Ultimate Source • Passing the Recursive Buck • Heading Toward Morality • Moral Error and Moral Disagreement

Original with comments: Hand vs. Fingers

Angry Atoms

Eliezer Yudkowsky, 31 March 2008 12:28AM

Followup to: Hand vs. Fingers

Fundamental physics - quarks 'n stuff - is far removed from the levels we can see, like hands and fingers. At best, you can know how to replicate the experiments which show that your hand (like everything else) is composed of quarks, and you may know how to derive a few equations for things like atoms and electron clouds and molecules.

At worst, the existence of quarks beneath your hand may just be something you were told. In which case it's questionable in one what sense you can be said to "know" it at all, even if you repeat back the same word "quark" that a physicist would use to convey knowledge to another physicist.

Either way, you can't actually see the identity between levels - no one has a brain large enough to visualize avogadros of quarks and recognize a hand-pattern in them.

But we at least understand what hands do. Hands push on things, exert forces on them. When we're told about atoms, we visualize little billiard balls bumping into each other. This makes it seem obvious that "atoms" can push on things too, by bumping into them.

Now this notion of atoms is not quite correct. But so far as human imagination goes, it's relatively easy to imagine our hand being made up of a little galaxy of swirling billiard balls, pushing on things when our "fingers" touch them. Democritus imagined this 2400 years ago, and there was a time, roughly 1803-1922, when Science thought he was right.

But what about, say, anger?

How could little billiard balls be angry? Tiny frowny faces on the billiard balls?

Put yourself in the shoes of, say, a hunter-gatherer - someone who may not even have a notion of writing, let alone the notion of using base matter to perform computations - someone who has no idea that such a thing as neurons exist. Then you can imagine the functional gap that your ancestors might have perceived between billiard balls and "Grrr! Aaarg!"

Forget about subjective experience for the moment, and consider the sheer behavioral gap between anger and billiard balls. The difference between what little billiard balls do, and what anger makes people do. Anger can make people raise their fists and hit someone - or say snide things behind their backs - or plant scorpions in their tents at night. Billiard balls just push on things.

Try to put yourself in the shoes of the hunter-gatherer who's never had the "Aha!" of information-processing. Try to avoid hindsight bias about things like neurons and computers. Only then will you be able to see the uncrossable explanatory gap:

How can you explain angry behavior in terms of billiard balls?

Well, the obvious materialist conjecture is that the little billiard balls push on your arm and make you hit someone, or push on your tongue so that insults come out.

But how do the little billiard balls know how to do this - or how to guide your tongue and fingers through long-term plots - if they aren't angry themselves?

And besides, if you're not seduced by - gasp! - scientism, you can see from a first-person perspective that this explanation is obviously false. Atoms can push on your arm, but they can't make you want anything.

Someone may point out that drinking wine can make you angry. But who says that wine is made exclusively of little billiard balls? Maybe wine just contains a potency of angerness.

Clearly, reductionism is just a flawed notion.

(The novice goes astray and says "The art failed me"; the master goes astray and says "I failed my art.")

What does it take to cross this gap? It's not just the idea of "neurons" that "process information" - if you say only this and nothing more, it just inserts a magical, unexplained level-crossing rule into your model, where you go from billiards to thoughts.

But an Artificial Intelligence programmer who knows how to create a chess-playing program out of base matter, has taken a genuine step toward crossing the gap. If you understand concepts like consequentialism, backward chaining, utility functions, and search trees, you can make merely causal/mechanical systems compute plans.

The trick goes something like this: For each possible chess move, compute the moves your opponent could make, then your responses to those moves, and so on; evaluate the furthest position you can see using some local algorithm (you might simply count up the material); then trace back using minimax to find the best move on the current board; then make that move.

More generally: If you have chains of causality inside the mind that have a kind of mapping - a mirror, an echo - to what goes on in the environment, then you can run a utility function over the end products of imagination, and find an action that achieves something which the utility function rates highly, and output that action. It is not necessary for the chains of causality inside the mind, that are similar to the environment, to be made out of billiard balls that have little auras of intentionality. Deep Blue's transistors do not need little chess pieces carved on them, in order to work. See also The Simple Truth.

All this is still tremendously oversimplified, but it should, at least, reduce the apparent length of the gap. If you can understand all that, you can see how a planner built out of base matter can be influenced by alcohol to output more angry behaviors. The billiard balls in the alcohol push on the billiard balls making up the utility function.

But even if you know how to write small AIs, you can't visualize the level-crossing between transistors and chess. There are too many transistors, and too many moves to check.

Likewise, even if you knew all the facts of neurology, you would not be able to visualize the level-crossing between neurons and anger - let alone the level-crossing between atoms and anger. Not the way you can visualize a hand consisting of fingers, thumb, and palm.

And suppose a cognitive scientist just flatly tells you "Anger is hormones"? Even if you repeat back the words, it doesn't mean you've crossed the gap. You may believe you believe it, but that's not the same as understanding what little billiard balls have to do with wanting to hit someone.

So you come up with interpretations like, "Anger is mere hormones, it's caused by little molecules, so it must not be justified in any moral sense - that's why you should learn to control your anger."

Or, "There isn't really any such thing as anger - it's an illusion, a quotation with no referent, like a mirage of water in the desert, or looking in the garage for a dragon and not finding one."

These are both tough pills to swallow (not that you should swallow them) and so it is a good easier to profess them than to believe them.

I think this is what non-reductionists/non-materialists think they are criticizing when they criticize reductive materialism.

But materialism isn't that easy. It's not as cheap as saying, "Anger is made out of atoms - there, now I'm done." That wouldn't explain how to get from billiard balls to hitting. You need the specific insights of computation, consequentialism, and search trees before you can start to close the explanatory gap.

All this was a relatively easy example by modern standards, because I restricted myself to talking about angry behaviors. Talking about outputs doesn't require you to appreciate how an algorithm feels from inside (cross a first-person/third-person gap) or dissolve a wrong question (untangle places where the interior of your own mind runs skew to reality).

Going from material substances that bend and break, burn and fall, push and shove, to angry behavior, is just a practice problem by the standards of modern philosophy. But it is an important practice problem. It can only be fully appreciated, if you realize how hard it would have been to solve before writing was invented. There was once an explanatory gap here - though it may not seem that way in hindsight, now that it's been bridged for generations.

Explanatory gaps can be crossed, if you accept help from science, and don't trust the view from the interior of your own mind.

Sequence: Reductionism

Hand vs. Fingers

Heat vs. Motion

Referenced by: Heat vs. Motion • Bell's Theorem: No EPR "Reality" • A Premature Word on AI • Thou Art Physics • Setting Up Metaethics • When Anthropomorphism Became Stupid • Raising the Sanity Waterline

Original with comments: Angry Atoms

Heat vs. Motion

Eliezer Yudkowsky, 01 April 2008 03:55AM

Followup to: Angry Atoms

After yesterday's post, it occurred to me that there's a much simpler example of reductionism jumping a gap of apparent-difference-in-kind: the reduction of heat to motion.

Today, the equivalence of heat and motion may seem too obvious in hindsight - everyone says that "heat is motion", therefore, it can't be a "weird" belief.

But there was a time when the kinetic theory of heat was a highly controversial scientific hypothesis, contrasting to belief in a caloric fluid that flowed from hot objects to cold objects. Still earlier, the main theory of heat was "Phlogiston!"

Suppose you'd separately studied kinetic theory and caloric theory. You now know something about kinetics: collisions, elastic rebounds, momentum, kinetic energy, gravity, inertia, free trajectories. Separately, you know something about heat: Temperatures, pressures, combustion, heat flows, engines, melting, vaporization.

Not only is this state of knowledge a plausible one, it is the state of knowledge possessed by e.g. Sadi Carnot, who, working strictly from within the caloric theory of heat, developed the principle of the Carnot cycle - a heat engine of maximum efficiency, whose existence implies the second law of thermodynamics. This in 1824, when kinetics was a highly developed science.

Suppose, like Carnot, you know a great deal about kinetics, and a great deal about heat, as separate entities. Separate entities of knowledge, that is: your brain has separate filing baskets for beliefs about kinetics and beliefs about heat. But from the inside, this state of knowledge feels like living in a world of moving things and hot things, a world where motion and heat are independent properties of matter.

Now a Physicist From The Future comes along and tells you: "Where there is heat, there is motion, and vice versa. That's why, for example, rubbing things together makes them hotter."

There are (at least) two possible interpretations you could attach to this statement, "Where there is heat, there is motion, and vice versa."

First, you could suppose that heat and motion exist separately - that the caloric theory is correct - but that among our universe's physical laws is a "bridging law" which states that, where objects are moving quickly, caloric will come into existence. And conversely, another bridging law says that caloric can exert pressure on things and make them move, which is why a hotter gas exerts more pressure on its enclosure (thus a steam engine can use steam to drive a piston).

Second, you could suppose that heat and motion are, in some as-yet-mysterious sense, the same thing.

"Nonsense," says Thinker 1, "the words 'heat' and 'motion' have two different meanings; that is why we have two different words. We know how to determine when we will call an observed phenomenon 'heat' - heat can melt things, or make them burst into flame. We know how to determine when we will say that an object is 'moving quickly' - it changes position; and when it crashes, it may deform, or shatter. Heat is concerned with change of substance; motion, with change of position and shape. To say that these two words have the same meaning is simply to confuse yourself."

"Impossible," says Thinker 2. "It may be that, in our world, heat and motion are associated by bridging laws, so that it is a law of physics that motion creates caloric, and vice versa. But I can easily imagine a world where rubbing things together does not make them hotter, and gases don't exert more pressure at higher temperatures. Since there are possible worlds where heat and motion are not associated, they must be different properties - this is true a priori."

Thinker 1 is confusing the quotation and the referent. 2 + 2 = 4, but "2 + 2" â‰  "4". The string "2 + 2" contains 5 characters (including whitespace) and the string "4" contains only 1 character. If you type the two strings into a Python interpreter, they yield the same output, -> 4. So you can't conclude, from looking at the strings "2 + 2" and "4", that just because the strings are different, they must have different "meanings" relative to the Python Interpreter.

The words "heat" and "kinetic energy" can be said to "refer to" the same thing, even before we know how heat reduces to motion, in the sense that we don't know yet what the reference is, but the references are in fact the same. You might imagine an Idealized Omniscient Science Interpreter that would give the same output when we typed in "heat" and "kinetic energy" on the command line.

I talk about the Science Interpreter to emphasize that, to dereference the pointer, you've got to step outside cognition. The end result of the dereference is something out there in reality, not in anyone's mind. So you can say "real referent" or "actual referent", but you can't evaluate the words locally, from the inside of your own head. You can't reason using the actual heat-referent - if you thought using real heat, thinking "1 million Kelvin" would vaporize your brain. But, by forming a belief about your belief about heat, you can talk about your belief about heat, and say things like "It's possible that my belief about heat doesn't much resemble real heat." You can't actually perform that comparison right there in your own mind, but you can talk about it.

Hence you can say, "My beliefs about heat and motion are not the same beliefs, but it's possible that actual heat and actual motion are the same thing." It's just like being able to acknowledge that "the morning star" and "the evening star" might be the same planet, while also understanding that you can't determine this just by examining your beliefs - you've got to haul out the telescope.

Thinker 2's mistake follows similarly. A physicist told him, "Where there is heat, there is motion" and P2 mistook this for a statement of physical law: The presence of caloric causes the existence of motion. What the physicist really means is more akin to an inferential rule: Where you are told there is "heat", deduce the presence of "motion".

From this basic projection of a multilevel model into a multilevel reality follows another, distinct error: the conflation of conceptual possibility with logical possibility. To Sadi Carnot, it is conceivable that there could be another world where heat and motion are not associated. To Richard Feynman, armed with specific knowledge of how to derive equations about heat from equations about motion, this idea is not only inconceivable, but so wildly inconsistent as to make one's head explode.

I should note, in fairness to philosophers, that there are philosophers who have said these things. For example, Hilary Putnam, writing on the "Twin Earth" thought experiment:

Once we have discovered that water (in the actual world) is H20, nothing counts as a possible world in which water isn't H20. In particular, if a "logically possible" statement is one that holds in some "logically possible world", it isn't logically possible that water isn't H20.

On the other hand, we can perfectly well imagine having experiences that would convince us (and that would make it rational to believe that) water isn't H20. In that sense, it is conceivable that water isn't H20. It is conceivable but it isn't logically possible! Conceivability is no proof of logical possibility.

It appears to me that "water" is being used in two different senses in these two paragraphs - one in which the word "water" refers to what we type into the Science Interpreter, and one in which "water" refers to what we get out of the Science Interpreter when we type "water" into it. In the first paragraph, Hilary seems to be saying that after we do some experiments and find out that water is H20, water becomes automatically redefined to mean H20. But you could coherently hold a different position about whether the word "water" now means "H20" or "whatever is really in that bottle next to me", so long as you use your terms consistently.

I believe the above has already been said as well? Anyway...

It is quite possible for there to be only one thing out-there-in-the-world, but for it to take on sufficiently different forms, and for you yourself to be sufficiently ignorant of the reduction, that it feels like living in a world containing two entirely different things. Knowledge concerning these two different phenomena may taught in two different classes, and studied by two different academic fields, located in two different buildings of your university.

You've got to put yourself quite a ways back, into a historically realistic frame of mind, to remember how different heat and motion once seemed. Though, depending on how much you know today, it may not be as hard as all that, if you can look past the pressure of conventionality (that is, "heat is motion" is an un-weird belief, "heat is not motion" is a weird belief). I mean, suppose that tomorrow the physicists stepped forward and said, "Our popularizations of science have always contained one lie. Actually, heat has nothing to do with motion." Could you prove they were wrong?

Saying "Maybe heat and motion are the same thing!" is easy. The difficult part is explaining how. It takes a great deal of detailed knowledge to get yourself to the point where you can no longer conceive of a world in which the two phenomena go separate ways. Reduction isn't cheap, and that's why it buys so much.

Or maybe you could say: "Reductionism is easy, reduction is hard." But it does kinda help to be a reductionist, I think, when it comes time to go looking for a reduction.

Sequence: Reductionism

Angry Atoms

Brain Breakthrough! It's Made of Neurons!

Referenced by: Reductive Reference • Heading Toward Morality

Original with comments: Heat vs. Motion

Brain Breakthrough! It's Made of Neurons!

Eliezer Yudkowsky, 01 April 2008 07:00PM

In an amazing breakthrough, a multinational team of scientists led by Nobel laureate Santiago Ramón y Cajal announced that the brain is composed of a ridiculously complicated network of tiny cells connected to each other by infinitesimal threads and branches.

The multinational team - which also includes the famous technician Antonie van Leeuwenhoek, and possibly Imhotep, promoted to the Egyptian god of medicine - issued this statement:

"The present discovery culminates years of research indicating that the convoluted squishy thing inside our skulls is even more complicated than it looks. Thanks to Cajal's application of a new staining technique invented by Camillo Golgi, we have learned that this structure is not a continuous network like the blood vessels of the body, but is actually composed of many tiny cells, or "neurons", connected to one another by even more tiny filaments.

"Other extensive evidence, beginning from Greek medical researcher Alcmaeon and continuing through Paul Broca's research on speech deficits, indicates that the brain is the seat of reason.

"Nemesius, the Bishop of Emesia, has previously argued that brain tissue is too earthy to act as an intermediary between the body and soul, and so the mental faculties are located in the ventricles of the brain. However, if this is correct, there is no reason why this organ should turn out to have an immensely complicated internal composition.

"Charles Babbage has independently suggested that many small mechanical devices could be collected into an 'Analytical Engine', capable of performing activities, such as arithmetic, which are widely believed to require thought. The work of Luigi Galvani and Hermann von Helmholtz suggests that the activities of neurons are electrochemical in nature, rather than mechanical pressures as previously believed. Nonetheless, we think an analogy with Babbage's 'Analytical Engine' suggests that a vastly complicated network of neurons could similarly exhibit thoughtful properties.

"We have found an enormously complicated material system located where the mind should be. The implications are shocking, and must be squarely faced. We believe that the present research offers strong experimental evidence that Benedictus Spinoza was correct, and René Descartes wrong: Mind and body are of one substance.

"In combination with the work of Charles Darwin showing how such a complicated organ could, in principle, have arisen as the result of processes not themselves intelligent, the bulk of scientific evidence now seems to indicate that intelligence is ontologically non-fundamental and has an extended origin in time. This strongly weighs against theories which assign mental entities an ontologically fundamental or causally primal status, including all religions ever invented.

"Much work remains to be done on discovering the specific identities between electrochemical interactions between neurons, and thoughts. Nonetheless, we believe our discovery offers the promise, though not yet the realization, of a full scientific account of thought. The problem may now be declared, if not solved, then solvable."

We regret that Cajal and most of the other researchers involved on the Project are no longer available for comment.

Sequence: Reductionism

Heat vs. Motion

Reductive Reference

Referenced by: Bell's Theorem: No EPR "Reality" • Timeless Physics • When Anthropomorphism Became Stupid • Striving to Accept • Raising the Sanity Waterline

Original with comments: Brain Breakthrough! It's Made of Neurons!

Reductive Reference

Eliezer Yudkowsky, 03 April 2008 01:37AM

Followup to: Reductionism, Explaining vs. Explaining Away, Hand vs. Fingers, Heat vs. Motion

The reductionist thesis (as I formulate it) is that human minds, for reasons of efficiency, use a multi-level map in which we separately think about things like "atoms" and "quarks", "hands" and "fingers", or "heat" and "kinetic energy". Reality itself, on the other hand, is single-level in the sense that it does not seem to contain atoms as separate, additional, causally efficacious entities over and above quarks.

Sadi Carnot formulated the (precursor to) the second law of thermodynamics using the caloric theory of heat, in which heat was just a fluid that flowed from hot things to cold things, produced by fire, making gases expand - the effects of heat were studied separately from the science of kinetics, considerably before the reduction took place. If you're trying to design a steam engine, the effects of all those tiny vibrations and collisions which we name "heat" can be summarized into a much simpler description than the full quantum mechanics of the quarks. Humans compute efficiently, thinking of only significant effects on goal-relevant quantities.

But reality itself does seem to use the full quantum mechanics of the quarks. I once met a fellow who thought that if you used General Relativity to compute a low-velocity problem, like an artillery shell, GR would give you the wrong answer - not just a slow answer, but an experimentally wrong answer - because at low velocities, artillery shells are governed by Newtonian mechanics, not GR. This is exactly how physics does not work. Reality just seems to go on crunching through General Relativity, even when it only makes a difference at the fourteenth decimal place, which a human would regard as a huge waste of computing power. Physics does it with brute force. No one has ever caught physics simplifying its calculations - or if someone did catch it, the Matrix Lords erased the memory afterward.

Our map, then, is very much unlike the territory; our maps are multi-level, the territory is single-level. Since the representation is so incredibly unlike the referent, in what sense can a belief like "I am wearing socks" be called true, when in reality itself, there are only quarks?

In case you've forgotten what the word "true" means, the classic definition was given by Alfred Tarski:

The statement "snow is white" is true if and only if snow is white.

In case you've forgotten what the difference is between the statement "I believe 'snow is white'" and "'Snow is white' is true", see here. Truth can't be evaluated just by looking inside your own head - if you want to know, for example, whether "the morning star = the evening star", you need a telescope; it's not enough just to look at the beliefs themselves.

This is the point missed by the postmodernist folks screaming, "But how do you know your beliefs are true?" When you do an experiment, you actually are going outside your own head. You're engaging in a complex interaction whose outcome is causally determined by the thing you're reasoning about, not just your beliefs about it. I once defined "reality" as follows:

Even when I have a simple hypothesis, strongly supported by all the evidence I know, sometimes I'm still surprised. So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies 'belief', and the latter thingy 'reality'."

The interpretation of your experiment still depends on your prior beliefs. I'm not going to talk, for the moment, about Where Priors Come From, because that is not the subject of this blog post. My point is that truth refers to an ideal comparison between a belief and reality. Because we understand that planets are distinct from beliefs about planets, we can design an experiment to test whether the belief "the morning star and the evening star are the same planet" is true. This experiment will involve telescopes, not just introspection, because we understand that "truth" involves comparing an internal belief to an external fact; so we use an instrument, the telescope, whose perceived behavior we believe to depend on the external fact of the planet.

Believing that the telescope helps us evaluate the "truth" of "morning star = evening star", relies on our prior beliefs about the telescope interacting with the planet. Again, I'm not going to address that in this particular blog post, except to quote one of my favorite Raymond Smullyan lines: "If the more sophisticated reader objects to this statement on the grounds of its being a mere tautology, then please at least give the statement credit for not being inconsistent." Similarly, I don't see the use of a telescope as circular logic, but as reflective coherence; for every systematic way of arriving at truth, there ought to be a rational explanation for how it works.

The question on the table is what it means for "snow is white" to be true, when, in reality, there are just quarks.

There's a certain pattern of neural connections making up your beliefs about "snow" and "whiteness" - we believe this, but we do not know, and cannot concretely visualize, the actual neural connections. Which are, themselves, embodied in a pattern of quarks even less known. Out there in the world, there are water molecules whose temperature is low enough that they have arranged themselves in tiled repeating patterns; they look nothing like the tangles of neurons. In what sense, comparing one (ever-fluctuating) pattern of quarks to the other, is the belief "snow is white" true?

Obviously, neither I nor anyone else can offer an Ideal Quark Comparer Function that accepts a quark-level description of a neurally embodied belief (including the surrounding brain) and a quark-level description of a snowflake (and the surrounding laws of optics), and outputs "true" or "false" over "snow is white". And who says the fundamental level is really about particle fields?

On the other hand, throwing out all beliefs because they aren't written as gigantic unmanageable specifications about quarks we can't even see... doesn't seem like a very prudent idea. Not the best way to optimize our goals.

It seems to me that a word like "snow" or "white" can be taken as a kind of promissory note - not a known specification of exactly which physical quark configurations count as "snow", but, nonetheless, there are things you call snow and things you don't call snow, and even if you got a few items wrong (like plastic snow), an Ideal Omniscient Science Interpreter would see a tight cluster in the center and redraw the boundary to have a simpler definition.

In a single-layer universe whose bottom layer is unknown, or uncertain, or just too large to talk about, the concepts in a multi-layer mind can be said to represent a kind of promissory note - we don't know what they correspond to, out there. But it seems to us that we can distinguish positive from negative cases, in a predictively productive way, so we think - perhaps in a fully general sense - that there is some difference of quarks, some difference of configurations at the fundamental level, which explains the differences that feed into our senses, and ultimately result in our saying "snow" or "not snow".

I see this white stuff, and it is the same on several occasions, so I hypothesize a stable latent cause in the environment - I give it the name "snow"; "snow" is then a promissory note referring to a believed-in simple boundary that could be drawn around the unseen causes of my experience.

Hilary Putnam's "Twin Earth" thought experiment, where water is not H20 but some strange other substance denoted XYZ, otherwise behaving much like water, and the subsequent philosophical debate, helps to highlight this issue. "Snow" doesn't have a logical definition known to us - it's more like an empirically determined pointer to a logical definition. This is true even if you believe that snow is ice crystals is low-temperature tiled water molecules. The water molecules are made of quarks. What if quarks turn out to be made of something else? What is a snowflake, then? You don't know - but it's still a snowflake, not a fire hydrant.

And of course, these very paragraphs I have just written, are likewise far above the level of quarks. "Sensing white stuff, visually categorizing it, and thinking 'snow' or 'not snow'" - this is also talking very far above the quarks. So my meta-beliefs are also promissory notes, for things that an Ideal Omniscient Science Interpreter might know about which configurations of the quarks (or whatever) making up my brain, correspond to "believing 'snow is white'".

But then, the entire grasp that we have upon reality, is made up of promissory notes of this kind. So, rather than calling it circular, I prefer to call it self-consistent.

This can be a bit unnerving - maintaining a precarious epistemic perch, in both object-level beliefs and reflection, far above a huge unknown underlying fundamental reality, and hoping one doesn't fall off.

On reflection, though, it's hard to see how things could be any other way.

So at the end of the day, the statement "reality does not contain hands as fundamental, additional, separate causal entities, over and above quarks" is not the same statement as "hands do not exist" or "I don't have any hands". There are no fundamental hands; hands are made of fingers, palm, and thumb, which in turn are made of muscle and bone, all the way down to elementary particle fields, which are the fundamental causal entities, so far as we currently know.

This is not the same as saying, "there are no 'hands'." It is not the same as saying, "the word 'hands' is a promissory note that will never be paid, because there is no empirical cluster that corresponds to it"; or "the 'hands' note will never be paid, because it is logically impossible to reconcile its supposed characteristics"; or "the statement 'humans have hands' refers to a sensible state of affairs, but reality is not in that state".

Just: There are patterns that exist in reality where we see "hands", and these patterns have something in common, but they are not fundamental.

If I really had no hands - if reality suddenly transitioned to be in a state that we would describe as "Eliezer has no hands" - reality would shortly thereafter correspond to a state we would describe as "Eliezer screams as blood jets out of his wrist stumps".

And this is true, even though the above paragraph hasn't specified any quark positions.

The previous sentence is likewise meta-true.

The map is multilevel, the territory is single-level. This doesn't mean that the higher levels "don't exist", like looking in your garage for a dragon and finding nothing there, or like seeing a mirage in the desert and forming an expectation of drinkable water when there is nothing to drink. The higher levels of your map are not false, without referent; they have referents in the single level of physics. It's not that the wings of an airplane unexist - then the airplane would drop out of the sky. The "wings of an airplane" exist explicitly in an engineer's multilevel model of an airplane, and the wings of an airplane exist implicitly in the quantum physics of the real airplane. Implicit existence is not the same as nonexistence. The exact description of this implicitness is not known to us - is not explicitly represented in our map. But this does not prevent our map from working, or even prevent it from being true.

Though it is a bit unnerving to contemplate that every single concept and belief in your brain, including these meta-concepts about how your brain works and why you can form accurate beliefs, are perched orders and orders of magnitude above reality...

Sequence: Reductionism

Brain Breakthrough! It's Made of Neurons!

Excluding the Supernatural

Referenced by: Zombies! Zombies? • Zombie Responses • The Generalized Anti-Zombie Principle • Where Philosophy Meets Science

Original with comments: Reductive Reference

Zombies! Zombies?

Eliezer Yudkowsky, 04 April 2008 09:55AM

Your "zombie", in the philosophical usage of the term, is putatively a being that is exactly like you in every respect - identical behavior, identical speech, identical brain; every atom and quark in exactly the same position, moving according to the same causal laws of motion - except that your zombie is not conscious.

It is furthermore claimed that if zombies are "possible" (a term over which battles are still being fought), then, purely from our knowledge of this "possibility", we can deduce a priori that consciousness is extra-physical, in a sense to be described below; the standard term for this position is "epiphenomenalism".

(For those unfamiliar with zombies, I emphasize that this is not a strawman. See, for example, the SEP entry on Zombies. The "possibility" of zombies is accepted by a substantial fraction, possibly a majority, of academic philosophers of consciousness.)

I once read somewhere, "You are not the one who speaks your thoughts - you are the one who hears your thoughts". In Hebrew, the word for the highest soul, that which God breathed into Adam, is N'Shama - "the hearer".

If you conceive of "consciousness" as a purely passive listening, then the notion of a zombie initially seems easy to imagine. It's someone who lacks the N'Shama, the hearer.

(Warning: Long post ahead. Very long 6,600-word post involving David Chalmers ahead. This may be taken as my demonstrative counterexample to Richard Chappell's Arguing with Eliezer Part II, in which Richard accuses me of not engaging with the complex arguments of real philosophers.)

When you open a refrigerator and find that the orange juice is gone, you think "Darn, I'm out of orange juice." The sound of these words is probably represented in your auditory cortex, as though you'd heard someone else say it. (Why do I think this? Because native Chinese speakers can remember longer digit sequences than English-speakers. Chinese digits are all single syllables, and so Chinese speakers can remember around ten digits, versus the famous "seven plus or minus two" for English speakers. There appears to be a loop of repeating sounds back to yourself, a size limit on working memory in the auditory cortex, which is genuinely phoneme-based.)

Let's suppose the above is correct; as a postulate, it should certainly present no problem for advocates of zombies. Even if humans are not like this, it seems easy enough to imagine an AI constructed this way (and imaginability is what the zombie argument is all about). It's not only conceivable in principle, but quite possible in the next couple of decades, that surgeons will lay a network of neural taps over someone's auditory cortex and read out their internal narrative. (Researchers have already tapped the lateral geniculate nucleus of a cat and reconstructed recognizable visual inputs.)

So your zombie, being physically identical to you down to the last atom, will open the refrigerator and form auditory cortical patterns for the phonemes "Darn, I'm out of orange juice". On this point, epiphenomalists would willingly agree.

But, says the epiphenomenalist, in the zombie there is no one inside to hear; the inner listener is missing. The internal narrative is spoken, but unheard. You are not the one who speaks your thoughts, you are the one who hears them.

It seems a lot more straightforward (they would say) to make an AI that prints out some kind of internal narrative, than to show that an inner listener hears it.

The Zombie Argument is that if the Zombie World is possible - not necessarily physically possible in our universe, just "possible in theory", or "imaginable", or something along those lines - then consciousness must be extra-physical, something over and above mere atoms. Why? Because even if you somehow knew the positions of all the atoms in the universe, you would still have be told, as a separate and additional fact, that people were conscious - that they had inner listeners - that we were not in the Zombie World, as seems possible.

Zombie-ism is not the same as dualism. Descartes thought there was a body-substance and a wholly different kind of mind-substance, but Descartes also thought that the mind-substance was a causally active principle, interacting with the body-substance, controlling our speech and behavior. Subtracting out the mind-substance from the human would leave a traditional zombie, of the lurching and groaning sort.

And though the Hebrew word for the innermost soul is N'Shama, that-which-hears, I can't recall hearing a rabbi arguing for the possibility of zombies. Most rabbis would probably be aghast at the idea that the divine part which God breathed into Adam doesn't actually do anything.

The technical term for the belief that consciousness is there, but has no effect on the physical world, is epiphenomenalism.

Though there are other elements to the zombie argument (I'll deal with them below), I think that the intuition of the passive listener is what first seduces people to zombie-ism. In particular, it's what seduces a lay audience to zombie-ism. The core notion is simple and easy to access: The lights are on but no one's home.

Philosophers are appealing to the intuition of the passive listener when they say "Of course the zombie world is imaginable; you know exactly what it would be like."

One of the great battles in the Zombie Wars is over what, exactly, is meant by saying that zombies are "possible". Early zombie-ist philosophers (the 1970s) just thought it was obvious that zombies were "possible", and didn't bother to define what sort of possibility was meant.

Because of my reading in mathematical logic, what instantly comes into my mind is logical possibility. If you have a collection of statements like (A->B),(B->C),(C->~A) then the compound belief is logically possible if it has a model - which, in the simple case above, reduces to finding a value assignment to A, B, C that makes all of the statements (A->B),(B->C), and (C->~A) true. In this case, A=B=C=0 works, as does A=0, B=C=1 or A=B=0, C=1.

Something will seem possible - will seem "conceptually possible" or "imaginable" - if you can consider the collection of statements without seeing a contradiction. But it is, in general, a very hard problem to see contradictions or to find a full specific model! If you limit yourself to simple Boolean propositions of the form ((A or B or C) and (B or ~C or D) and (D or ~A or ~C) ...), conjunctions of disjunctions of three variables, then this is a very famous problem called 3-SAT, which is one of the first problems ever to be proven NP-complete.

So just because you don't see a contradiction in the Zombie World at first glance, it doesn't mean that no contradiction is there. It's like not seeing a contradiction in the Riemann Hypothesis at first glance. From conceptual possibility ("I don't see a problem") to logical possibility in the full technical sense, is a very great leap. It's easy to make it an NP-complete leap, and with first-order theories you can make it superexponential. And it's logical possibility of the Zombie World, not conceptual possibility, that is needed to suppose that a logically omniscient mind could know the positions of all the atoms in the universe, and yet need to be told as an additional non-entailed fact that we have inner listeners.

Just because you don't see a contradiction yet, is no guarantee that you won't see a contradiction in another 30 seconds. "All odd numbers are prime. Proof: 3 is prime, 5 is prime, 7 is prime..."

So let us ponder the Zombie Argument a little longer: Can we think of a counterexample to the assertion "Consciousness has no third-party-detectable causal impact on the world"?

If you close your eyes and concentrate on your inward awareness, you will begin to form thoughts, in your internal narrative, that go along the lines of "I am aware" and "My awareness is separate from my thoughts" and "I am not the one who speaks my thoughts, but the one who hears them" and "My stream of consciousness is not my consciousness" and "It seems like there is a part of me which I can imagine being eliminated without changing my outward behavior."

You can even say these sentences out loud, as you meditate. In principle, someone with a super-fMRI could probably read the phonemes out of your auditory cortex; but saying it out loud removes all doubt about whether you have entered the realms of testability and physical consequences.

This certainly seems like the inner listener is being caught in the act of listening by whatever part of you writes the internal narrative and flaps your tongue.

Imagine that a mysterious race of aliens visit you, and leave you a mysterious black box as a gift. You try poking and prodding the black box, but (as far as you can tell) you never succeed in eliciting a reaction. You can't make the black box produce gold coins or answer questions. So you conclude that the black box is causally inactive: "For all X, the black box doesn't do X." The black box is an effect, but not a cause; epiphenomenal; without causal potency. In your mind, you test this general hypothesis to see if it is true in some trial cases, and it seems to be true - "Does the black box turn lead to gold? No. Does the black box boil water? No."

But you can see the black box; it absorbs light, and weighs heavy in your hand. This, too, is part of the dance of causality. If the black box were wholly outside the causal universe, you couldn't see it; you would have no way to know it existed; you could not say, "Thanks for the black box." You didn't think of this counterexample, when you formulated the general rule: "All X: Black box doesn't do X". But it was there all along.

(Actually, the aliens left you another black box, this one purely epiphenomenal, and you haven't the slightest clue that it's there in your living room. That was their joke.)

If you can close your eyes, and sense yourself sensing - if you can be aware of yourself being aware, and think "I am aware that I am aware" - and say out loud, "I am aware that I am aware" - then your consciousness is not without effect on your internal narrative, or your moving lips. You can see yourself seeing, and your internal narrative reflects this, and so do your lips if you choose to say it out loud.

I have not seen the above argument written out that particular way - "the listener caught in the act of listening" - though it may well have been said before.

But it is a standard point - which zombie-ist philosophers accept! - that the Zombie World's philosophers, being atom-by-atom identical to our own philosophers, write identical papers about the philosophy of consciousness.

At this point, the Zombie World stops being an intuitive consequence of the idea of a passive listener.

Philosophers writing papers about consciousness would seem to be at least one effect of consciousness upon the world. You can argue clever reasons why this is not so, but you have to be clever.

You would intuitively suppose that if your inward awareness went away, this would change the world, in that your internal narrative would no longer say things like "There is a mysterious listener within me," because the mysterious listener would be gone. It is usually right after you focus your awareness on your awareness, that your internal narrative says "I am aware of my awareness", which suggests that if the first event never happened again, neither would the second. You can argue clever reasons why this is not so, but you have to be clever.

You can form a propositional belief that "Consciousness is without effect", and not see any contradiction at first, if you don't realize that talking about consciousness is an effect of being conscious. But once you see the connection from the general rule that consciousness has no effect, to the specific implication that consciousness has no effect on how philosophers write papers about consciousness, zombie-ism stops being intuitive and starts requiring you to postulate strange things.

One strange thing you might postulate is that there's a Zombie Master, a god within the Zombie World who surreptitiously takes control of zombie philosophers and makes them talk and write about consciousness.

A Zombie Master doesn't seem impossible. Human beings often don't sound all that coherent when talking about consciousness. It might not be that hard to fake their discourse, to the standards of, say, a human amateur talking in a bar. Maybe you could take, as a corpus, one thousand human amateurs trying to discuss consciousness; feed them into a non-conscious but sophisticated AI, better than today's models but not self-modifying; and get back discourse about "consciousness" that sounded as sensible as most humans, which is to say, not very.

But this speech about "consciousness" would not be spontaneous. It would not be produced within the AI. It would be a recorded imitation of someone else talking. That is just a holodeck, with a central AI writing the speech of the non-player characters. This is not what the Zombie World is about.

By supposition, the Zombie World is atom-by-atom identical to our own, except that the inhabitants lack consciousness. Furthermore, the atoms in the Zombie World move under the same laws of physics as in our own world. If there are "bridging laws" that govern which configurations of atoms evoke consciousness, those bridging laws are absent. But, by hypothesis, the difference is not experimentally detectable. When it comes to saying whether a quark zigs or zags or exerts a force on nearby quarks - anything experimentally measurable - the same physical laws govern.

The Zombie World has no room for a Zombie Master, because a Zombie Master has to control the zombie's lips, and that control is, in principle, experimentally detectable. The Zombie Master moves lips, therefore it has observable consequences. There would be a point where an electron zags, instead of zigging, because the Zombie Master says so. (Unless the Zombie Master is actually in the world, as a pattern of quarks - but then the Zombie World is not atom-by-atom identical to our own, unless you think this world also contains a Zombie Master.)

When a philosopher in our world types, "I think the Zombie World is possible", his fingers strike keys in sequence: Z-O-M-B-I-E. There is a chain of causality that can be traced back from these keystrokes: muscles contracting, nerves firing, commands sent down through the spinal cord, from the motor cortex - and then into less understood areas of the brain, where the philosopher's internal narrative first began talking about "consciousness".

And the philosopher's zombie twin strikes the same keys, for the same reason, causally speaking. There is no cause within the chain of explanation for why the philosopher writes the way he does, which is not also present in the zombie twin. The zombie twin also has an internal narrative about "consciousness", that a super-fMRI could read out of the auditory cortex. And whatever other thoughts, or other causes of any kind, led to that internal narrative, they are exactly the same in our own universe and in the Zombie World.

So you can't say that the philosopher is writing about consciousness because of consciousness, while the zombie twin is writing about consciousness because of a Zombie Master or AI chatbot. When you trace back the chain of causality behind the keyboard, to the internal narrative echoed in the auditory cortex, to the cause of the narrative, you must find the same physical explanation in our world as in the zombie world.

As the most formidable advocate of zombie-ism, David Chalmers, writes:

Think of my zombie twin in the universe next door. He talks about conscious experience all the time—in fact, he seems obsessed by it. He spends ridiculous amounts of time hunched over a computer, writing chapter after chapter on the mysteries of consciousness. He often comments on the pleasure he gets from certain sensory qualia, professing a particular love for deep greens and purples. He frequently gets into arguments with zombie materialists, arguing that their position cannot do justice to the realities of conscious experience.

And yet he has no conscious experience at all! In his universe, the materialists are right and he is wrong. Most of his claims about conscious experience are utterly false. But there is certainly a physical or functional explanation of why he makes the claims he makes. After all, his universe is fully law-governed, and no events therein are miraculous, so there must be some explanation of his claims.

...Any explanation of my twin's behavior will equally count as an explanation of my behavior, as the processes inside his body are precisely mirrored by those inside mine. The explanation of his claims obviously does not depend on the existence of consciousness, as there is no consciousness in his world. It follows that the explanation of my claims is also independent of the existence of consciousness.

Chalmers is not arguing against zombies; those are his actual beliefs!

This paradoxical situation is at once delightful and disturbing. It is not obviously fatal to the nonreductive position, but it is at least something that we need to come to grips
with...

I would seriously nominate this as the largest bullet ever bitten in the history of time. And that is a backhanded compliment to David Chalmers: A lesser mortal would simply fail to see the implications, or refuse to face them, or rationalize a reason it wasn't so.

Why would anyone bite a bullet that large? Why would anyone postulate unconscious zombies who write papers about consciousness for exactly the same reason that our own genuinely conscious philosophers do?

Not because of the first intuition I wrote about, the intuition of the passive listener. That intuition may say that zombies can drive cars or do math or even fall in love, but it doesn't say that zombies write philosophy papers about their passive listeners.

The zombie argument does not rest solely on the intuition of the passive listener. If this was all there was to the zombie argument, it would be dead by now, I think. The intuition that the "listener" can be eliminated without effect, would go away as soon as you realized that your internal narrative routinely seems to catch the listener in the act of listening.

No, the drive to bite this bullet comes from an entirely different intuition - the intuition that no matter how many atoms you add up, no matter how many masses and electrical charges interact with each other, they will never necessarily produce a subjective sensation of the mysterious redness of red. It may be a fact about our physical universe (Chalmers says) that putting such-and-such atoms into such-and-such a position, evokes a sensation of redness; but if so, it is not a necessary fact, it is something to be explained above and beyond the motion of the atoms.

But if you consider the second intuition on its own, without the intuition of the passive listener, it is hard to see why it implies zombie-ism. Maybe there's just a different kind of stuff, apart from and additional to atoms, that is not causally passive - a soul that actually does stuff, a soul that plays a real causal role in why we write about "the mysterious redness of red". Take out the soul, and... well, assuming you just don't fall over in a coma, you certainly won't write any more papers about consciousness!

This is the position taken by Descartes and most other ancient thinkers: The soul is of a different kind, but it interacts with the body. Descartes's position is technically known as substance dualism - there is a thought-stuff, a mind-stuff, and it is not like atoms; but it is causally potent, interactive, and leaves a visible mark on our universe.

Zombie-ists are property dualists - they don't believe in a separate soul; they believe that matter in our universe has additional properties beyond the physical.

"Beyond the physical"? What does that mean? It means the extra properties are there, but they don't influence the motion of the atoms, like the properties of electrical charge or mass. The extra properties are not experimentally detectable by third parties; you know you are conscious, from the inside of your extra properties, but no scientist can ever directly detect this from outside.

So the additional properties are there, but not causally active. The extra properties do not move atoms around, which is why they can't be detected by third parties.

And that's why we can (allegedly) imagine a universe just like this one, with all the atoms in the same places, but the extra properties missing, so that everything goes on the same as before, but no one is conscious.

The Zombie World may not be physically possible, say the zombie-ists - because it is a fact that all the matter in our universe has the extra properties, or obeys the bridging laws that evoke consciousness - but the Zombie World is logically possible: the bridging laws could have been different.

But, once you realize that conceivability is not the same as logical possibility, and that the Zombie World isn't even all that intuitive, why say that the Zombie World is logically possible?

Why, oh why, say that the extra properties are epiphenomenal and indetectable?

We can put this dilemma very sharply: Chalmers believes that there is something called consciousness, and this consciousness embodies the true and indescribable substance of the mysterious redness of red. It may be a property beyond mass and charge, but it's there, and it is consciousness. Now, having said the above, Chalmers furthermore specifies that this true stuff of consciousness is epiphenomenal, without causal potency - but why say that?

Why say that you could subtract this true stuff of consciousness, and leave all the atoms in the same place doing the same things? If that's true, we need some separate physical explanation for why Chalmers talks about "the mysterious redness of red". That is, there exists both a mysterious redness of red, which is extra-physical, and an entirely separate reason, within physics, why Chalmers talks about the "mysterious redness of red".

Chalmers does confess that these two things seem like they ought to be related, but really, why do you need both? Why not just pick one or the other?

Once you've postulated that there is a mysterious redness of red, why not just say that it interacts with your internal narrative and makes you talk about the "mysterious redness of red"?

Isn't Descartes taking the simpler approach, here? The strictly simpler approach?

Why postulate an extramaterial soul, and then postulate that the soul has no effect on the physical world, and then postulate a mysterious unknown material process that causes your internal narrative to talk about conscious experience?

Why not postulate the true stuff of consciousness which no amount of mere mechanical atoms can add up to, and then, having gone that far already, let this true stuff of consciousness have causal effects like making philosophers talk about consciousness?

I am not endorsing Descartes's view. But at least I can understand where Descartes is coming from. Consciousness seems mysterious, so you postulate a mysterious stuff of consciousness. Fine.

But now the zombie-ists postulate that this mysterious stuff doesn't do anything, so you need a whole new explanation for why you say you're conscious.

That isn't vitalism. That's something so bizarre that vitalists would spit out their coffee. "When fires burn, they release phlogiston. But phlogiston doesn't have any experimentally detectable impact on our universe, so you'll have to go looking for a separate explanation of why a fire can melt snow." What?

Are property dualists under the impression that if they postulate a new active force, something that has a causal impact on observables, they will be sticking their necks out too far?

Me, I'd say that if you postulate a mysterious, separate, additional, inherently mental property of consciousness, above and beyond positions and velocities, then, at that point, you have already stuck your neck out as far as it can go. To postulate this stuff of consciousness, and then further postulate that it doesn't do anything - for the love of cute kittens, why?

There isn't even an obvious career motive. "Hi, I'm a philosopher of consciousness. My subject matter is the most important thing in the universe and I should get lots of funding? Well, it's nice of you to say so, but actually the phenomenon I study doesn't do anything whatsoever." (Argument from career impact is not valid, but I say it to leave a line of retreat.)

Chalmers critiques substance dualism on the grounds that it's hard to see what new theory of physics, what new substance that interacts with matter, could possibly explain consciousness. But property dualism has exactly the same problem. No matter what kind of dual property you talk about, how exactly does it explain consciousness?

When Chalmers postulated an extra property that is consciousness, he took that leap across the unexplainable. How does it help his theory to further specify that this extra property has no effect? Why not just let it be causal?

If I were going to be unkind, this would be the time to drag in the dragon - to mention Carl Sagan's parable of the dragon in the garage. "I have a dragon in my garage." Great! I want to see it, let's go! "You can't see it - it's an invisible dragon." Oh, I'd like to hear it then. "Sorry, it's an inaudible dragon." I'd like to measure its carbon dioxide output. "It doesn't breathe." I'll toss a bag of flour into the air, to outline its form. "The dragon is permeable to flour."

One motive for trying to make your theory unfalsifiable, is that deep down you fear to put it to the test. Sir Roger Penrose (physicist) and Stuart Hameroff (neurologist) are substance dualists; they think that there is something mysterious going on in quantum, that Everett is wrong and that the "collapse of the wave-function" is physically real, and that this is where consciousness lives and how it exerts causal effect upon your lips when you say aloud "I think therefore I am." Believing this, they predicted that neurons would protect themselves from decoherence long enough to maintain macroscopic quantum states.

This is in the process of being tested, and so far, prospects are not looking good for Penrose -

- but Penrose's basic conduct is scientifically respectable. Not Bayesian, maybe, but still fundamentally healthy. He came up with a wacky hypothesis. He said how to test it. He went out and tried to actually test it.

As I once said to Stuart Hameroff, "I think the hypothesis you're testing is completely hopeless, and your experiments should definitely be funded. Even if you don't find exactly what you're looking for, you're looking in a place where no one else is looking, and you might find something interesting."

So a nasty dismissal of epiphenomenalism would be that zombie-ists are afraid to say the consciousness-stuff can have effects, because then scientists could go looking for the extra properties, and fail to find them.

I don't think this is actually true of Chalmers, though. If Chalmers lacked self-honesty, he could make things a lot easier on himself.

(But just in case Chalmers is reading this and does have falsification-fear, I'll point out that if epiphenomenalism is false, then there is some other explanation for that-which-we-call consciousness, and it will eventually be found, leaving Chalmers's theory in ruins; so if Chalmers cares about his place in history, he has no motive to endorse epiphenomenalism unless he really thinks it's true.)

Chalmers is one of the most frustrating philosophers I know. Sometimes I wonder if he's pulling an "Atheism Conquered". Chalmers does this really sharp analysis... and then turns left at the last minute. He lays out everything that's wrong with the Zombie World scenario, and then, having reduced the whole argument to smithereens, calmly accepts it.

Chalmers does the same thing when he lays out, in calm detail, the problem with saying that our own beliefs in consciousness are justified, when our zombie twins say exactly the same thing for exactly the same reasons and are wrong.

On Chalmers's theory, Chalmers saying that he believes in consciousness cannot be causally justified; the belief is not caused by the fact itself. In the absence of consciousness, Chalmers would write the same papers for the same reasons.

On epiphenomenalism, Chalmers saying that he believes in consciousness cannot be justified as the product of a process that systematically outputs true beliefs, because the zombie twin writes the same papers using the same systematic process and is wrong.

Chalmers admits this. Chalmers, in fact, explains the argument in great detail in his book. Okay, so Chalmers has solidly proven that he is not justified in believing in epiphenomenal consciousness, right? No. Chalmers writes:

Conscious experience lies at the center of our epistemic universe; we have access to it directly. This raises the question: what is it that justifies our beliefs about our experiences, if it is not a causal link to those experiences, and if it is not the mechanisms by which the beliefs are formed? I think the answer to this is clear: it is having the experiences that justifies the beliefs. For example, the very fact that I have a red experience now provides justification for my belief that I am having a red experience...

Because my zombie twin lacks experiences, he is in a very different epistemic situation from me, and his judgments lack the corresponding justification. It may be tempting to object that if my belief lies in the physical realm, its justification must lie in the physical realm; but this is a non sequitur. From the fact that there is no justification in the physical realm, one might conclude that the physical portion of me (my brain, say) is not justified in its belief. But the question is whether I am justified in the belief, not whether my brain is justified in the belief, and if property dualism is correct than there is more to me than my brain.

So - if I've got this thesis right - there's a core you, above and beyond your brain, that believes it is not a zombie, and directly experiences not being a zombie; and so its beliefs are justified.

But Chalmers just wrote all that stuff down, in his very physical book, and so did the zombie-Chalmers.

The zombie Chalmers can't have written the book because of the zombie's core self above the brain; there must be some entirely different reason, within the laws of physics.

It follows that even if there is a part of Chalmers hidden away that is conscious and believes in consciousness, directly and without mediation, there is also a separable subspace of Chalmers - a causally closed cognitive subsystem that acts entirely within physics - and this "outer self" is what speaks Chalmers's internal narrative, and writes papers on consciousness.

I do not see any way to evade the charge that, on Chalmers's own theory, this separable outer Chalmers is deranged. This is the part of Chalmers that is the same in this world, or the Zombie World; and in either world it writes philosophy papers on consciousness for no valid reason. Chalmers's philosophy papers are not output by that inner core of awareness and belief-in-awareness, they are output by the mere physics of the internal narrative that makes Chalmers's fingers strike the keys of his computer.

And yet this deranged outer Chalmers is writing philosophy papers that just happen to be perfectly right, by a separate and additional miracle. Not a logically necessary miracle (then the Zombie World would not be logically possible). A physically contingent miracle, that happens to be true in what we think is our universe, even though science can never distinguish our universe from the Zombie World.

Or at least, that would seem to be the implication of what the self-confessedly deranged outer Chalmers is telling us.

I think I speak for all reductionists when I say Huh?

That's not epicycles. That's, "Planetary motions follow these epicycles - but epicycles don't actually do anything - there's something else that makes the planets move the same way the epicycles say they should, which I haven't been able to explain - and by the way, I would say this even if there weren't any epicycles."

I have a nonstandard perspective on philosophy because I look at everything with an eye to designing an AI; specifically, a self-improving Artificial General Intelligence with stable motivational structure.

When I think about designing an AI, I ponder principles like probability theory, the Bayesian notion of evidence as differential diagnostic, and above all, reflective coherence. Any self-modifying AI that starts out in a reflectively inconsistent state won't stay that way for long.

If a self-modifying AI looks at a part of itself that concludes "B" on condition A - a part of itself that writes "B" to memory whenever condition A is true - and the AI inspects this part, determines how it (causally) operates in the context of the larger universe, and the AI decides that this part systematically tends to write false data to memory, then the AI has found what appears to be a bug, and the AI will self-modify not to write "B" to the belief pool under condition A.

Any epistemological theory that disregards reflective coherence is not a good theory to use in constructing self-improving AI. This is a knockdown argument from my perspective, considering what I intend to actually use philosophy for. So I have to invent a reflectively coherent theory anyway. And when I do, by golly, reflective coherence turns out to make intuitive sense.

So that's the unusual way in which I tend to think about these things. And now I look back at Chalmers:

The causally closed "outer Chalmers" (that is not influenced in any way by the "inner Chalmers" that has separate additional awareness and beliefs) must be carrying out some systematically unreliable, unwarranted operation which in some unexplained fashion causes the internal narrative to produce beliefs about an "inner Chalmers" that are correct for no logical reason in what happens to be our universe.

But there's no possible warrant for the outer Chalmers or any reflectively coherent self-inspecting AI to believe in this mysterious correctness. A good AI design should, I think, look like a reflectively coherent intelligence embodied in a causal system, with a testable theory of how that selfsame causal system produces systematically accurate beliefs on the way to achieving its goals.

So the AI will scan Chalmers and see a closed causal cognitive system producing an internal narrative that is uttering nonsense. Nonsense that seems to have a high impact on what Chalmers thinks should be considered a morally valuable person.

This is not a necessary problem for Friendly AI theorists. It is only a problem if you happen to be an epiphenomenalist. If you believe either the reductionists (consciousness happens within the atoms) or the substance dualists (consciousness is causally potent immaterial stuff), people talking about consciousness are talking about something real, and a reflectively consistent Bayesian AI can see this by tracing back the chain of causality for what makes people say "consciousness".

According to Chalmers, the causally closed cognitive system of Chalmers's internal narrative is (mysteriously) malfunctioning in a way that, not by necessity, but just in our universe, miraculously happens to be correct. Furthermore, the internal narrative asserts "the internal narrative is mysteriously malfunctioning, but miraculously happens to be correctly echoing the justified thoughts of the epiphenomenal inner core", and again, in our universe, miraculously happens to be correct.

Oh, come on!

Shouldn't there come a point where you just give up on an idea? Where, on some raw intuitive level, you just go: What on Earth was I thinking?

Humanity has accumulated some broad experience with what correct theories of the world look like. This is not what a correct theory looks like.

"Argument from incredulity," you say. Fine, you want it spelled out? The said Chalmersian theory postulates multiple unexplained complex miracles. This drives down its prior probability, by the conjunction rule of probability and Occam's Razor. It is therefore dominated by at least two theories which postulate fewer miracles, namely:

- Substance dualism:

- There is a stuff of consciousness which is not yet understood, an extraordinary super-physical stuff that visibly affects our world; and this stuff is what makes us talk about consciousness.

- Not-quite-faith-based reductionism:

- That-which-we-name "consciousness" happens within physics, in a way not yet understood, just like what happened the last three thousand times humanity ran into something mysterious.

- Your intuition that no material substance can possibly add up to consciousness is incorrect. If you actually knew exactly why you talk about consciousness, this would give you new insights, of a form you can't now anticipate; and afterward you would realize that your arguments about normal physics having no room for consciousness were flawed.

Compare to:

- Epiphenomenal property dualism:

- Matter has additional consciousness-properties which are not yet understood. These properties are epiphenomenal with respect to ordinarily observable physics - they make no difference to the motion of particles.

- Separately, there exists a not-yet-understood reason within normal physics why philosophers talk about consciousness and invent theories of dual properties.

- Miraculously, when philosophers talk about consciousness, the bridging laws of our world are exactly right to make this talk about consciousness correct, even though it arises from a malfunction (drawing of logically unwarranted conclusions) in the causally closed cognitive system that types philosophy papers.

I know I'm speaking from limited experience, here. But based on my limited experience, the Zombie Argument may be a candidate for the most deranged idea in all of philosophy.

There are times when, as a rationalist, you have to believe things that seem weird to you. Relativity seems weird, quantum mechanics seems weird, natural selection seems weird.

But these weirdnesses are pinned down by massive evidence. There's a difference between believing something weird because science has confirmed it overwhelmingly -

- versus believing a proposition that seems downright deranged, because of a great big complicated philosophical argument centered around unspecified miracles and giant blank spots not even claimed to be understood -

- in a case where even if you accept everything that has been told to you so far, afterward the phenomenon will still seem like a mystery and still have the same quality of wondrous impenetrability that it had at the start.

The correct thing for a rationalist to say at this point, if all of David Chalmers's arguments seem individually plausible - which they don't seem to me - is:

"Okay... I don't know how consciousness works... I admit that... and maybe I'm approaching the whole problem wrong, or asking the wrong questions... but this zombie business can't possibly be right. The arguments aren't nailed down enough to make me believe this - especially when accepting it won't make me feel any less confused. On a core gut level, this just doesn't look like the way reality could really really work."

Mind you, I am not saying this is a substitute for careful analytic refutation of Chalmers's thesis. System 1 is not a substitute for System 2, though it can help point the way. You still have to track down where the problems are specifically.

Chalmers wrote a big book, not all of which is available through free Google preview. I haven't duplicated the long chains of argument where Chalmers lays out the arguments against himself in calm detail. I've just tried to tack on a final refutation of Chalmers's last presented defense, which Chalmers has not yet countered to my knowledge. Hit the ball back into his court, as it were.

But, yes, on a core level, the sane thing to do when you see the conclusion of the zombie argument, is to say "That can't possibly be right" and start looking for a flaw.

Sequence: Quantum Physics

Reductionism

Belief in the Implied Invisible

Sequence: Zombies

Zombie Responses

Referenced by: Zombie Responses • The Generalized Anti-Zombie Principle • Identity Isn't In Specific Atoms • The Conscious Sorites Paradox • Mach's Principle: Anti-Epiphenomenal Physics • Timeless Identity • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity • Against Modal Logics • Nonperson Predicates • Belief in Self-Deception • I'm Not Saying People Are Stupid • The Correct Contrarian Cluster

Original with comments: Zombies! Zombies?

Zombie Responses

Eliezer Yudkowsky, 05 April 2008 12:42AM

Continuation of: Zombies! Zombies?

I'm a bit tired today, having stayed up until 3AM writing yesterday's >6000-word post on zombies, so today I'll just reply to Richard, and tie up a loose end I spotted the next day.

Besides, TypePad's nitwit, un-opt-out-able 50-comment pagination "feature", that doesn't work with the Recent Comments sidebar, means that we might as well jump the discussion here before we go over the 50-comment limit.

(A) Richard Chappell writes:

A terminological note (to avoid unnecessary confusion): what you call 'conceivable', others of us would merely call "apparently conceivable".

The gap between "I don't see a contradiction yet" and "this is logically possible" is so huge (it's NP-complete even in some simple-seeming cases) that you really should have two different words. As the zombie argument is boosted to the extent that this huge gap can be swept under the rug of minor terminological differences, I really think it would be a good idea to say "conceivable" versus "logically possible" or maybe even have a still more visible distinction. I can't choose professional terminology that has already been established, but in a case like this, I might seriously refuse to use it.

Maybe I will say "apparently conceivable" for the kind of information that zombie advocates get by imagining Zombie Worlds, and "logically possible" for the kind of information that is established by exhibiting a complete model or logical proof. Note the size of the gap between the information you can get by closing your eyes and imagining zombies, and the information you need to carry the argument for epiphenomenalism.

That is, your view would be characterized as a form of Type-A materialism, the view that zombies are not even (genuinely) conceivable, let alone metaphysically possible.

Type-A materialism is a large bundle; you shouldn't attribute the bundle to me until you see me agree with each of the parts. I think that someone who asks "What is consciousness?" is asking a legitimate question, has a legitimate demand for insight; I don't necessarily think that the answer takes the form of "Here is this stuff that has all the properties you would attribute to consciousness, for such-and-such reason", but may to some extent consist of insights that cause you to realize you were asking the question the wrong way.

This is not being eliminative about consciousness. It is being realistic about what kind of insights to expect, faced with a problem that (1) seems like it must have some solution, (2) seems like it cannot possibly have any solution, and (3) is being discussed in a fashion that has a great big dependence on the not-fully-understood ad-hoc architecture of human cognition.

(1) You haven't, so far as I can tell, identified any logical contradiction in the description of the zombie world. You've just pointed out that it's kind of strange. But there are many bizarre possible worlds out there. That's no reason to posit an implicit contradiction. So it's still completely mysterious to me what this alleged contradiction is supposed to be.

Okay, I'll spell it out from a materialist standpoint:

1. The zombie world, by definition, contains all parts of our world that are closed with respect to causality. In particular, it contains the cause of my saying, "I think therefore I am."

2. When I focus my inward awareness on my inward awareness, I shortly thereafter experience my internal narrative saying "I am focusing my inward awareness on my inward awareness", and can, if I choose, say so out loud.

3. Intuitively, it sure seems like my inward awareness is causing my internal narrative to say certain things.

4. The word "consciousness", if it has any meaning at all, refers to that-which-is or that-which-causes or that-which-makes-me-say-I-have inward awareness.

5. From (3) and (4) it would follow that if the zombie world is closed with respect to the causes of my saying "I think therefore I am", the zombie world contains that which we refer to as "consciousness".

6. By definition, the zombie world does not contain consciousness.

7. (3) seems to me to have a rather high probability of being empirically true. Therefore I evaluate a high empirical probability that the zombie world is logically impossible.

You can save the Zombie World by letting the cause of my internal narrative's saying "I think therefore I am" be something entirely other than consciousness. In conjunction with the assumption that consciousness does exist, this is the part that struck me as deranged.

But if the above is conceivable, then isn't the Zombie World conceivable?

No, because the two constructions of the Zombie World involve giving the word "consciousness" different empirical referents, like "water" in our world meaning H20 versus "water" in Putnam's Twin Earth meaning XYZ. For the Zombie World to be logically possible, it does not suffice that, for all you knew about how the empirical world worked, the word "consciousness" could have referred to an epiphenomenon that is entirely different from the consciousness we know. The Zombie World lacks consciousness, not "consciousness" - it is a world without H20, not a world without "water". This is what is required to carry the empirical statement, "You could eliminate the referent of whatever is meant by "consciousness" from our world, while keeping all the atoms in the same place."

Which is to say: I hold that it is an empirical fact, given what the word "consciousness" actually refers to, that it is logically impossible to eliminate consciousness without moving any atoms. What it would mean to eliminate "consciousness" from a world, rather than consciousness, I will not speculate.

(2) It's misleading to say it's "miraculous" (on the property dualist view) that our qualia line up so neatly with the physical world. There's a natural law which guarantees this, after all. So it's no more miraculous than any other logically contingent nomic necessity (e.g. the constants in our physical laws).

It is the natural law itself that is "miraculous" - counts as an additional complex-improbable element of the theory to be postulated, without having been itself justified in terms of things already known. One postulates (a) an inner world that is conscious (b) a malfunctioning outer world that talks about consciousness for no reason (c) that the two align perfectly. C does not follow from A and B, and so is a separate postulate.

I agree that this usage of "miraculous" conflicts with the philosophical sense of violating a natural law; I meant it in the sense of improbability appearing from no apparent source, a la perpetual motion belief. Hence the word was ill-chosen in context.

That is, Zombie (or 'Outer') Chalmers doesn't actually conclude anything, because his utterances are meaningless. A fortiori, he doesn't conclude anything unwarrantedly. He's just making noises; these are no more susceptible to epistemic assessment than the chirps of a bird.

Looking at this from an AI-design standpoint, it seems to me like you should be able to build an AI that systematically refines an inner part of itself that correlates (in the sense of mutual information or systematic relations) to the environment, perhaps including floating-point numbers of a sort that I would call "probabilities" because they obey the internal relations mandated by Cox's Theorems when the AI encounters new information - pardon me, new sense inputs.

You will say that, unless the AI is more than mere transistors - unless it has the dual aspect - the AI has no beliefs.

I think my views on this were expressed pretty clearly in "The Simple Truth".

To me, it seems pretty straightforward to construct notions of maps that correlate to territories in systematic ways, without mentioning anything other than things of pure physical causality. The AI outputs a map of Texas. Another AI flies with the map to Texas and checks to see if the highways are in the corresponding places, chirping "True" when it detects a match and "False" when it detects a mismatch. You can refuse to call this "a map of Texas" but the AIs themselves are still chirping "True" or "False", and the said AIs are going to chirp "False" when they look at Chalmers's belief in an epiphenomenal inner core, and I for one would agree with them.

It's clear that the function of mapping reality is performed strictly by Outer Chalmers. The whole business of producing belief representations is handled by Bayesian structure in causal interactions. There's nothing left for the Inner Chalmers to do, but bless the whole affair with epiphenomenal meaning. So when it comes to talking about "accuracy", let alone "systematic accuracy", it seems like we should be able to determine it strictly by looking at the Outer Chalmers.

(B) In yesterday's text, I left out an assumption when I wrote:

If a self-modifying AI looks at a part of itself that concludes "B" on condition A - a part of itself that writes "B" to memory whenever condition A is true - and the AI inspects this part, determines how it (causally) operates in the context of the larger universe, and the AI decides that this part systematically tends to write false data to memory, then the AI has found what appears to be a bug, and the AI will self-modify not to write "B" to the belief pool under condition A...

But there's no possible warrant for the outer Chalmers or any reflectively coherent self-inspecting AI to believe in this mysterious correctness. A good AI design should, I think, look like a reflectively coherent intelligence embodied in a causal system, with a testable theory of how that selfsame causal system produces systematically accurate beliefs on the way to achieving its goals.

Actually, you need an additional assumption to the above, which is that a "good AI design" (the kind I was thinking of, anyway) judges its own rationality in a modular way; it enforces global rationality by enforcing local rationality. If there is a piece that, relative to its context, is locally systematically unreliable - for some possible beliefs "B_i" and conditions A_i, it adds some "B_i" to the belief pool under local condition A_i, where reflection by the system indicates that B_i is not true (or in the case of probabilistic beliefs, not accurate) when the local condition A_i is true, then this is a bug.

This kind of modularity is a way to make the problem tractable, and it's how I currently think about the first-generation AI design, but it may not be the only way to make the problem tractable. Obviously a lot of handwaving here, but you get the general idea.

The notion is that a causally closed cognitive system - such as an AI designed by its programmers to use only causally efficacious parts, or an AI whose theory of its own functioning is entirely testable, or the outer Chalmers that writes philosophy papers - which believes that it has an epiphenomenal inner self, must be doing something systematically unreliable because it would conclude the same thing in a Zombie World. A mind all of whose parts are systematically locally reliable, relative to their contexts, would be systematically globally reliable. Ergo, a mind which is globally unreliable must contain at least one locally unreliable part. So a causally closed cognitive system inspecting itself for local reliability must discover that at least one step involved in adding the belief of an epiphenomenal inner self, is unreliable.

If there are other ways for minds to be reflectively coherent which avoid this proof of disbelief in zombies, philosophers are welcome to try and specify them.

The reason why I have to specify all this is that otherwise you get a kind of extremely cheap reflective coherence where the AI can never label itself unreliable. E.g. if the AI finds a part of itself that computes 2 + 2 = 5 (in the surrounding context of counting sheep) the AI will reason: "Well, this part malfunctions and says that 2 + 2 = 5... but by pure coincidence, 2 + 2 is equal to 5, or so it seems to me... so while the part looks systematically unreliable, I better keep it the way it is, or it will handle this special case wrong." That's why I talk about enforcing global reliability by enforcing local systematic reliability - if you just compare your global beliefs to your global beliefs, you don't go anywhere.

This does have a general lesson: Show your arguments are globally reliable by virtue of each step being locally reliable, don't just compare the arguments' conclusions to your intuitions.

(C) An anonymous poster wrote:

A sidepoint, this, but I believe your etymology for "n'shama" is wrong. It is related to the word for "breath", not "hear". The root for "hear" contains an ayin, which n'shama does not.

Now that's what I call a miraculously misleading coincidence - although the word N'Shama arose for completely different reasons, it sounded exactly the right way to make me think it referred to an inner listener.

Oops.

Sequence: Zombies

Zombies! Zombies?

The Generalized Anti-Zombie Principle

Original with comments: Zombie Responses

The Generalized Anti-Zombie Principle

Eliezer Yudkowsky, 05 April 2008 11:16PM

Followup to: Zombies! Zombies?

"Each problem that I solved became a rule which served afterwards to solve other problems."
-- Rene Descartes, Discours de la Methode

"Zombies" are putatively beings that are atom-by-atom identical to us, governed by all the same third-party-visible physical laws, except that they are not conscious.

Though the philosophy is complicated, the core argument against zombies is simple: When you focus your inward awareness on your inward awareness, soon after your internal narrative (the little voice inside your head that speaks your thoughts) says "I am aware of being aware", and then you say it out loud, and then you type it into a computer keyboard, and create a third-party visible blog post.

Consciousness, whatever it may be - a substance, a process, a name for a confusion - is not epiphenomenal; your mind can catch the inner listener in the act of listening, and say so out loud. The fact that I have typed this paragraph would at least seem to refute the idea that consciousness has no experimentally detectable consequences.

I hate to say "So now let's accept this and move on," over such a philosophically controversial question, but it seems like a considerable majority of Overcoming Bias commenters do accept this. And there are other conclusions you can only get to after you accept that you cannot subtract consciousness and leave the universe looking exactly the same. So now let's accept this and move on.

The form of the Anti-Zombie Argument seems like it should generalize, becoming an Anti-Zombie Principle. But what is the proper generalization?

Let's say, for example, that someone says: "I have a switch in my hand, which does not affect your brain in any way; and iff this switch is flipped, you will cease to be conscious." Does the Anti-Zombie Principle rule this out as well, with the same structure of argument?

It appears to me that in the case above, the answer is yes. In particular, you can say: "Even after your switch is flipped, I will still talk about consciousness for exactly the same reasons I did before. If I am conscious right now, I will still be conscious after you flip the switch."

Philosophers may object, "But now you're equating consciousness with talking about consciousness! What about the Zombie Master, the chatbot that regurgitates a remixed corpus of amateur human discourse on consciousness?"

But I did not equate "consciousness" with verbal behavior. The core premise is that, among other things, the true referent of "consciousness" is also the cause in humans of talking about inner listeners.

As I argued (at some length) in the sequence on words, what you want in defining a word is not always a perfect Aristotelian necessary-and-sufficient definition; sometimes you just want a treasure map that leads you to the extensional referent. So "that which does in fact make me talk about an unspeakable awareness" is not a necessary-and-sufficient definition. But if what does in fact cause me to discourse about an unspeakable awareness, is not "consciousness", then...

...then the discourse gets pretty futile. That is not a knockdown argument against zombies - an empirical question can't be settled by mere difficulties of discourse. But if you try to defy the Anti-Zombie Principle, you will have problems with the meaning of your discourse, not just its plausibility.

Could we define the word "consciousness" to mean "whatever actually makes humans talk about 'consciousness'"? This would have the powerful advantage of guaranteeing that there is at least one real fact named by the word "consciousness". Even if our belief in consciousness is a confusion, "consciousness" would name the cognitive architecture that generated the confusion. But to establish a definition is only to promise to use a word consistently; it doesn't settle any empirical questions, such as whether our inner awareness makes us talk about our inner awareness.

Let's return to the Off-Switch.

If we allow that the Anti-Zombie Argument applies against the Off-Switch, then the Generalized Anti-Zombie Principle does not say only, "Any change that is not in-principle experimentally detectable (IPED) cannot remove your consciousness." The switch's flipping is experimentally detectable, but it still seems highly unlikely to remove your consciousness.

Perhaps the Anti-Zombie Principle says, "Any change that does not affect you in any IPED way cannot remove your consciousness"?

But is it a reasonable stipulation to say that flipping the switch does not affect you in any IPED way? All the particles in the switch are interacting with the particles composing your body and brain. There are gravitational effects - tiny, but real and IPED. The gravitational pull from a one-gram switch ten meters away is around 6 * 10-16 m/s^2. That's around half a neutron diameter per second per second, far below thermal noise, but way above the Planck level.

We could flip the switch light-years away, in which case the flip would have no immediate causal effect on you (whatever "immediate" means in this case) (if the Standard Model of physics is correct).

But it doesn't seem like we should have to alter the thought experiment in this fashion. It seems that, if a disconnected switch is flipped on the other side of a room, you should not expect your inner listener to go out like a light, because the switch "obviously doesn't change" that which is the true cause of your talking about an inner listener. Whatever you really are, you don't expect the switch to mess with it.

This is a large step.

If you deny that it is a reasonable step, you had better never go near a switch again. But still, it's a large step.

The key idea of reductionism is that our maps of the universe are multi-level to save on computing power, but physics seems to be strictly single-level. All our discourse about the universe takes place using references far above the level of fundamental particles.

The switch's flip does change the fundamental particles of your body and brain. It nudges them by whole neutron diameters away from where they would have otherwise been.

In ordinary life, we gloss a change this small by saying that the switch "doesn't affect you". But it does affect you. It changes everything by whole neutron diameters! What could possibly be remaining the same? Only the description that you would give of the higher levels of organization - the cells, the proteins, the spikes traveling along a neural axon. As the map is far less detailed than the territory, it must map many different states to the same description.

Any reasonable sort of humanish description of the brain that talks about neurons and activity patterns (or even the conformations of individual microtubules making up axons and dendrites) won't change when you flip a switch on the other side of the room. Nuclei are larger than neutrons, atoms are larger than nuclei, and by the time you get up to talking about the molecular level, that tiny little gravitational force has vanished from the list of things you bother to track.

But if you add up enough tiny little gravitational pulls, they will eventually yank you across the room and tear you apart by tidal forces, so clearly a small effect is not "no effect at all".

Maybe the tidal force from that tiny little pull, by an amazing coincidence, pulls a single extra calcium ion just a tiny bit closer to an ion channel, causing it to be pulled in just a tiny bit sooner, making a single neuron fire infinitesimally sooner than it would otherwise have done, a difference which amplifies chaotically, finally making a whole neural spike occur that otherwise wouldn't have occurred, sending you off on a different train of thought, that triggers an epileptic fit, that kills you, causing you to cease to be conscious...

If you add up a lot of tiny quantitative effects, you get a big quantitative effect - big enough to mess with anything you care to name. And so claiming that the switch has literally zero effect on the things you care about, is taking it too far.

But with just one switch, the force exerted is vastly less than thermal uncertainties, never mind quantum uncertainties. If you don't expect your consciousness to flicker in and out of existence as the result of thermal jiggling, then you certainly shouldn't expect to go out like a light when someone sneezes a kilometer away.

The alert Bayesian will note that I have just made an argument about expectations, states of knowledge, justified beliefs about what can and can't switch off your consciousness.

This doesn't necessarily destroy the Anti-Zombie Argument. Probabilities are not certainties, but the laws of probability are theorems; if rationality says you can't believe something on your current information, then that is a law, not a suggestion.

Still, this version of the Anti-Zombie Argument is weaker. It doesn't have the nice, clean, absolutely clear-cut status of, "You can't possibly eliminate consciousness while leaving all the atoms in exactly the same place." (Or for "all the atoms" substitute "all causes with in-principle experimentally detectable effects", and "same wavefunction" for "same place", etc.)

But the new version of the Anti-Zombie Argument still carries. You can say, "I don't know what consciousness really is, and I suspect I may be fundamentally confused about the question. But if the word refers to anything at all, it refers to something that is, among other things, the cause of my talking about consciousness. Now, I don't know why I talk about consciousness. But it happens inside my skull, and I expect it has something to do with neurons firing. Or maybe, if I really understood consciousness, I would have to talk about an even more fundamental level than that, like microtubules, or neurotransmitters diffusing across a synaptic channel. But still, that switch you just flipped has an effect on my neurotransmitters and microtubules that's much, much less than thermal noise at 310 Kelvin. So whatever the true cause of my talking about consciousness may be, I don't expect it to be hugely affected by the gravitational pull from that switch. Maybe it's just a tiny little infinitesimal bit affected? But it's certainly not going to go out like a light. I expect to go on talking about consciousness in almost exactly the same way afterward, for almost exactly the same reasons."

This application of the Anti-Zombie Principle is weaker. But it's also much more general. And, in terms of sheer common sense, correct.

The reductionist and the substance dualist actually have two different versions of the above statement. The reductionist furthermore says, "Whatever makes me talk about consciousness, it seems likely that the important parts take place on a much higher functional level than atomic nuclei. Someone who understood consciousness could abstract away from individual neurons firing, and talk about high-level cognitive architectures, and still describe how my mind produces thoughts like 'I think therefore I am'. So nudging things around by the diameter of a nucleon, shouldn't affect my consciousness (except maybe with very small probability, or by a very tiny amount, or not until after a significant delay)."

The substance dualist furthermore says, "Whatever makes me talk about consciousness, it's got to be something beyond the computational physics we know, which means that it might very well involve quantum effects. But still, my consciousness doesn't flicker on and off whenever someone sneezes a kilometer away. If it did, I would notice. It would be like skipping a few seconds, or coming out of a general anesthetic, or sometimes saying, "I don't think therefore I'm not." So since it's a physical fact that thermal vibrations don't disturb the stuff of my awareness, I don't expect flipping the switch to disturb it either."

Either way, you shouldn't expect your sense of awareness to vanish when someone says the word "Abracadabra", even if that does have some infinitesimal physical effect on your brain -

But hold on! If you hear someone say the word "Abracadabra", that has a very noticeable effect on your brain - so large, even your brain can notice it. It may alter your internal narrative; you may think, "Why did that person just say 'Abracadabra'?"

Well, but still you expect to go on talking about consciousness in almost exactly the same way afterward, for almost exactly the same reasons.

And again, it's not that "consciousness" is being equated to "that which makes you talk about consciousness". It's just that consciousness, among other things, makes you talk about consciousness. So anything that makes your consciousness go out like a light, should make you stop talking about consciousness.

If we do something to you, where you don't see how it could possibly change your internal narrative - the little voice in your head that sometimes says things like "I think therefore I am", whose words you can choose to say aloud - then it shouldn't make you cease to be conscious.

And this is true even if the internal narrative is just "pretty much the same", and the causes of it are also pretty much the same; among the causes that are pretty much the same, is whatever you mean by "consciousness".

If you're wondering where all this is going, and why it's important to go to such tremendous lengths to ponder such an obvious-seeming Generalized Anti-Zombie Principle, then consider the following debate:

Albert: "Suppose I replaced all the neurons in your head with tiny robotic artificial neurons that had the same connections, the same local input-output behavior, and analogous internal state and learning rules."

Bernice: "That's killing me! There wouldn't be a conscious being there anymore."

Charles: "Well, there'd still be a conscious being there, but it wouldn't be me."

Sir Roger Penrose: "The thought experiment you propose is impossible. You can't duplicate the behavior of neurons without tapping into quantum gravity. That said, there's not much point in me taking further part in this conversation." (Wanders away.)

Albert: "Suppose that the replacement is carried out one neuron at a time, and the swap occurs so fast that it doesn't make any difference to global processing."

Bernice: "How could that possibly be the case?"

Albert: "The little robot swims up to the neuron, surrounds it, scans it, learns to duplicate it, and then suddenly takes over the behavior, between one spike and the next. In fact, the imitation is so good, that your outward behavior is just the same as it would be if the brain were left undisturbed. Maybe not exactly the same, but the causal impact is much less than thermal noise at 310 Kelvin."

Charles: "So what?"

Albert: "So don't your beliefs violate the Generalized Anti-Zombie Principle? Whatever just happened, it didn't change your internal narrative! You'll go around talking about consciousness for exactly the same reason as before."

Bernice: "Those little robots are a Zombie Master. They'll make me talk about consciousness even though I'm not conscious. The Zombie World is possible if you allow there to be an added, extra, experimentally detectable Zombie Master - which those robots are."

Charles: "Oh, that's not right, Bernice. The little robots aren't plotting how to fake consciousness, or processing a corpus of text from human amateurs. They're doing the same thing neurons do, just in silicon instead of carbon."

Albert: "Wait, didn't you just agree with me?"

Charles: "I never said the new person wouldn't be conscious. I said it wouldn't be me."

Albert: "Well, obviously the Anti-Zombie Principle generalizes to say that this operation hasn't disturbed the true cause of your talking about this me thing."

Charles: "Uh-uh! Your operation certainly did disturb the true cause of my talking about consciousness. It substituted a different cause in its place, the robots. Now, just because that new cause also happens to be conscious - talks about consciousness for the same generalized reason - doesn't mean it's the same cause that was originally there."

Albert: "But I wouldn't even have to tell you about the robot operation. You wouldn't notice. If you think, going on introspective evidence, that you are in an important sense "the same person" that you were five minutes ago, and I do something to you that doesn't change the introspective evidence available to you, then your conclusion that you are the same person that you were five minutes ago should be equally justified. Doesn't the Generalized Anti-Zombie Principle say that if I do something to you that alters your consciousness, let alone makes you a completely different person, then you ought to notice somehow?"

Bernice: "Not if you replace me with a Zombie Master. Then there's no one there to notice."

Charles: "Introspection isn't perfect. Lots of stuff goes on inside my brain that I don't notice."

Albert: "You're postulating epiphenomenal facts about consciousness and identity!"

Bernice: "No I'm not! I can experimentally detect the difference between neurons and robots."

Charles: "No I'm not! I can experimentally detect the moment when the old me is replaced by a new person."

Albert: "Yeah, and I can detect the switch flipping! You're detecting something that doesn't make a noticeable difference to the true cause of your talk about consciousness and personal identity. And the proof is, you'll talk just the same way afterward."

Bernice: "That's because of your robotic Zombie Master!"

Charles: "Just because two people talk about 'personal identity' for similar reasons doesn't make them the same person."

I think the Generalized Anti-Zombie Principle supports Albert's position, but the reasons shall have to wait for future posts. I need other prerequisites, and besides, this post is already too long.

But you see the importance of the question, "How far can you generalize the Anti-Zombie Argument and have it still be valid?"

The makeup of future galactic civilizations may be determined by the answer...

Sequence: Zombies

Zombie Responses

GAZP vs. GLUT

Referenced by: GAZP vs. GLUT • Belief in the Implied Invisible • Quantum Explanations • Identity Isn't In Specific Atoms • Where Experience Confuses Physicists • Einstein's Speed • Mach's Principle: Anti-Epiphenomenal Physics • Timeless Identity • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity • Causality and Moral Responsibility • Could Anything Be Right? • Dreams of Friendliness

Original with comments: The Generalized Anti-Zombie Principle

GAZP vs. GLUT

Eliezer Yudkowsky, 07 April 2008 01:51AM

Followup to: The Generalized Anti-Zombie Principle

In "The Unimagined Preposterousness of Zombies", Daniel Dennett says:

To date, several philosophers have told me that they plan to accept my challenge to offer a non-question-begging defense of zombies, but the only one I have seen so far involves postulating a "logically possible" but fantastic being — a descendent of Ned Block's Giant Lookup Table fantasy...

A Giant Lookup Table, in programmer's parlance, is when you implement a function as a giant table of inputs and outputs, usually to save on runtime computation. If my program needs to know the multiplicative product of two inputs between 1 and 100, I can write a multiplication algorithm that computes each time the function is called, or I can precompute a Giant Lookup Table with 10,000 entries and two indices. There are times when you do want to do this, though not for multiplication - times when you're going to reuse the function a lot and it doesn't have many possible inputs; or when clock cycles are cheap while you're initializing, but very expensive while executing.

Giant Lookup Tables get very large, very fast. A GLUT of all possible twenty-ply conversations with ten words per remark, using only 850-word Basic English, would require 7.6 * 10^585 entries.

Replacing a human brain with a Giant Lookup Table of all possible sense inputs and motor outputs (relative to some fine-grained digitization scheme) would require an unreasonably large amount of memory storage. But "in principle", as philosophers are fond of saying, it could be done.

The GLUT is not a zombie in the classic sense, because it is microphysically dissimilar to a human. (In fact, a GLUT can't really run on the same physics as a human; it's too large to fit in our universe. For philosophical purposes, we shall ignore this and suppose a supply of unlimited memory storage.)

But is the GLUT a zombie at all? That is, does it behave exactly like a human without being conscious?

The GLUT-ed body's tongue talks about consciousness. Its fingers write philosophy papers. In every way, so long as you don't peer inside the skull, the GLUT seems just like a human... which certainly seems like a valid example of a zombie: it behaves just like a human, but there's no one home.

Unless the GLUT is conscious, in which case it wouldn't be a valid example.

I can't recall ever seeing anyone claim that a GLUT is conscious. (Admittedly my reading in this area is not up to professional grade; feel free to correct me.) Even people who are accused of being (gasp!) functionalists don't claim that GLUTs can be conscious.

GLUTs are the reductio ad absurdum to anyone who suggests that consciousness is simply an input-output pattern, thereby disposing of all troublesome worries about what goes on inside.

So what does the Generalized Anti-Zombie Principle (GAZP) say about the Giant Lookup Table (GLUT)?

At first glance, it would seem that a GLUT is the very archetype of a Zombie Master - a distinct, additional, detectable, non-conscious system that animates a zombie and makes it talk about consciousness for different reasons.

In the interior of the GLUT, there's merely a very simple computer program that looks up inputs and retrieves outputs. Even talking about a "simple computer program" is overshooting the mark, in a case like this. A GLUT is more like ROM than a CPU. We could equally well talk about a series of switched tracks by which some balls roll out of a previously stored stack and into a trough - period; that's all the GLUT does.

A spokesperson from People for the Ethical Treatment of Zombies replies: "Oh, that's what all the anti-mechanists say, isn't it? That when you look in the brain, you just find a bunch of neurotransmitters opening ion channels? If ion channels can be conscious, why not levers and balls rolling into bins?"

"The problem isn't the levers," replies the functionalist, "the problem is that a GLUT has the wrong pattern of levers. You need levers that implement things like, say, formation of beliefs about beliefs, or self-modeling... Heck, you need the ability to write things to memory just so that time can pass for the computation. Unless you think it's possible to program a conscious being in Haskell."

"I don't know about that," says the PETZ spokesperson, "all I know is that this so-called zombie writes philosophical papers about consciousness. Where do these philosophy papers come from, if not from consciousness?"

Good question! Let us ponder it deeply.

There's a game in physics called Follow-The-Energy. Richard Feynman's father played it with young Richard:

It was the kind of thing my father would have talked about: "What makes it go? Everything goes because the sun is shining." And then we would have fun discussing it:
"No, the toy goes because the spring is wound up," I would say. "How did the spring get wound up?" he would ask.
"I wound it up."
"And how did you get moving?"
"From eating."
"And food grows only because the sun is shining. So it's because the sun is shining that all these things are moving." That would get the concept across that motion is simply the transformation of the sun's power.

When you get a little older, you learn that energy is conserved, never created or destroyed, so the notion of using up energy doesn't make much sense. You can never change the total amount of energy, so in what sense are you using it?

So when physicists grow up, they learn to play a new game called Follow-The-Negentropy - which is really the same game they were playing all along; only the rules are mathier, the game is more useful, and the principles are harder to wrap your mind around conceptually.

Rationalists learn a game called Follow-The-Improbability, the grownup version of "How Do You Know?" The rule of the rationalist's game is that every improbable-seeming belief needs an equivalent amount of evidence to justify it. (This game has amazingly similar rules to Follow-The-Negentropy.)

Whenever someone violates the rules of the rationalist's game, you can find a place in their argument where a quantity of improbability appears from nowhere; and this is as much a sign of a problem as, oh, say, an ingenious design of linked wheels and gears that keeps itself running forever.

The one comes to you and says: "I believe with firm and abiding faith that there's an object in the asteroid belt, one foot across and composed entirely of chocolate cake; you can't prove that this is impossible." But, unless the one had access to some kind of evidence for this belief, it would be highly improbable for a correct belief to form spontaneously. So either the one can point to evidence, or the belief won't turn out to be true. "But you can't prove it's impossible for my mind to spontaneously generate a belief that happens to be correct!" No, but that kind of spontaneous generation is highly improbable, just like, oh, say, an egg unscrambling itself.

In Follow-The-Improbability, it's highly suspicious to even talk about a specific hypothesis without having had enough evidence to narrow down the space of possible hypotheses. Why aren't you giving equal air time to a decillion other equally plausible hypotheses? You need sufficient evidence to find the "chocolate cake in the asteroid belt" hypothesis in the hypothesis space - otherwise there's no reason to give it more air time than a trillion other candidates like "There's a wooden dresser in the asteroid belt" or "The Flying Spaghetti Monster threw up on my sneakers."

In Follow-The-Improbability, you are not allowed to pull out big complicated specific hypotheses from thin air without already having a corresponding amount of evidence; because it's not realistic to suppose that you could spontaneously start discussing the true hypothesis by pure coincidence.

A philosopher says, "This zombie's skull contains a Giant Lookup Table of all the inputs and outputs for some human's brain." This is a very large improbability. So you ask, "How did this improbable event occur? Where did the GLUT come from?"

Now this is not standard philosophical procedure for thought experiments. In standard philosophical procedure, you are allowed to postulate things like "Suppose you were riding a beam of light..." without worrying about physical possibility, let alone mere improbability. But in this case, the origin of the GLUT matters; and that's why it's important to understand the motivating question, "Where did the improbability come from?"

The obvious answer is that you took a computational specification of a human brain, and used that to precompute the Giant Lookup Table. (Thereby creating uncounted googols of human beings, some of them in extreme pain, the supermajority gone quite mad in a universe of chaos where inputs bear no relation to outputs. But damn the ethics, this is for philosophy.)

In this case, the GLUT is writing papers about consciousness because of a conscious algorithm. The GLUT is no more a zombie, than a cellphone is a zombie because it can talk about consciousness while being just a small consumer electronic device. The cellphone is just transmitting philosophy speeches from whoever happens to be on the other end of the line. A GLUT generated from an originally human brain-specification is doing the same thing.

"All right," says the philosopher, "the GLUT was generated randomly, and just happens to have the same input-output relations as some reference human."

How, exactly, did you randomly generate the GLUT?

"We used a true randomness source - a quantum device."

But a quantum device just implements the Branch Both Ways instruction; when you generate a bit from a quantum randomness source, the deterministic result is that one set of universe-branches (locally connected amplitude clouds) see 1, and another set of universes see 0. Do it 4 times, create 16 (sets of) universes.

So, really, this is like saying that you got the GLUT by writing down all possible GLUT-sized sequences of 0s and 1s, in a really damn huge bin of lookup tables; and then reaching into the bin, and somehow pulling out a GLUT that happened to correspond to a human brain-specification. Where did the improbability come from?

Because if this wasn't just a coincidence - if you had some reach-into-the-bin function that pulled out a human-corresponding GLUT by design, not just chance - then that reach-into-the-bin function is probably conscious, and so the GLUT is again a cellphone, not a zombie. It's connected to a human at two removes, instead of one, but it's still a cellphone! Nice try at concealing the source of the improbability there!

Now behold where Follow-The-Improbability has taken us: where is the source of this body's tongue talking about an inner listener? The consciousness isn't in the lookup table. The consciousness isn't in the factory that manufactures lots of possible lookup tables. The consciousness was in whatever pointed to one particular already-manufactured lookup table, and said, "Use that one!"

You can see why I introduced the game of Follow-The-Improbability. Ordinarily, when we're talking to a person, we tend to think that whatever is inside the skull, must be "where the consciousness is". It's only by playing Follow-The-Improbability that we can realize that the real source of the conversation we're having, is that-which-is-responsible-for the improbability of the conversation - however distant in time or space, as the Sun moves a wind-up toy.

"No, no!" says the philosopher. "In the thought experiment, they aren't randomly generating lots of GLUTs, and then using a conscious algorithm to pick out one GLUT that seems humanlike! I am specifying that, in this thought experiment, they reach into the inconceivably vast GLUT bin, and by pure chance pull out a GLUT that is identical to a human brain's inputs and outputs! There! I've got you cornered now! You can't play Follow-The-Improbability any further!"

Oh. So your specification is the source of the improbability here.

When we play Follow-The-Improbability again, we end up outside the thought experiment, looking at the philosopher.

That which points to the one GLUT that talks about consciousness, out of all the vast space of possibilities, is now... the conscious person asking us to imagine this whole scenario. And our own brains, which will fill in the blank when we imagine, "What will this GLUT say in response to 'Talk about your inner listener'?"

The moral of this story is that when you follow back discourse about "consciousness", you generally find consciousness. It's not always right in front of you. Sometimes it's very cleverly hidden. But it's there. Hence the Generalized Anti-Zombie Principle.

If there is a Zombie Master in the form of a chatbot that processes and remixes amateur human discourse about "consciousness", the humans who generated the original text corpus are conscious.

If someday you come to understand consciousness, and look back, and see that there's a program you can write which will output confused philosophical discourse that sounds an awful lot like humans without itself being conscious - then when I ask "How did this program come to sound similar to humans?" the answer is that you wrote it to sound similar to conscious humans, rather than choosing on the criterion of similarity to something else. This doesn't mean your little Zombie Master is conscious - but it does mean I can find consciousness somewhere in the universe by tracing back the chain of causality, which means we're not entirely in the Zombie World.

But suppose someone actually did reach into a GLUT-bin and by genuinely pure chance pulled out a GLUT that wrote philosophy papers?

Well, then it wouldn't be conscious. IMHO.

I mean, there's got to be more to it than inputs and outputs.

Otherwise even a GLUT would be conscious, right?

Oh, and for those of you wondering how this sort of thing relates to my day job...

In this line of business you meet an awful lot of people who think that an arbitrarily generated powerful AI will be "moral". They can't agree among themselves on why, or what they mean by the word "moral"; but they all agree that doing Friendly AI theory is unnecessary. And when you ask them how an arbitrarily generated AI ends up with moral outputs, they proffer elaborate rationalizations aimed at AIs of that which they deem "moral"; and there are all sorts of problems with this, but the number one problem is, "Are you sure the AI would follow the same line of thought you invented to argue human morals, when, unlike you, the AI doesn't start out knowing what you want it to rationalize?" You could call the counter-principle Follow-The-Decision-Information, or something along those lines. You can account for an AI that does improbably nice things by telling me how you chose the AI's design from a huge space of possibilities, but otherwise the improbability is being pulled out of nowhere - though more and more heavily disguised, as rationalized premises are rationalized in turn.

So I've already done a whole series of posts which I myself generated using Follow-The-Improbability. But I didn't spell out the rules explicitly at that time, because I hadn't done the thermodynamic posts yet...

Just thought I'd mention that. It's amazing how many of my Overcoming Bias posts would coincidentally turn out to include ideas surprisingly relevant to discussion of Friendly AI theory... if you believe in coincidence.

Sequence: Zombies

The Generalized Anti-Zombie Principle

Belief in the Implied Invisible

Referenced by: Recognizing Intelligence

Original with comments: GAZP vs. GLUT

Belief in the Implied Invisible

Eliezer Yudkowsky, 08 April 2008 07:40AM

Followup to: The Generalized Anti-Zombie Principle

One generalized lesson not to learn from the Anti-Zombie Argument is, "Anything you can't see doesn't exist."

It's tempting to conclude the general rule. It would make the Anti-Zombie Argument much simpler, on future occasions, if we could take this as a premise. But unfortunately that's just not Bayesian.

Suppose I transmit a photon out toward infinity, not aimed at any stars, or any galaxies, pointing it toward one of the great voids between superclusters. Based on standard physics, in other words, I don't expect this photon to intercept anything on its way out. The photon is moving at light speed, so I can't chase after it and capture it again.

If the expansion of the universe is accelerating, as current cosmology holds, there will come a future point where I don't expect to be able to interact with the photon even in principle - a future time beyond which I don't expect the photon's future light cone to intercept my world-line. Even if an alien species captured the photon and rushed back to tell us, they couldn't travel fast enough to make up for the accelerating expansion of the universe.

Should I believe that, in the moment where I can no longer interact with it even in principle, the photon disappears?

No.

It would violate Conservation of Energy. And the second law of thermodynamics. And just about every other law of physics. And probably the Three Laws of Robotics. It would imply the photon knows I care about it and knows exactly when to disappear.

It's a silly idea.

But if you can believe in the continued existence of photons that have become experimentally undetectable to you, why doesn't this imply a general license to believe in the invisible?

(If you want to think about this question on your own, do so before the jump...)

Though I failed to Google a source, I remember reading that when it was first proposed that the Milky Way was our galaxy - that the hazy river of light in the night sky was made up of millions (or even billions) of stars - that Occam's Razor was invoked against the new hypothesis. Because, you see, the hypothesis vastly multiplied the number of "entities" in the believed universe. Or maybe it was the suggestion that "nebulae" - those hazy patches seen through a telescope - might be galaxies full of stars, that got the invocation of Occam's Razor.

Lex parsimoniae: Entia non sunt multiplicanda praeter necessitatem.

That was Occam's original formulation, the law of parsimony: Entities should not be multiplied beyond necessity.

If you postulate billions of stars that no one has ever believed in before, you're multiplying entities, aren't you?

No. There are two Bayesian formalizations of Occam's Razor: Solomonoff Induction, and Minimum Message Length. Neither penalizes galaxies for being big.

Which they had better not do! One of the lessons of history is that what-we-call-reality keeps turning out to be bigger and bigger and huger yet. Remember when the Earth was at the center of the universe? Remember when no one had invented Avogadro's number? If Occam's Razor was weighing against the multiplication of entities every time, we'd have to start doubting Occam's Razor, because it would have consistently turned out to be wrong.

In Solomonoff induction, the complexity of your model is the amount of code in the computer program you have to write to simulate your model. The amount of code, not the amount of RAM it uses, or the number of cycles it takes to compute. A model of the universe that contains billions of galaxies containing billions of stars, each star made of a billion trillion decillion quarks, will take a lot of RAM to run - but the code only has to describe the behavior of the quarks, and the stars and galaxies can be left to run themselves. I am speaking semi-metaphorically here - there are things in the universe besides quarks - but the point is, postulating an extra billion galaxies doesn't count against the size of your code, if you've already described one galaxy. It just takes a bit more RAM, and Occam's Razor doesn't care about RAM.

Why not? The Minimum Message Length formalism, which is nearly equivalent to Solomonoff Induction, may make the principle clearer: If you have to tell someone how your model of the universe works, you don't have to individually specify the location of each quark in each star in each galaxy. You just have to write down some equations. The amount of "stuff" that obeys the equation doesn't affect how long it takes to write the equation down. If you encode the equation into a file, and the file is 100 bits long, then there are 2^100 other models that would be around the same file size, and you'll need roughly 100 bits of supporting evidence. You've got a limited amount of probability mass; and a priori, you've got to divide that mass up among all the messages you could send; and so postulating a model from within a model space of 2^100 alternatives, means you've got to accept a 2-100 prior probability penalty - but having more galaxies doesn't add to this.

Postulating billions of stars in billions of galaxies doesn't affect the length of your message describing the overall behavior of all those galaxies. So you don't take a probability hit from having the same equations describing more things. (So long as your model's predictive successes aren't sensitive to the exact initial conditions. If you've got to specify the exact positions of all the quarks for your model to predict as well as it does, the extra quarks do count as a hit.)

If you suppose that the photon disappears when you are no longer looking at it, this is an additional law in your model of the universe. It's the laws that are "entities", costly under the laws of parsimony. Extra quarks are free.

So does it boil down to, "I believe the photon goes on existing as it wings off to nowhere, because my priors say it's simpler for it to go on existing than to disappear"?

This is what I thought at first, but on reflection, it's not quite right. (And not just because it opens the door to obvious abuses.)

I would boil it down to a distinction between belief in the implied invisible, and belief in the additional invisible.

When you believe that the photon goes on existing as it wings out to infinity, you're not believing that as an additional fact.

What you believe (assign probability to) is a set of simple equations; you believe these equations describe the universe. You believe these equations because they are the simplest equations you could find that describe the evidence. These equations are highly experimentally testable; they explain huge mounds of evidence visible in the past, and predict the results of many observations in the future.

You believe these equations, and it is a logical implication of these equations that the photon goes on existing as it wings off to nowhere, so you believe that as well.

Your priors, or even your probabilities, don't directly talk about the photon. What you assign probability to is not the photon, but the general laws. When you assign probability to the laws of physics as we know them, you automatically contribute that same probability to the photon continuing to exist on its way to nowhere - if you believe the logical implications of what you believe.

It's not that you believe in the invisible as such, from reasoning about invisible things. Rather the experimental evidence supports certain laws, and belief in those laws logically implies the existence of certain entities that you can't interact with. This is belief in the implied invisible.

On the other hand, if you believe that the photon is eaten out of existence by the Flying Spaghetti Monster - maybe on this just one occasion - or even if you believed without reason that the photon hit a dust speck on its way out - then you would be believing in a specific extra invisible event, on its own. If you thought that this sort of thing happened in general, you would believe in a specific extra invisible law. This is belief in the additional invisible.

The whole matter would be a lot simpler, admittedly, if we could just rule out the existence of entities we can't interact with, once and for all - have the universe stop existing at the edge of our telescopes. But this requires us to be very silly.

Saying that you shouldn't ever need a separate and additional belief about invisible things - that you only believe invisibles that are logical implications of general laws which are themselves testable, and even then, don't have any further beliefs about them that are not logical implications of visibly testable general rules - actually does seem to rule out all abuses of belief in the invisible, when applied correctly.

Perhaps I should say, "you should assign unaltered prior probability to additional invisibles", rather than saying, "do not believe in them." But if you think of a belief as something evidentially additional, something you bother to track, something where you bother to count up support for or against, then it's questionable whether we should ever have additional beliefs about additional invisibles.

There are exotic cases that break this in theory. (E.g: The epiphenomenal demons are watching you, and will torture 3^^^3 victims for a year, somewhere you can't ever verify the event, if you ever say the word "Niblick".) But I can't think of a case where the principle fails in human practice.

Added: To make it clear why you would sometimes want to think about implied invisibles, suppose you're going to launch a spaceship, at nearly the speed of light, toward a faraway supercluster. By the time the spaceship gets there and sets up a colony, the universe's expansion will have accelerated too much for them to ever send a message back. Do you deem it worth the purely altruistic effort to set up this colony, for the sake of all the people who will live there and be happy? Or do you think the spaceship blips out of existence before it gets there? This could be a very real question at some point.

Sequence: Quantum Physics

Zombies! Zombies?

Quantum Explanations

Sequence: Zombies

GAZP vs. GLUT

Zombies: The Movie

Referenced by: Distinct Configurations • On Being Decoherent • Decoherence is Simple • Decoherence is Falsifiable and Testable • The Quantum Physics Sequence • And the Winner is... Many-Worlds! • No Logical Positivist I

Original with comments: Belief in the Implied Invisible

Quantum Explanations

Eliezer Yudkowsky, 09 April 2008 08:15AM

This post is part of the Quantum Physics Sequence.

I think I must now temporarily digress from the sequence on zombies (which was a digression from the discussion of reductionism, which was a digression from the Mind Projection Fallacy) in order to discuss quantum mechanics. The reasons why this belongs in the middle of a discussion on zombies in the middle of a discussion of reductionism in the middle of a discussion of the Mind Projection Fallacy, will become apparent eventually.

It's a sequence that has been weighing on my mind, demanding to be written, for a quite a long time. Years. This seems like a good time to insert it.

I wrote the "Intuitive Explanation of Bayesian Reasoning" because people were complaining that Bayes's Theorem was "counterintuitive" - in fact it was famously counterintuitive - and this did not seem right. The equation just did not seem complicated enough to deserve the fearsome reputation it had. So I tried explaining it my way, and I did not manage to reach my original target of elementary school students, but I get frequent grateful emails from formerly confused folks ranging from reporters to outside academic college professors.

I am not a physicist, and physicists famously hate it when non-professional-physicists talk about QM. But I do have some experience with explaining mathy things that are allegedly "hard to understand".

Besides, as a Bayesian, I don't believe in phenomena that are inherently confusing. Confusion exists in our models of the world, not in the world itself. If a subject is widely known as confusing, not just difficult... you shouldn't leave it at that. It doesn't satisfice; it is not an okay place to be. Maybe you can fix the problem, maybe you can't; but you shouldn't be happy to leave students confused.

The first way in which my introduction is going to depart from the traditional, standard introduction to QM, is that I am not going to tell you that quantum mechanics is supposed to be confusing.

I am not going to tell you that it's okay for you to not understand quantum mechanics, because no one understands quantum mechanics, as Richard Feynman once claimed. There was a historical time when this was true, but we no longer live in that era.

I am not going to tell you: "You don't understand quantum mechanics, you just get used to it." (As von Neumann is reputed to have said; back in the dark decades when, in fact, no one did understand quantum mechanics.)

Explanations are supposed to make you less confused. If you feel like you don't understand something, this indicates a problem - either with you, or your teacher - but at any rate a problem; and you should move to resolve the problem.

I am not going to tell you that quantum mechanics is weird, bizarre, confusing, or alien. QM is counterintuitive, but that is a problem with your intuitions, not a problem with quantum mechanics. Quantum mechanics has been around for billions of years before the Sun coalesced from interstellar hydrogen. Quantum mechanics was here before you were, and if you have a problem with that, you are the one who needs to change. QM sure won't. There are no surprising facts, only models that are surprised by facts; and if a model is surprised by the facts, it is no credit to that model.

It is always best to think of reality as perfectly normal. Since the beginning, not one unusual thing has ever happened.

The goal is to become completely at home in a quantum universe. Like a native. Because, in fact, that is where you live.

In the coming sequence on quantum mechanics, I am going to consistently speak as if quantum mechanics is perfectly normal; and when human intuitions depart from quantum mechanics, I am going to make fun of the intuitions for being weird and unusual. This may seem odd, but the point is to swing your mind around to a native quantum point of view.

Another thing: The traditional introduction to quantum mechanics closely follows the order in which quantum mechanics was discovered.

The traditional introduction starts by saying that matter sometimes behaves like little billiard balls bopping around, and sometimes behaves like crests and troughs moving through a pool of water. Then the traditional introduction gives some examples of matter acting like a little billiard ball, and some examples of it acting like an ocean wave.

Now, it happens to be a historical fact that, back when students of matter were working all this stuff out and had no clue about the true underlying math, those early scientists first thought that matter was like little billiard balls. And then that it was like waves in the ocean. And then that it was like billiard balls again. And then the early scientists got really confused, and stayed that way for several decades, until it was finally sorted out in the second half of the twentieth century.

Dragging a modern-day student through all this may be a historically realistic approach to the subject matter, but it also ensures the historically realistic outcome of total bewilderment. Talking to aspiring young physicists about "wave/particle duality" is like starting chemistry students on the Four Elements.

An electron is not a billiard ball, and it's not a crest and trough moving through a pool of water. An electron is a mathematically different sort of entity, all the time and under all circumstances, and it has to be accepted on its own terms.

The universe is not wavering between using particles and waves, unable to make up its mind. It's only human intuitions about QM that swap back and forth. The intuitions we have for billiard balls, and the intuitions we have for crests and troughs in a pool of water, both look sort of like they're applicable to electrons, at different times and under different circumstances. But the truth is that both intuitions simply aren't applicable.

If you try to think of an electron as being like a billiard ball on some days, and like an ocean wave on other days, you will confuse the living daylights out of yourself.

Yet it's your eyes that are wobbling and unstable, not the world.

Furthermore:

The order in which humanity discovered things is not necessarily the best order in which to teach them. First, humanity noticed that there were other animals running around. Then we cut them open and found that they were full of organs. Then we examined the organs carefully and found they were made of tissues. Then we looked at the tissues under a microscope and discovered cells, which are made of proteins and some other chemically synthesized stuff. Which are made of molecules, which are made of atoms, which are made of protons and neutrons and electrons which are way simpler than entire animals but were discovered tens of thousands of years later.

Physics doesn't start by talking about biology. So why should it start by talking about very high-level complicated phenomena, like, say, the observed results of experiments?

The ordinary way of teaching QM keeps stressing the experimental results. Now I do understand why that sounds nice from a rationalist perspective. Believe me, I understand.

But it seems to me that the upshot is dragging in big complicated mathematical tools that you need to analyze real-world situations, before the student understands what fundamentally goes on the simplest cases.

It's like trying to teach programmers how to write concurrent multithreaded programs before they know how to add two variables together, because concurrent multithreaded programs are closer to everyday life. Being close to everyday life is not always a strong recommendation for what to teach first.

Maybe the monomaniacal focus on experimental observations made sense in the dark decades when no one understood what was fundamentally going on, and you couldn't start there, and all your models were just mysterious maths that gave good experimental predictions... you can still find this view of quantum physics presented in many books... but maybe today it's worth trying a different angle? The result of the standard approach is standard confusion.

The classical world is strictly implicit in the quantum world, but seeing from a classical perspective makes everything bigger and more complicated. Everyday life is a higher level of organization, like molecules versus quarks - huge catalogue of molecules, six quarks. I think it is worth trying to teach from the perspective of the quantum world first, and talking about classical experimental results afterward.

I am not going to start with the normal classical world and then talk about a bizarre quantum backdrop hidden behind the scenes. The quantum world is the scene and it defines normality.

I am not going to talk as if the classical world is real life, and occasionally the classical world transmits a request for an experimental result to a quantum-physics server, and the quantum-physics server does some peculiar calculations and transmits back a classical experimental result. I am going to talk as if the quantum world is the really real and the classical world something far away. Not just because that makes it easier to be a native of a quantum universe, but because, at a core level, it's the truth.

Finally, I am going to take a strictly realist perspective on quantum mechanics - the quantum world is really out there, our equations describe the territory and not our maps of it, and the classical world only exists implicitly within the quantum one. I am not going to discuss non-realist views in the early stages of my introduction, except to say why you should not be confused by certain intuitions that non-realists draw upon for support. I am not going to apologize for this, and I would like to ask any non-realists on the subject of quantum mechanics to wait and hold their comments until called for in a later post. Do me this favor, please. I think non-realism is one of the main things that confuses prospective students, and prevents them from being able to concretely visualize quantum phenomena. I will discuss the issues explicitly in a future post.

But everyone should be aware that, even though I'm not going to discuss the issue at first, there is a sizable community of scientists who dispute the realist perspective on QM. Myself, I don't think it's worth figuring both ways; I'm a pure realist, for reasons that will become apparent. But if you read my introduction, you are getting my view. It is not only my view. It is probably the majority view among theoretical physicists, if that counts for anything (though I will argue the matter separately from opinion polls). Still, it is not the only view that exists in the modern physics community. I do not feel obliged to present the other views right away, but I feel obliged to warn my readers that there are other views, which I will not be presenting during the initial stages of the introduction.

To sum up, my goal will be to teach you to think like a native of a quantum universe, not a reluctant tourist.

Embrace reality. Hug it tight.

Sequence: Quantum Physics

Belief in the Implied Invisible

Configurations and Amplitude

Referenced by: Configurations and Amplitude • No Individual Particles • Identity Isn't In Specific Atoms • Decoherent Essences • Bell's Theorem: No EPR "Reality" • Timeless Physics • Timeless Identity • Why Quantum? • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Quantum Explanations

Configurations and Amplitude

Eliezer Yudkowsky, 10 April 2008 07:41AM

This post is part of the Quantum Physics Sequence.
Previously in series: Quantum Explanations

So the universe isn't made of little billiard balls, and it isn't made of crests and troughs in a pool of aether... Then what is the stuff that stuff is made of?

(Diagrams stolen from qubit.org and edited for my purposes.)

In Figure 1, we see, at A, a half-silvered mirror, and two photon detectors, 1 and 2.

Early scientists, when they ran experiments like this, became confused about what the results meant. They would send a photon toward the half-silvered mirror, and half the time they would see the detector at 1 click, and the other half the time they would see the detector at 2 click.

The early scientists - you're going to laugh at this - thought that the silver mirror deflected the photon half the time, and let it through half the time.

Ha, ha! As if the half-silvered mirror did different things on different occasions! I want you to let go of this idea, because if you cling to what early scientists thought, you will become extremely confused. The half-silvered mirror obeys the same rule every time.

If you were going to write a computer program that was this experiment - not a computer program that predicted the result of the experiment, but a computer program that resembled the underlying reality - it might look sort of like this:

At the start of the program (the start of the experiment, the start of time) there's a certain mathematical entity, called a configuration. You can think of this configuration as corresponding to "There is one photon heading from the photon source toward the half-silvered mirror", or just "A photon heading toward A."

A configuration can store a single complex value - "complex" as in the complex numbers (a + bi). At the start of the program, there's already a complex number stored in the configuration "A photon heading toward A." The exact value doesn't matter so long as it's not zero. We'll let the configuration "A photon heading toward A" have a value of (-1 + 0i).

All this is a fact within the territory, not a description of anyone's knowledge. A configuration isn't a proposition or a possibility or a way the world can be. A configuration is a variable in the program - you can think of it as a kind of memory location whose index is "A photon heading toward A" - and it's out there in the territory.

As the complex numbers that get assigned to configurations are not positive real numbers between 0 and 1, there is no danger of confusing them with probabilities. "A photon heading toward A" has complex value -1, which is hard to see as a degree of belief. The complex numbers are values within the program, again out there in the territory. We'll call the complex numbers amplitudes.

There are two other configurations, which we'll call "A photon going from A to Detector 1" and "A photon going from A to Detector 2." These configurations don't have a complex value yet; it gets assigned as the program runs.

We are going to calculate the amplitudes of "A photon going from A toward 1" and "A photon going from A toward 2" using the value of "A photon going toward A", and the rule that describes the half-silvered mirror at A.

Roughly speaking, the half-silvered mirror rule is "Multiply by 1 when the photon goes straight, and multiply by i when the photon turns at a right angle." This is the universal rule that relates the amplitude of the configuration of "a photon going in", to the amplitude that goes to the configurations of "a photon coming out straight" or "a photon being deflected".

So we pipe the amplitude of the configuration "A photon going toward A", which is (-1 + 0i), into the half-silvered mirror at A, and this transmits an amplitude of (-1 + 0i)*i = (0 + -i) to "A photon going from A toward 1", and also transmits an amplitude of (-1 + 0i)*1 = (-1 + 0i) to "A photon going from A toward 2".

In the Figure 1 experiment, these are all the configurations and all the transmitted amplitude we need to worry about, so we're done. Or, if you want to think of "Detector 1 gets a photon" and "Detector 2 gets a photon" as separate configurations, they'd just inherit their values from "A->1" and "A->2" respectively. (Actually, the values inherited should be multiplied by another complex factor, corresponding from the distance from A to the detector; but we will ignore that for today, and suppose that all distances traveled in our experiments happen to correspond to a complex factor of 1.)

So the final program state is:

Configuration "A photon going toward A": (-1 + 0i)
Configuration "A photon going from A toward 1": (0 + -i)
Configuration "A photon going from A toward 2": (-1 + 0i)
and optionally
Configuration "Detector 1 gets a photon": (0 + -i)
Configuration "Detector 2 gets a photon": (-1 + 0i)

This same result occurs - the same amplitudes stored in the same configurations - every time you run the program (every time you do the experiment).

Now, for complicated reasons that we aren't going to go into today - considerations that belong on a higher level of organization than fundamental quantum mechanics, the same way that atoms are more complicated than quarks - there's no simple measuring instrument that can directly tell us the exact amplitudes of each configuration. We can't directly see the program state.

So how do physicists know what the amplitudes are?

We do have a magical measuring tool that can tell us the squared modulus of a configuration's amplitude. If the original complex amplitude is (a + bi), we can get the positive real number (a^2 + b^2). Think of the Pythagorean theorem: if you imagine the complex number as a little arrow stretching out from the origin on a two-dimensional plane, then the magic tool tells us the squared length of the little arrow, but it doesn't tell us the direction the arrow is pointing.

To be more precise, the magic tool actually just tells us the ratios of the squared lengths of the amplitudes in some configurations. We don't know how long the arrows are in an absolute sense, just how long they are relative to each other. But this turns out to be enough information to let us reconstruct the laws of physics - the rules of the program. And so I can talk about amplitudes, not just ratios of squared moduli.

When we wave the magic tool over "Detector 1 gets a photon" and "Detector 2 gets a photon", we discover that these configurations have the same squared modulus - the lengths of the arrows are the same. Thus speaks the magic tool. By doing more complicated experiments (to be seen shortly), we can tell that the original complex numbers had a ratio of i to 1.

And what is this magical measuring tool?

Well, from the perspective of everyday life - way, way, way above the quantum level and a lot more complicated - the magical measuring tool is that we send some photons toward the half-silvered mirror, one at a time, and count up how many photons arrive at Detector 1 versus Detector 2 over a few thousand trials. The ratio of these values is the ratio of the squared moduli of the amplitudes. But the reason for this is not something we are going to consider yet. Walk before you run. It is not possible to understand what happens all the way up at the level of everyday life, before you understand what goes on in much simpler cases.

For today's purposes, we have a magical squared-modulus-ratio reader. And the magic tool tells us that the little two-dimensional arrow for the configuration "Detector 1 gets a photon" has the same squared length as for "Detector 2 gets a photon". That's all.

You may wonder, "Given that the magic tool works this way, what motivates us to use quantum theory, instead of thinking that the half-silvered mirror reflects the photon around half the time?"

Well, that's just begging to be confused - putting yourself into a historically realistic frame of mind like that and using everyday intuitions. Did I say anything about a little billiard ball going one way or the other and possibly bouncing off a mirror? That's not how reality works. Reality is about complex amplitudes flowing between configurations, and the laws of the flow are stable.

But if you insist on seeing a more complicated situation that billiard-ball ways of thinking can't handle, here's a more complicated experiment:

In Figure 2, B and C are full mirrors, and A and D are half-mirrors. The line from D to E is dashed for reasons that will become apparent, but amplitude is flowing from D to E under exactly the same laws.

Now let's apply the rules we learned before:

At the beginning of time "A photon heading toward A" has amplitude (-1 + 0i).

We proceed to compute the amplitude for the configurations "A photon going from A to B" and "A photon going from A to C".

"A photon going from A to B" = i * "A photon heading toward A" = (0 + -i)

Similarly,

"A photon going from A to C" = 1 * "A photon heading toward A" = (-1 + 0i)

The full mirrors behave (as one would expect) like half of a half-silvered mirror - a full mirror just bends things by right angles and multiplies them by i. (To state this slightly more precisely: For a full mirror, the amplitude that flows, from the configuration of a photon heading in, to the configuration of a photon heading out at a right angle, is multiplied by a factor of i.)

So:

"A photon going from B to D" = i * "A photon going from A to B" = (1 + 0i)
"A photon going from C to D" = i * "A photon going from A to C" = (0 + -i)

"B to D" and "C to D" are two different configurations - we don't simply write "A photon at D" - because the photons are arriving at two different angles in these two different configurations. And what D does to a photon, depends on the angle at which the photon arrives.

Again, the rule (speaking loosely) is that when a half-silvered mirror bends light at a right angle, the amplitude that flows from the photon-going-in configuration to the photon-going-out configuration, is the amplitude of the photon-going-in configuration multiplied by i. And when two configurations are related by a half-silvered mirror letting light straight through, the amplitude that flows from the photon-going-in configuration is multiplied by 1.

So:

- From the configuration "A photon going from B to D", with original amplitude (1 + 0i)

- Amplitude of (1 + 0i) * i = (0 + i) flows to "A photon going from D to E"

- Amplitude of (1 + 0i) * 1 = (1 + 0i) flows to "A photon going from D to F".

- From the configuration "A photon going from C to D", with original amplitude (0 + -i)

- Amplitude of (0 + -i) * i = (1 + 0i) flows to "A photon going from D to F"

- Amplitude of (0 + -i) * 1 = (0 + -i) flows to "A photon going from D to E".

Therefore:

- The total amplitude flowing to configuration "A photon going from D to E" is (0 + i) + (0 + -i) = (0 + 0i) = 0.

- The total amplitude flowing to configuration "A photon going from D to F" is (1 + 0i) + (1 + 0i) = (2 + 0i).

(You may want to try working this out yourself on pen and paper if you lost track at any point.)

But the upshot, from that super-high-level "experimental" perspective that we think of as normal life, is that we see no photons detected at E. Every photon seems to end up at F. The ratio of squared moduli between "D to E" and "D to F" is 0 to 4. That's why the line from D to E is dashed, in this figure.

This is not something it is possible to explain by thinking of half-silvered mirrors deflecting little incoming billiard balls half the time. You've got to think in terms of amplitude flows.

If half-silvered mirrors deflected a little billiard ball half the time, in this setup, the little ball would end up at Detector 1 around half the time and Detector 2 around half the time. Which it doesn't. So don't think that.

You may say, "But wait a minute! I can think of another hypothesis that accounts for this result. What if, when a half-silvered mirror reflects a photon, it does something to the photon that ensures it doesn't get reflected next time? And when it lets a photon go through straight, it does something to the photon so it gets reflected next time."

Now really, there's no need to go making the rules so complicated. Occam's Razor, remember. Just stick with simple, normal amplitude flows between configurations.

But if you want another experiment that disproves your new alternative hypothesis, it's this one:

Here, we've left the whole experimental setup the same, and just put a little blocking object between B and D. This ensures that the amplitude of "A photon going from B to D" is 0.

Once you eliminate the amplitude contributions from that configuration, you end up with totals of (1 + 0i) in "A photon going from D to F", and (0 + -i) in "A photon going from D to E".

The squared moduli of (1 + 0i) and (0 + -i) are both 1, so the magic measuring tool should tell us that the ratio of squared moduli is 1. Way back up at the level where physicists exist, we should find that Detector 1 goes off half the time, and Detector 2 half the time.

The same thing happens if we put the block between C and D. The amplitudes are different, but the ratio of the squared moduli is still 1, so Detector 1 goes off half the time and Detector 2 goes off half the time.

This cannot possibly happen with a little billiard ball that either does or doesn't get reflected by the half-silvered mirrors.

Because complex numbers can have opposite directions, like 1 and -1, or i and -i, amplitude flows can cancel each other out. Amplitude flowing from configuration X into configuration Y can be canceled out by an equal and opposite amplitude flowing from configuration Z into configuration Y. In fact, that's exactly what happens in this experiment.

In probability theory, when something can either happen one way or another, X or ~X, then P(Z) = P(Z|X)P(X) + P(Z|~X)P(~X). And all probabilities are positive. So if you establish that the probability of Z happening given X is 1/2, and the probability of X happening is 1/3, then the total probability of Z happening is at least 1/6 no matter what goes on in the case of ~X. There's no such thing as negative probability, less-than-impossible credence, or (0 + i) credibility, so degrees of belief can't cancel each other out like amplitudes do.

Not to mention that probability is in the mind to begin with; and we are talking about the territory, the program-that-is-reality, not talking about human cognition or states of partial knowledge.

By the same token, configurations are not propositions, not statements, not ways the world could be. Configurations are not semantic constructs. Adjectives like probable and possible do not apply to them; they are not beliefs or sentences or possible worlds. They are not true or false but simply real.

In the experiment of Figure 2, at right, do not be tempted to think anything like: "The photon goes to either B or C, but it could have gone the other way, and this possibility interferes with its ability to go to E..."

It makes no sense to think of something that "could have happened but didn't" exerting an effect on the world. We can imagine things that could have happened but didn't - like thinking, "Gosh, that car almost hit me" - and our imagination can have an effect on our future behavior. But the event of imagination is a real event, that actually happens, and that is what has the effect. It's your imagination of the unreal event - your very real imagination, implemented within a quite physical brain - that affects your behavior.

To think that the actual event of a car hitting you - this event which could have happened to you, but in fact didn't - is directly exerting a causal effect on your behavior, is mixing up the map with the territory.

What affects the world is real. (If things can affect the world without being "real", it's hard to see what the word "real" means.) Configurations and amplitude flows are causes, and they have visible effects; they are real. Configurations are not possible worlds and amplitudes are not degrees of belief, any more than your chair is a possible world or the sky is a degree of belief.

So what is a configuration, then?

Well, you'll be getting a clearer idea of that in future posts.

But to give you a quick idea of how the real picture differs from the simplified version we saw today...

Our experimental setup only dealt with one moving particle, a single photon. Real configurations are about multiple particles. Tomorrow's post will deal with the case of more than one particle, and that should give you a much clearer idea of what a configuration is.

Each configuration we talked about, should have described a joint position of all the particles in the mirrors and detectors, not just the position of one photon bopping around.

In fact, the really real configurations are over joint positions of all the particles in the universe, including the particles making up the experimenters. You can see why I'm saving the notion of experimental results for later posts.

In the real world, amplitude is a continuous distribution over a continuous space of configurations. Today's "configurations" were blocky and digital, and so were our "amplitude flows". It was as if we were talking about a photon teleporting from one place to another.

We'll get atoms and molecules and humans and all that stuff, out of a differentiable amplitude distribution in a continuous configuration space, later.

If none of that made sense, don't worry. It will be cleared up in future posts. Just wanted to give you some idea of where this was heading.

Sequence: Quantum Physics

Quantum Explanations

Joint Configurations

Referenced by: Joint Configurations • Distinct Configurations • The So-Called Heisenberg Uncertainty Principle • Decoherence as Projection • Bell's Theorem: No EPR "Reality" • Many Worlds, One Best Guess • Timeless Physics • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Configurations and Amplitude

Joint Configurations

Eliezer Yudkowsky, 11 April 2008 05:00AM

This post is part of the Quantum Physics Sequence.
Previously in series: Configurations and Amplitude

The key to understanding configurations, and hence the key to understanding quantum mechanics, is realizing on a truly gut level that configurations are about more than one particle.

Continuing from yesterday, here's an altered version of the experiment where we send in two photons toward D at the same time, from the sources B and C.

The starting configuration then is:

"A photon going from B to D, and a photon going from C to D."

Again, let's say the starting configuration has amplitude (-1 + 0i).

And remember, the rule of the half-silvered mirror (at D) is that a right-angle deflection multiplies by i, and a straight line multiplies by 1.

So the amplitude flows from the starting configuration, separately considering the four cases of deflection/non-deflection of each photon, are:

1. The "B to D" photon is deflected and the "C to D" photon is deflected. This amplitude flows to the configuration "A photon going from D to E, and a photon going from D to F". The amplitude flowing is (-1 + 0i) * i * i = (1 + 0i).

2. The "B to D" photon is deflected and the "C to D" photon goes straight. This amplitude flows to the configuration "Two photons going from D to E". The amplitude flowing is (-1 + 0i) * i * 1 = (0 + -i).

3. The "B to D" photon goes straight and the "C to D" photon is deflected. This amplitude flows to the configuration "Two photons going from D to F". The amplitude flowing is (-1 + 0i) * 1 * i = (0 + -i).

4. The "B to D" photon goes straight and the "C to D" photon goes straight. This amplitude flows to the configuration "A photon going from D to F, and a photon going from D to E". The amplitude flowing is (-1 + 0i) * 1 * 1 = (-1 + 0i).

Now - and this is a very important and fundamental idea in quantum mechanics - the amplitudes in cases 1 and 4 are flowing to the same configuration. Whether the B photon and C photon both go straight, or both are deflected, the resulting configuration is one photon going toward E and another photon going toward F.

So we add up the two incoming amplitude flows from case 1 and case 4, and get a total amplitude of (1 + 0i) + (-1 + 0i) = 0.

When we wave our magic squared-modulus-ratio reader over the three final configurations, we'll find that "Two photons at Detector 1" and "Two photons at Detector 2" have the same squared modulus, but "A photon at Detector 1 and a photon at detector 2" has squared modulus zero.

Way up at the level of experiment, we never find Detector 1 and Detector 2 both going off. We'll find Detector 1 going off twice, or Detector 2 going off twice, with equal frequency.

(Assuming I've gotten the math and physics right. I didn't actually perform the experiment. If I got this wrong, I'm sure 50 commenters will tell me so in very short order.)

The configuration's identity is not, "The B photon going toward E and the C photon going toward F." Then the resultant configurations in case 1 and case 4 would not be equal. Case 1 would be, "B photon to E, C photon to F" and case 4 would be "B photon to F, C photon to E". These would be two distinguishable configurations, if configurations had photon-tracking structure.

So we would not add up the two amplitudes and cancel them out. We would keep the amplitudes in two separate configurations. The total amplitudes would have non-zero squared moduli. And when we ran the experiment, we would find (around half the time) that Detector 1 and Detector 2 each registered one photon. Which doesn't happen, if my calculations are correct.

Configurations don't keep track of where particles come from. A configuration's identity is just, "A photon here, a photon there; an electron here, an electron there." No matter how you get into that situation, so long as there are the same species of particles in the same places, it counts as the same configuration.

I say again that the question "What kind of information does the configuration's structure incorporate?" has experimental consequences. You can deduce, from experiment, the way that reality itself must be treating configurations.

In a classical universe, there would be no experimental consequences. If the photon were like a little billiard ball that either went one way or the other, and the configurations were our beliefs about possible states the system could be in, and instead of amplitudes we had probabilities, it would not make a difference whether we tracked the origin of photons or threw the information away.

In a classical universe, I could assign a 25% probability to both photons going to E, a 25% probability of both photons going to F, a 25% probability of the B photon going to E and the C photon going to F, and 25% probability of the B photon going to F and the C photon going to E. Or, since I personally don't care which of the two latter cases occurred, I could decide to collapse the two possibilities into one possibility and add up their probabilities, and just say, "A 50% probability that each detector gets one photon."

With probabilities, we can aggregate events as we like - draw our boundaries around sets of possible worlds as we please - and the numbers will still work out the same.

But you can't arbitrarily collapse configurations together, or split them apart, in your model, and get the same experimental predictions. Our magical tool tells us the ratios of squared moduli. When you add two complex numbers, the squared modulus of the sum is not the sum of the squared moduli of the parts:

Squared_Modulus(C1 + C2) != Squared_Modulus(C1) + Squared_Modulus(C2)

E.g:

S_M((2 + i) + (1 + -i)) = S_M(3 + 0i) = 3^2 + 0^2 = 9
(S_M(2 + i) + S_M(1 + -i)) = ((2^2 + 1^2) + (1^2 + (-1)^2)) = ((4 + 1) + (1 + 1)) = 7

Or in today's experiment of discourse, we had flows of (1 + 0i) and (-1 + 0i) cancel out, adding to 0 whose squared modulus is 0, where the squared modulus of the parts would have been 1 and 1.

Because the squared modulus of the sum does not equal the squared modulus of the parts, the question of which configurations are the same or distinct is experimentally distinguishable; a matter of objective fact. You cannot merge configurations that should be distinct, or split configurations that should be identical, without ending up with different results.

The probability of two mutually exclusive events equals the probability of the first event plus the probability of the second event. This is what lets us group possible worlds however we like during our calculations, without affecting our final probabilities.

Suppose I roll two six-sided dice (all the way up at the level of everyday life, where something like a classical level emerges). Then, when I calculate the probability of events like "both dice will show odd numbers", it does not depend whether I make a table of 36 possible outcomes for the exact numbers, or if I decide to think about 1, 3, 5 as "odd faces" and 2, 4, 6 as "even faces" and make a table of 4 possible outcomes. I can split up my knowledge any way I like - throw away information that doesn't make a difference to my current calculations, or keep the information, depending on my whim - and my predictions will always come out the same.

If in place of Squared_Modulus, our magical tool was some linear function - any function where F(X + Y) = F(X) + F(Y) - then all the quantumness would instantly vanish and be replaced by a classical physics. (A different classical physics, not the same illusion of classicality we hallucinate from inside the higher levels of organization in our own quantum world.)

If amplitudes were just probabilities, they couldn't cancel out when flows collided. If configurations were just states of knowledge, you could reorganize them however you liked.

But the configurations are nailed in place, indivisible and unmergeable without changing the laws of physics.

And part of what is nailed, is the way that configurations treat multiple particles. A configuration says, "A photon here, a photon there", not "This photon here, that photon there". "This photon here, that photon there" does not have a different identity from "That photon here, this photon there."

The result - we'll talk more about this in future posts, but it's visible already in today's experiment - is that you can't factorize the physics of our universe to be about particles with individual identities.

Part of the reason why humans have trouble coming to grips with perfectly normal quantum physics, is that humans bizarrely keep trying to factor reality into a sum of individually real billiard balls.

Ha ha! Silly humans.

Sequence: Quantum Physics

Configurations and Amplitude

Distinct Configurations

Referenced by: Distinct Configurations • Can You Prove Two Particles Are Identical? • No Individual Particles • The So-Called Heisenberg Uncertainty Principle • On Being Decoherent • Bell's Theorem: No EPR "Reality" • Spooky Action at a Distance: The No-Communication Theorem • Many Worlds, One Best Guess • Timeless Physics • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Joint Configurations

Distinct Configurations

Eliezer Yudkowsky, 12 April 2008 04:42AM

This post is part of the Quantum Physics Sequence.
Previously in series: Joint Configurations

Yesterday's experiment carried two key lessons:

First, we saw that because amplitude flows can cancel out, and because our magic measure of squared modulus is not linear, the identity of configurations is nailed down - you can't reorganize configurations the way you can regroup possible worlds. Which configurations are the same, and which are distinct, has experimental consequences; it is an observable fact.

Second, we saw that configurations are about multiple particles. If there are two photons entering the apparatus, that doesn't mean there are two initial configurations. Instead the initial configuration's identity is "Two photons coming in." (Ideally, each configuration we talk about would include every particle in the experiment - including the particles making up the mirrors and detectors. And in the real universe, every configuration is about all the particles... everywhere.)

What makes for distinct configurations is not distinct particles. Each configuration is about every particle. What makes configurations distinct, is particles occupying different positions - at least one particle in a different state.

To take one important demonstration...

This is the same experiment as Figure 2, with one important change: Between A and C has been placed a sensitive thingy, S. The key attribute of S is that if a photon goes past S, S ends up in a slightly different state.

Let's say that the two possible states of S are Yes and No. S starts out in state No, and ends up in state Yes if a photon goes past.

Then the initial configuration is:

"A photon heading toward A; and S in state No." (-1 + 0i)

Next, the action of the half-silvered mirror at A. In the previous version of this experiment, without the sensitive thingy, the two resultant configurations were "A to B" with amplitude -i and "A to C" with amplitude -1. Now, though, a new element has been introduced into the system, and all configurations are about all particles, and so every configuration mentions the new element. So the amplitude flows from the initial configuration are to:

"Photon from A to B; and S in state No." (0 + -i)
"Photon from A to C; and S in state Yes." (-1 + 0i)

Next, the action of the full mirrors at B and C:

"Photon from B to D; and S in state No." (1 + 0i)
"Photon from C to D; and S in state Yes." (0 + -i)

And then the action of the half-mirror at D, on the amplitude flowing from both of the above configurations:

1. "Photon from D to E; and S in state No." (0 + i)

2. "Photon from D to F; and S in state No." (1 + 0i)

3. "Photon from D to E; and S in state Yes." (0 + -i)

4. "Photon from D to F; and S in state Yes." (1 + 0i)

When we did this experiment without the sensitive thingy, the amplitude flows (1) and (3) of (0 + i) and (0 + -i) to the "D to E" configuration canceled each other out. We were left with no amplitude for a photon going to Detector 1 (way up at the experimental level, we never observe a photon striking Detector 1).

But in this case, the two amplitude flows (1) and (3) are now to distinct configurations; at least one entity, S, is in a different state between (1) and (3). The amplitudes don't cancel out.

When we wave our magical squared-modulus-ratio detector over the four final configurations, we find that the squared moduli of all are equal: 25% probability each. Way up at the level of the real world, we find that the photon has an equal chance of striking Detector 1 and Detector 2.

All the above is true, even if we, the researchers, don't care about the state of S. Unlike possible worlds, configurations cannot be regrouped on a whim. The laws of physics say the two configurations are distinct; it's not a question of how we can most conveniently parse up the world.

All the above is true, even if we don't bother to look at the state of S. The configurations (1) and (3) are distinct in physics, even if we don't know the distinction.

All the above is true, even if we don't know S exists. The configurations (1) and (3) are distinct whether or not we have distinct mental representations for the two possibilities.

All the above is true, even if we're in space, and S transmits a new photon off toward the interstellar void in two distinct directions, depending on whether the photon of interest passed it or not. So that we couldn't ever find out whether S had been in Yes or No. The state of S would be embodied in the photon transmitted off to nowhere. The lost photon can be an implied invisible, and the state of S pragmatically undetectable; but the configurations are still distinct.

(The main reason it wouldn't work, is if S were nudged, but S had an original spread in configuration space that was larger than the nudge. Then you couldn't rely on the nudge to separate the amplitude distribution over configuration space into distinct lumps. In reality, all this takes place within a differentiable amplitude distribution over a continuous configuration space.)

Configurations are not belief states. Their distinctness is an objective fact with experimental consequences. The configurations are distinct even if no one knows the state of S; distinct even if no intelligent entity can ever find out. The configurations are distinct so long as at least one particle in the universe anywhere is in a different position. This is experimentally demonstrable.

Why am I emphasizing this? Because back in the dark ages when no one understood quantum physics...

Okay, so imagine that you've got no clue what's really going on, and you try the experiment in Figure 2, and no photons show up at Detector 1. Cool.

You also discover that when you put a block between B and D, or a block between A and C, photons show up at Detector 1 and Detector 2 in equal proportions. But only one at a time - Detector 1 or Detector 2 goes off, not both simultaneously.

So, yes, it does seem to you like you're dealing with a particle - the photon is only in one place at one time, every time you see it (we'll talk about the real reasons for this later).

And yet there's some kind of... mysterious phenomenon... that prevents the photon from showing up in Detector 1. And this mysterious phenomenon depends on the photon being able to go both ways. Even though the photon only shows up in one detector or the other, which shows, you would think, that the photon is only in one place at a time.

Which makes the whole pattern of the experiments seem pretty bizarre! After all, the photon either goes from A to C, or from A to B; one or the other. (Or so you would think, if you were instinctively trying to break reality down into individually real particles.) But when you block off one course or the other, you start getting different experimental results!

It's like the photon wants to be allowed to go both ways, even though (you would think) it only goes one way or the other. And it can tell if you try to block it off, without actually going there - if it'd gone there, it would have run into the block, and not hit any detector at all.

It's as if mere possibilities could have causal effects, in defiance of what the word "real" is usually thought to mean...

But it's a bit early to jump to conclusions like that, when you don't have a complete picture of what goes on inside the experiment.

So it occurs to you to put a sensor between A and C, so you can tell which way the photon really goes on each occasion.

And the mysterious phenomenon goes away.

I mean, now how crazy is that? What kind of paranoia does that inspire in some poor scientist?

Okay, so in the 21st century we realize in order to "know" a photon's history, the particles making up your brain have to be correlated with the photon's history. If having a tiny little sensitive thingy S that correlates to the photon's history, is enough to distinguish the final configurations and prevent the amplitude flows from canceling; then an entire sensor with a digital display, never mind a human brain, will put septillions of particles in different positions and prevent the amplitude flows from canceling.

But if you hadn't worked that out yet...

Then you would ponder the sensor having banished the Mysterious Phenomenon, and think:

The photon doesn't just want to be physically free to go either way. It's not a little wave going along an unblocked pathway, because then just having a physically unblocked pathway would be enough.

No... I'm not allowed to know which way the photon went.

The mysterious phenomenon... doesn't want me looking at it too closely... while it's doing its mysterious thing.

It's not physical possibilities that have an effect on reality... only epistemic possibilities. If I know which way the photon went, it's no longer plausible that it went the other way... which cuts off the mysterious phenomenon as effectively as putting a block between B and D.

I have to not observe which way the photon went, in order for it to always end up at Detector 2. It has to be reasonable that the photon could have gone to either B or C. What I can know is the determining factor, regardless of which physical paths I leave open or closed.

STOP THE PRESSES! MIND IS FUNDAMENTAL AFTER ALL! CONSCIOUS AWARENESS DETERMINES OUR EXPERIMENTAL RESULTS!

You can still read this kind of stuff. In physics textbooks. Even now, when a majority of theoretical physicists know better. Stop the presses. Please, stop the presses.

Hindsight is 20/20; and so it's easy to say that, in hindsight, there were certain clues that this interpretation was not correct.

Like, if you put the sensor between A and C but don't read it, the mysterious phenomenon still goes away, and the photon still sometimes ends up at Detector 1. (Oh, but you could have read it, and possibilities are real now...)

But it doesn't even have to be a sensor, a scientific instrument that you built. A single particle that gets nudged far enough will dispel the interference. A photon radiating off to where you'll never see it again can do the trick. Not much human involvement there. Not a whole lot of conscious awareness.

Maybe before you pull the dualist fire alarm on human brains being physically special, you should provide experimental proof that a rock can't play the same role in dispelling the Mysterious Phenomenon as a human researcher?

But that's hindsight, and it's easy to call the shots in hindsight. Do you really think you could've done better than John von Neumann, if you'd been alive at the time? The point of this kind of retrospective analysis is to ask what kind of fully general clues you could have followed, and whether there are any similar clues you're ignoring now on current mysteries.

Though it is a little embarrassing that, even after the theory of amplitudes and configurations had been worked out - with the theory now giving the definite prediction that any nudged particle would do the trick - early scientists still didn't get it.

But you see... it had been established as Common Wisdom that configurations were possibilities, it was epistemic possibility that mattered, amplitudes were a very strange sort of partial information, and conscious observation made quantumness go away. And that it was best to avoid thinking too hard about the whole business, so long as your experimental predictions came out right.

Sequence: Quantum Physics

Joint Configurations

Where Philosophy Meets Science

Referenced by: Where Philosophy Meets Science • Classical Configuration Spaces • On Being Decoherent • Decoherence as Projection • Entangled Photons • Spooky Action at a Distance: The No-Communication Theorem • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity • For The People Who Are Still Alive

Original with comments: Distinct Configurations

Where Philosophy Meets Science

Eliezer Yudkowsky, 12 April 2008 09:21PM

This post is part of the Quantum Physics Sequence.
Followup to: Distinct Configurations

Looking back on early quantum physics - not for purposes of admonishing the major figures, or to claim that we could have done better if we'd been born into that era; but in order to try and learn a moral, and do better next time - looking back on the dark ages of quantum physics, I say, I would nominate as the "most basic" error...

...not that they tried to reverse course on the last three thousand years of science suggesting that mind was complex within physics rather than fundamental in physics. This is Science, and we do have revolutions here. Every now and then you've got to reverse a trend. The future is always absurd and never unlawful.

I would nominate, as the basic error not to repeat next time, that the early scientists forgot that they themselves were made out of particles.

I mean, I'm sure that most of them knew it in theory.

And yet they didn't notice that putting a sensor to detect a passing electron, or even knowing about the electron's history, was an example of "particles in different places". So they didn't notice that a quantum theory of distinct configurations already explained the experimental result, without any need to invoke consciousness.

In the ancestral environment, humans were often faced with the adaptively relevant task of predicting other humans. For which purpose you thought of your fellow humans as having thoughts, knowing things and feeling things, rather than thinking of them as being made up of particles. In fact, many hunter-gatherer tribes may not even have known that particles existed. It's much more intuitive - it feels simpler - to think about someone "knowing" something, than to think about their brain's particles occupying a different state. It's easier to phrase your explanations in terms of what people know; it feels more natural; it leaps more readily to mind.

Just as, once upon a time, it was easier to imagine Thor throwing lightning bolts, than to imagine Maxwell's Equations - even though Maxwell's Equations can be described by a computer program vastly smaller than the program for an intelligent agent like Thor.

So the ancient physicists found it natural to think, "I know where the photon was... what difference could that make?" Not, "My brain's particles' current state correlates to the photon's history... what difference could that make?"

And, similarly, because it felt easy and intuitive to model reality in terms of people knowing things, and the decomposition of knowing into brain states did not leap so readily to mind, it seemed like a simple theory to say that a configuration could have amplitude only "if you didn't know better".

To turn the dualistic quantum hypothesis into a formal theory - one that could be written out as a computer program, without human scientists deciding when an "observation" occurred - you would have to specify what it meant for an "observer" to "know" something, in terms your computer program could compute.

So is your theory of fundamental physics going to examine all the particles in a human brain, and decide when those particles "know" something, in order to compute the motions of particles? But then how do you compute the motion of the particles in the brain itself? Wouldn't there be a potential infinite recursion?

But so long as the terms of the theory were being processed by human scientists, they just knew when an "observation" had occurred. You said an "observation" occurred whenever it had to occur in order for the experimental predictions to come out right - a subtle form of constant tweaking.

(Remember, the basics of quantum theory were formulated before Alan Turing said anything about Turing machines, and way before the concept of computation was popularly known. The distinction between an effective formal theory, and one that required human interpretation, was not as clear then as now. Easy to pinpoint the problems in hindsight; you shouldn't learn the lesson that problems are usually this obvious in foresight.)

Looking back, it may seem like one meta-lesson to learn from history, is that philosophy really matters in science - it's not just some adjunct of a separate academic field.

After all, the early quantum scientists were doing all the right experiments. It was their interpretation that was off. And the problems of interpretation were not the result of their getting the statistics wrong.

Looking back, it seems like the errors they made, were errors in the kind of thinking that we would describe as, well, "philosophical".

When we look back and ask, "How could the early quantum scientists have done better, even in principle?" it seems that the insights they needed were philosophical ones.

And yet it wasn't professional philosophers who swooped in and solved the problem and cleared up the mystery and made everything normal again. It was, well, physicists.

Arguably, Leibniz was at least as foresightful about quantum physics, as Democritus was once thought to have been foresightful about atoms. But that is hindsight. It's the result of looking at the solution, and thinking back, and saying, "Hey, Leibniz said something like that."

Even where one philosopher gets it right in advance, it's usually science that ends up telling us which philosopher is right - not the prior consensus of the philosophical community.

I think this has something fundamental to say about the nature of philosophy, and the interface between philosophy and science.

It was once said that every science begins as philosophy, but then grows up and leaves the philosophical womb, so that at any given time, "Philosophy" is what we haven't turned into science yet.

I suggest that when we look at the history of quantum physics, and say, "The insights they needed were philosophical insights," what we are really seeing, is that the insight they needed was of a form that is not yet taught in standardized academic classes, and not yet reduced to calculation.

Once upon a time, the notion of the scientific method - updating beliefs based on experimental evidence - was a philosophical notion. But it was not championed by professional philosophers. It was the real-world power of science that showed that scientific epistemology was good epistemology, not a prior consensus of philosophers.

Today, this philosophy of belief-updating is beginning to be reduced to calculation - statistics, Bayesian probability theory.

But back in Galileo's era, it was solely vague verbal arguments that said you should try to produce numerical predictions of experimental results, rather than consulting the Bible or Aristotle.

At the frontier of science, and especially, at the frontier of scientific chaos and scientific confusion, you find problems of thinking that are not taught in academic courses, and that have not been reduced to calculation. And this will seem like a domain of philosophy; it will seem that you must do philosophical thinking in order to sort out the confusion. But when history looks back, I'm afraid, it is usually not a professional philosopher who wins all the marbles - because it takes intimate involvement with the scientific domain in order to do the philosophical thinking. Even if, afterward, it all seems knowable a priori; and even if, afterward, some philosopher out there actually got it a priori; even so, it takes intimate involvement to see it in practice, and experimental results to tell the world which philosopher won.

I suggest that, like ethics, philosophy really is important, but it is only practiced effectively from within a science. Trying to do the philosophy of a frontier science, as a separate academic profession, is as much a mistake as trying to have separate ethicists. You end up with ethicists who speak mainly to other ethicists, and philosophers who speak mainly to other philosophers.

This is not to say that there is no place for professional philosophers in the world. Some problems are so chaotic that there is no established place for them at all in the halls of science. But those "professional philosophers" would be very, very wise to learn every scrap of relevant-seeming science that they can possibly get their hands on. They should not be surprised at the prospect that experiment, and not debate, will finally settle the argument. They should not flinch from running their own experiments, if they can possibly think of any.

That, I think, is the lesson of history.

PS: Speaking of history, tonight is April 12, the anniversary of Yuri's Night - Yuri Gagarin's first flight into space - and there may be a celebration in your area.

Sequence: Quantum Physics

Distinct Configurations

Can You Prove Two Particles Are Identical?

Referenced by: Can You Prove Two Particles Are Identical? • The So-Called Heisenberg Uncertainty Principle • Which Basis Is More Fundamental? • Where Physics Meets Experience • Bell's Theorem: No EPR "Reality" • Thou Art Physics • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Where Philosophy Meets Science

Can You Prove Two Particles Are Identical?

Eliezer Yudkowsky, 14 April 2008 07:06AM

This post is part of the Quantum Physics Sequence.
Followup to: Where Philosophy Meets Science, Joint Configurations

Behold, I present you with two electrons. They have the same mass. They have the same charge. In every way that we've tested them so far, they seem to behave the same way.

But is there any way we can know that the two electrons are really, truly, entirely indistinguishable?

The one who is wise in philosophy but not in physics will snort dismissal, saying, "Of course not. You haven't found an experiment yet that distinguishes these two electrons. But who knows, you might find a new experiment tomorrow that does."

Just because your current model of reality files all observed electrons in the same mental bucket, doesn't mean that tomorrow's physics will do the same. That's mixing up the map with the territory. Right?

It took a while to discover atomic isotopes. Maybe someday we'll discover electron isotopes whose masses are different in the 20th decimal place. In fact, for all we know, the electron has a tiny little tag on it, too small for your current microscopes to see, reading 'This is electron #7,234,982,023,348...' So that you could in principle toss this one electron into a bathtub full of electrons, and then fish it out again later. Maybe there's some way to know in principle, maybe not - but for now, surely, this is one of those things that science just doesn't know.

That's what you would think, if you were wise in philosophy but not in physics.

But what kind of universe could you possibly live in, where a simple experiment can tell you whether it's possible in principle to tell two things apart?

Maybe aliens gave you a tiny little device with two tiny little boxes, and a tiny little light that goes on when you put two identical things into the boxes?

But how do you know that's what the device really does? Maybe the device was just built with measuring instruments that go to the 10th decimal place but not any further.

Imagine that we take this problem to an analytic philosopher named Bob, and Bob says:

"Well, for one thing, you can't even get absolute proof that the two particles actually exist, as opposed to being some kind of hallucination created in you by the Dark Lords of the Matrix. We call it 'the problem of induction'."

Yes, we've heard of the problem of induction. Though the Sun has risen on billions of successive mornings, we can't know with absolute certainty that, tomorrow, the Sun will not transform into a giant chocolate cake. But for the Sun to transform to chocolate cake requires more than an unanticipated discovery in physics. It requires the observed universe to be a lie. Can any experiment give us an equally strong level of assurance that two particles are identical?

"Well, I Am Not A Physicist," says Bob, "but obviously, the answer is no."

Why?

"I already told you why: No matter how many experiments show that two particles are similar, tomorrow you might discover an experiment that distinguishes between them."

Oh, but Bob, now you're just taking your conclusion as a premise. What you said is exactly what we want to know. Is there some achievable state of evidence, some sequence of discoveries, from within which you can legitimately expect never to discover a future experiment that distinguishes between two particles?

"I don't believe my logic is circular. But, since you challenge me, I'll formalize the reasoning.

"Suppose there are particles {P1, P2, ...} and a suite of experimental tests {E1, E2, ...} Each of these experimental tests, according to our best current model of the world, has a causal dependency on aspects {A1, A2...} of the particles P, where an aspect might be something like 'mass' or 'electric charge'.

"Now these experimental tests can establish very reliably - to the limit of our belief that the universe is not outright lying to us - that the depended-on aspects of the particles are similar, up to some limit of measurable precision.

"But we can always imagine an additional aspect A0 that is not depended-on by any of our experimental measures. Perhaps even an epiphenomenal aspect. Some philosophers will argue over whether an epiphenomenal aspect can be truly real, but just because we can't legitimately know about something's existence doesn't mean it's not there. Alternatively, we can always imagine an experimental difference in any quantitative aspect, such as mass, that is too small to detect, but real.

"These extra properties or marginally different properties are conceivable, therefore logically possible. This shows you need additional information, not present in the experiments, to definitely conclude the particles are identical."

That's an interesting argument, Bob, but you say you haven't studied physics.

"No, not really."

Maybe you shouldn't be doing all this philosophical analysis before you've studied physics. Maybe you should beg off the question, and let a philosopher who's studied physics take over.

"Would you care to point out a particular flaw in my logic?"

Oh... not at the moment. We're just saying, You Are Not A Physicist. Maybe you shouldn't be so glib when it comes to saying what physicists can or can't know.

"They can't know two particles are perfectly identical. It is not possible to imagine an experiment that proves two particles are perfectly identical."

Impossible to imagine? You don't know that. You just know you haven't imagined such an experiment yet. But perhaps that simply demonstrates a limit on your imagination, rather than demonstrating a limit on physical possibility. Maybe if you knew a little more physics, you would be able to conceive of such an experiment?

"I'm sorry, this isn't a question of physics, it's a question of epistemology. To believe that all aspects of two particles are perfectly identical, requires a different sort of assurance than any experimental test can provide. Experimental tests only fail to establish a difference; they do not prove identity. What particular physics experiments you can do, is a physics question, and I don't claim to know that. But what experiments can justify believing is an epistemological question, and I am a professional philosopher; I expect to understand that question better than any physicist who hasn't studied formal epistemology."

And of course, Bob is wrong.

Bob isn't being stupid. He'd be right in any classical universe. But we don't live in a classical universe.

Our ability to perform an experiment that tells us positively that two particles are entirely identical, goes right to the heart of what distinguishes the quantum from the classical; the core of what separates the way reality actually works, from anything any pre-20th-century human ever imagined about how reality might work.

If you have a particle P1 and a particle P2, and it's possible in the experiment for both P1 and P2 to end up in either of two possible locations L1 or L2, then the observed distribution of results will depend on whether "P1 at L1, P2 at L2" and "P1 at L2, P2 at L1" is the same configuration, or two distinct configurations. If they're the same configuration, we add up the amplitudes flowing in, then take the squared modulus. If they're different configurations, we keep the amplitudes separate, take the squared moduli separately, then add the resulting probabilities. As (1 + 1)2 != (1^2 + 1^2), it's not hard to distinguish the experimental results after a few trials.

(Yes, half-integer spin changes this picture slightly. Which I'm not going into in this series of blog posts. If all epistemological confusions are resolved, half-integer spin is a difficulty of mere mathematics, so the issue doesn't belong here. Half-integer spin doesn't change the experimental testability of particle equivalences, or alter the fact that particles have no individual identities.)

And the flaw in Bob's logic? It was a fundamental assumption that Bob couldn't even see, because he had no alternative concept for contrast. Bob talked about particles P1 and P2 as if they were individually real and independently real. This turns out to assume that which is to be proven. In our universe, the individually and fundamentally real entities are configurations of multiple particles, and the amplitude flows between them. Bob failed to imagine the sequence of experimental results which established to physicists that this was, in fact, how reality worked.

Bob failed to imagine the evidence which falsified his basic and invisibly assumed ontology - the discoveries that changed the whole nature of the game; from a world that was the sum of individual particles, to a world that was the sum of amplitude flows between multi-particle configurations.

And so Bob's careful philosophical reasoning ended up around as useful as Kant's conclusion that space, by its very nature, was flat. Turned out, Kant was just reproducing an invisible assumption built into how his parietal cortex was modeling space. Kant's imaginings were evidence only about his imagination - grist for cognitive science, not physics.

Be careful not to underestimate, through benefit of hindsight, how surprising it would seem, a priori, that you could perfectly identify two particles through experiment. Be careful not to underestimate how entirely and perfectly reasonable Bob's analysis would have seemed, if you didn't have quantum assumptions to contrast to classical ones.

Experiments tell us things about the nature of reality which you just plain wouldn't expect from a priori reasoning. Experiments falsify assumptions we can't even see. Experiments tell us how to do things that seem logically impossible. Experiments deliver surprises from blind spots we don't even know exist.

Bear this in mind, the next time you're wondering whether mere empirical science might have something totally unexpected to say about some impossible-seeming philosophical question.

Sequence: Quantum Physics

Where Philosophy Meets Science

Classical Configuration Spaces

Referenced by: No Individual Particles • Entangled Photons • Bell's Theorem: No EPR "Reality" • Spooky Action at a Distance: The No-Communication Theorem • Timeless Identity • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Can You Prove Two Particles Are Identical?

Classical Configuration Spaces

Eliezer Yudkowsky, 15 April 2008 08:40AM

This post is part of the Quantum Physics Sequence.
Previously in series: Distinct Configurations

Once upon a time, there was a student who went to a math lecture. When the lecture was over, he approached one of the other students, and said, "I couldn't follow that at all. The professor was talking about rotating 8-dimensional objects! How am I supposed to visualize something rotating in 8 dimensions?"
"Easy," replied the other student, "you visualize it rotating in N dimensions, then let N go to 8."
-- old joke

Quantum configuration space isn't quite like classical configuration space. But in this case, considering that 8 dimensions is peanuts in quantum physics, even I concede that you ought to start with classical configuration space first.

(I apologize for the homemade diagrams, but this blog post already used up all available time...)

In classical physics, a configuration space is a way of visualizing the state of an entire system as a single point in a higher-dimensional space.

Suppose that a system is composed of two particles, A and B, each on the same 1-dimensional line. (We'll call the two directions on the line, "forward" and "back".)

Then we can view the state of the complete system A+B as a single point in 2-dimensional space.

If you look at state 1, for example, it describes a state of the system where B is far forward and A is far back. We can view state 1 as being embodied either in two 1-dimensional positions (the representation on the right), or view it as one 2-dimensional position (the representation on the left).

To help grasp the idea of viewing a system as a point, this alternate graph shows A and B on the same line.

When A and B are far apart, they both move toward each other. However, B moves slower than A. Also, B wants to be closer to A than A wants to be close to B, so as B gets too close, A runs away...

(At least that's what I had in mind while trying to draw the system evolution.)

The system evolution can be shown as a discrete series of states: Time=1, Time=2, Time=3... But in configuration space, I can draw the system evolution as a smooth trajectory.

If I had the time (to learn to use the appropriate software), I'd be drawing neat-o 3D diagrams at this point. Like the diagram at right, only with, like, actual graphics.

You may have previously heard the phrase, "time is the 4th dimension". But the diagram at right shows the evolution over time of a 1-dimensional universe with two particles. So time is the third dimension, the first dimension being the position of particle A, and the second dimension being the position of particle B.

All these neat pictures are simplified, even relative to classical physics.

In classical physics, each particle has a 3-dimensional position and a 3-dimensional velocity. So to specify the complete state of a 7-particle system would require 42 real numbers, which you could view as one point in 42-dimensional space.

Hence the joke.

Configuration spaces get very high-dimensional, very fast. That's why we're sticking with 2 particles in a 1-dimensional universe. Anything more than that, I can't draw on paper - you've just got to be able to visualize it in N dimensions.

So far as classical physics is concerned, it's a matter of taste whether you would want to imagine a system state as a point in configuration space, or imagine the individual particles. Mathematically, the two systems are isomorphic - in classical physics, that is. So what's the benefit of imagining a classical configuration space?

Well, for one thing, it makes it possible to visualize joint probability distributions.

The grey area in the diagram represents a probability distribution over potential states of the A+B system.

If this is my state of knowledge, I think the system is somewhere in the region represented by the grey area. I believe that if I knew the actual states of both A and B, and visualized the A+B system as a point, the point would be inside the grey.

Three sample possibilities within the probability distribution are shown, and the corresponding systems.

And really the probability distribution should be lighter or darker, corresponding to volumes of decreased or increased probability density. It's a probability distribution, not a possibility distribution. I didn't make an effort to represent this in the diagram - I probably should have - but you can imagine it if you like. Or pretend that the slightly darker region in the upper left is a volume of increased probability density, rather than a fluke of penciling.

Once you've hit on the idea of using a bounded volume in configuration space to represent possibility, or a cloud with lighter and darker parts to represent probability, you can ask how your knowledge about a system develops over time. If you know how each system state (point in configuration space) develops dynamically into a future system state, and you draw a little cloud representing your current probability distribution, you can project that cloud into the future.

Here I start out with uncertainty represented by the squarish grey box in the first configuration space, at bottom right.

All the points in the first grey box, correspond to system states, that dynamically develop over time, into new system states, corresponding to points in the grey rectangle in the second configuration space at middle right.

Then, my little rectangle of uncertainty develops over time into a wiggly figure, three major possibility-nodes connected by thin strings of probability density, as shown at top right.

In this figure I also tried to represent the idea of conserved probability volume - the same total volume of possibility, with points evolving to other points with the same local density, at each successive time. This is Liouville's Theorem, which is the key to the Second Law of Thermodynamics, as I have previously described.

Neat little compact volumes of uncertainty develop over time, under the laws of physics, into big wiggly volumes of uncertainty. If you want to describe the new volumes of uncertainty compactly, in less than a gazillion gigabytes, you've got to draw larger boundaries around them. Once you draw the new larger boundary, your uncertainty never shrinks, because probability flow is conservative. So entropy always increases. That's the second law of thermodynamics.

Just figured I'd mention that, as long as I was drawing diagrams... you can see why this "visualize a configuration space" trick is useful, even in classical physics.

Another idea that's easier to visualize in configuration space is the idea of conditional independence between two probabilistic variables.

Conditional independence happens when the joint probability distribution is the product of the individual probability distributions:

P(A,B) = P(A) x P(B)

The vast majority of possible probability distributions are not conditionally independent, the same way that the vast majority of shapes are not rectangular. Actually, this is oversimplifying: It's not enough for the volume of possibilities to be rectangular. The probability density has to factorize into a product of probability densities on each side.

The vast majority of shapes are not rectangles, the vast majority of color patterns are not plaid. It's conditional independence, not conditional dependence, that is the unusual special case.

(I bet when you woke up this morning, you didn't think that today you would be visualizing plaid patterns in N dimensions.)

In the figure reprised here at right, my little cloud of uncertainty is not rectangular.

Hence, my uncertainty about A and my uncertainty about B are not independent.

If you tell me A is far forward, I will conclude B is far back. If you tell me A is in the middle of its 1-dimensional universe, I will conclude that B is likewise in the middle.

If I tell you A is far back, what do you conclude about B?

Aaaand that's classical configuration space, folks. It doesn't add anything mathematically to classical physics, but it can help human beings visualize system dynamics and probability densities. It seemed worth filtering into a separate post, because configuration space is a modular concept, useful for other ideas.

Quantum physics inherently takes place in a configuration space. You can't take it out. Tomorrow, we'll see why.

Sequence: Quantum Physics

Can You Prove Two Particles Are Identical?

The Quantum Arena

Referenced by: The Quantum Arena • Decoherence • Decoherence is Pointless • Relative Configuration Space • Timeless Physics • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Classical Configuration Spaces

The Quantum Arena

Eliezer Yudkowsky, 15 April 2008 07:00PM

This post is part of the Quantum Physics Sequence.
Previously in series: Classical Configuration Spaces

Yesterday, we looked at configuration spaces in classical physics. In classical physics, configuration spaces are a useful, but optional, point of view.

Today we look at quantum physics, which inherently takes place inside a configuration space, and cannot be taken out.

For a start, as you might guess, in quantum physics we deal with distributions of complex amplitudes, rather than probability distributions made up of positive real numbers. At left, I've used up 3 dimensions drawing a complex distribution over the position of one particle, A.

You may recall that yesterday, 3 dimensions let us display the position of two 1-dimensional particles plus the system evolution over time. Today, it's taking us 3 dimensions just to visualize an amplitude distribution over the position of one 1-dimensional particle at a single moment in time. Which is why we did classical configuration spaces first.

To clarify the meaning of the above diagram, the left-to-right direction is the position of A.

The up-and-down direction, and the invisible third dimension that leaps out of the paper, are devoted to the complex amplitudes. Since a complex amplitude has a real and imaginary part, they use up 2 of our 3 dimensions.

Richard Feynman said to just imagine the complex amplitudes as little 2-dimensional arrows. This is as good a representation as any; little 2D arrows behave just the same way complex numbers do. (You add little arrows by starting at the origin, and moving along each arrow in sequence. You multiply little arrows by adding the angles and multiplying the lengths. This is isomorphic to the complex field.) So we can think of each position of the A particle as having a little arrow associated to it.

As you can see, the position of A bulges in two places - a big bulge to the left, and a smaller bulge at right. Way up at the level of classical observation, there would be a large probability (integrating over the squared modulus) of finding A somewhere to the left, and a smaller probability of finding it at the small bulge to the right.

Drawing a neat little graph of the A+B system would involve having a complex amplitude for each joint position of the A and B particles, which you could visualize as a hypersurface in 4 dimensions. I'd draw it for you, but I left my 4-dimensional pencil in the pocket of the 3rd leg of my other pants.

You may recall from yesterday that a plaid rectangular probability distribution factorizes into the product of two independent probability distributions.

This kind of independence-structure is one of several keys to recovering the illusion of individual particles from quantum amplitude distributions. If the amplitude distribution roughly factorizes, has subsystems A and B with Amplitude(X,Y) ~ Amplitude(X) * Amplitude(Y), then X and Y will seem to evolve roughly independently of each other.

But maintaining the illusion of individuality is harder in quantum configuration spaces, because of the identity of particles. This identity cuts down the size of a 2-particle configuration space by 1/2, cuts down the size of a 3-particle configuration space by 1/6, and so on. Here, the diminished configuration space is shown for the 2-particle case:

The quantum configuration space is over joint possibilities like "a particle here, a particle there", not "this particle here, that particle there". What would have been a neat little plaid pattern gets folded in on itself.

You might think that you could recover the structure by figuring out which particle is "really which" - i.e. if you see a "particle far forward, particle in middle", you can guess that the first particle is A, and the second particle is B, because only A can be far forward; B just stays in the middle. (This configuration would lie in at the top of the original plaid pattern, the part that got folded over).

The problem with this is the little triangular region, where the folded plaid intersects itself. In this region, the folded-over amplitude distribution gets superposed, added together. Which makes an experimental difference, because the squared modulus of the sum is not the sum of the squared moduli.

In that little triangular region of quantum configuration space, there is simply no fact of the matter as to "which particle is which". Actually, there never was any such fact; but there was an illusion of individuality, which in this case has broken down.

But even that isn't the ultimate reason why you can't take quantum physics out of configuration space.

In classical configuration spaces, you can take a single point in the configuration space, and the single point describes the entire state of a classical system. So you can take a single point in classical configuration space, and ask how the corresponding system develops over time. You can take a single point in classical configuration space, and ask, "Where does this one point go?"

The development over time of quantum systems depends on things like the second derivative of the amplitude distribution. Our laws of physics describe how amplitude distributions develop into new amplitude distributions. They do not describe, even in principle, how one configuration develops into another configuration.

(I pause to observe that physics books make it way, way, way too hard to figure out this extremely important fact. You'd think they'd tell you up front, "Hey, the evolution of a quantum system depends on stuff like the second derivative of the amplitude distribution, so you can't possibly break it down into the evolution of individual configurations." When I first saw the Schrödinger Equation it confused the hell out of me, because I thought the equation was supposed to apply to single configurations.)

If I've understood the laws of physics correctly, quantum mechanics still has an extremely important property of locality: You can determine the instantaneous change in the amplitude of a single configuration using only the infinitesimal neighborhood. If you forget that the space is continuous and think of it as a mesh of computer processors, each processor would only have to talk to its immedatien neighbors to figure out what to do next. You do have to talk to your neighbors - but only your next-door neighbors, no telephone calls across town. (Technical term: "Markov neighborhood.")

Conway's Game of Life has the discrete version of this property; the future state of each cell depends only on its own state and the state of neighboring cells.

The second derivative - Laplacian, actually - is not a point property. But it is a local property, where knowing the immediate neighborhood tells you everything, regardless of what the rest of the distribution looks like. Potential energy, which also plays a role in the evolution of the amplitude, can be computed at a single positional configuration (if I've understood correctly).

There are mathematical transformations physicists use for their convenience, like viewing the system as an amplitude distribution over momenta rather than positions, which throw away this neighborhood structure (e.g. by making potential energy a non-locally-computable property). Well, mathematical convenience is a fine thing. But I strongly suspect that the physically real wavefunction has local dynamics. This kind of locality seems like an extremely important property, a candidate for something hardwired into the nature of reality and the structure of causation. Imposing locality is part of the jump from Newtonian mechanics to Special Relativity.

The temporal behavior of each amplitude in configuration space depends only on the amplitude at neighboring points. But you cannot figure out what happens to the amplitude of a point in quantum configuration space, by looking only at that one point. The future amplitude depends on the present second derivative of the amplitude distribution.

So you can't say, as you can in classical physics, "If I had infinite knowledge about the system, all the particles would be in one definite position, and then I could figure out the exact future state of the system."

If you had a point mass of amplitude, an infinitely sharp spike in the quantum arena, the amplitude distribution would not be twice differentiable and the future evolution of the system would be undefined. The known laws of physics would crumple up like tinfoil. Individual configurations don't have quantum dynamics; amplitude distributions do.

A point mass of amplitude, concentrated into a single exact position in configuration space, does not correspond to a precisely known state of the universe. It is physical nonsense.

It's like asking, in Conway's Game of Life: "What is the future state of this one cell, regardless of the cells around it?" The immediate future of the cell depends on its immediate neighbors; its distant future may depend on distant neighbors.

Imagine trying to say, in a classical universe, "Well, we've got this probability distribution over this classical configuration space... but to find out where the system evolves, where the probability flows from each point, we've got to twice differentiate the probability distribution to figure out the dynamics."

In classical physics, the position of a particle is a separate fact from its momentum. You can know exactly where a particle is, but not know exactly how fast it is moving.

In Conway's Game of Life, the velocity of a glider is not a separate, additional fact about the board. Cells are only "alive" or "dead", and the apparent motion of a glider arises from a configuration that repeats itself as the cell rules are applied. If you know the life/death state of all the cells in a glider, you know the glider's velocity; they are not separate facts.

In quantum physics, there's an amplitude distribution over a configuration space of particle positions. Quantum dynamics specify how that amplitude distribution evolves over time. Maybe you start with a blob of amplitude centered over position X, and then a time T later, the amplitude distribution has evolved to have a similarly-shaped blob of amplitude at position X+D. Way up at the level of human researchers, this looks like a particle with velocity D/T. But at the quantum level this behavior arises purely out of the amplitude distribution over positions, and the laws for how amplitude distributions evolve over time.

In quantum physics, if you know the exact current amplitude distribution over particle positions, you know the exact future behavior of the amplitude distribution. Ergo, you know how blobs of amplitude appear to propagate through the configuration space. Ergo, you know how fast the "particles" are "moving". Full knowledge of the amplitude distribution over positions implies full knowledge of momenta.

Imagine trying to say, in a classical universe, "I twice differentiate the probability distribution over these particles' positions, to physically determine how fast they're going. So if I learned new information about where the particles were, they might end up moving at different speeds. If I got very precise information about where the particles were, this would physically cause the particles to start moving very fast, because the second derivative of probability would be very large." Doesn't sound all that sensible, does it? Don't try to interpret this nonsense - it's not even analogously correct. We'll look at the horribly misnamed "Heisenberg Uncertainty Principle" later.

But that's why you can't take quantum physics out of configuration space. Individual configurations don't have physics. Amplitude distributions have physics.

(Though you can regard the entire state of a quantum system - the whole amplitude distribution - as a single point in a space of infinite dimensionality: "Hilbert space." But this is just a convenience of visualization. You imagine it in N dimensions, then let N go to infinity.)

Sequence: Quantum Physics

Classical Configuration Spaces

Feynman Paths

Referenced by: Feynman Paths • No Individual Particles • The So-Called Heisenberg Uncertainty Principle • Which Basis Is More Fundamental? • The Born Probabilities • Entangled Photons • Timeless Physics • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: The Quantum Arena

Feynman Paths

Eliezer Yudkowsky, 17 April 2008 06:32AM

This post is part of the Quantum Physics Sequence.
Previously in series: The Quantum Arena

At this point I would like to introduce another key idea in quantum mechanics. Unfortunately, this idea was introduced so well in chapter 2 of QED: The Strange Theory of Light and Matter by Richard Feynman, that my mind goes blank when trying to imagine how to introduce it any other way. As a compromise with just stealing his entire book, I stole one diagram - a diagram of how a mirror really works.

In elementary school, you learn that the angle of incidence equals the angle of reflection. But actually, saith Feynman, each part of the mirror reflects at all angles.

So why is it that, way up at the human level, the mirror seems to reflect with the angle of incidence equal to the angle of reflection?

Because in quantum mechanics, amplitude that flows to identical configurations (particles of the same species in the same places) is added together, regardless of how the amplitude got there.

To find the amplitude for a photon to go from S to P, you've got to add up the amplitudes for all the different ways the photon could get there - by bouncing off the mirror at A, bouncing off the mirror at B...

The rule of the Feynman "path integral" is that each of the paths from S to P contributes an amplitude of constant magnitude but varying phase, and the phase varies with the total time along the path. It's as if the photon is a tiny spinning clock - the hand of the clock stays the same length, but it turns around at a constant rate for each unit of time.

Feynman graphs the time for the photon to go from S to P via A, B, C, ... Observe: the total time changes less between "the path via F" and "the path via G", then the total time changes between "the path via A" and "the path via B". So the phase of the complex amplitude changes less, too.

And when you add up all the ways the photon can go from S to P, you find that most of the amplitude comes from the middle part of the mirror - the contributions from other parts of the mirror tend to mostly cancel each other out, as shown at the bottom of Feynman's figure.

There is no answer to the question "Which part of the mirror did the photon really come from?" Amplitude is flowing from all of these configurations. But if we were to ignore all the parts of the mirror except the middle, we would calculate essentially the same amount of total amplitude.

This means that a photon, which can get from S to P by striking any part of the mirror, will behave pretty much as if only a tiny part of the mirror exists - the part where the photon's angle of incidence equals the angle of reflection.

Unless you start playing clever tricks using your knowledge of quantum physics.

For example, you can scrape away parts of the mirror at regular intervals, deleting some little arrows and leaving others. Keep A and its little arrow; scrape away B so that it has no little arrow (at least no little arrow going to P). Then a distant part of the mirror can contribute amplitudes that add up with each other to a big final amplitude, because you've removed the amplitudes that were out of phase.

In which case you can make a mirror that reflects with the angle of incidence not equal to the angle of reflection. It's called a diffraction grating. But it reflects different wavelengths of light at different angles, so a diffraction grating is not quite a "mirror" in the sense you might imagine; it produces little rainbows of color, like a droplet of oil on the surface of water.

How fast does the little arrow rotate? As fast as the photon's wavelength - that's what a photon's wavelength is. The wavelength of yellow light is ~570 nanometers: If yellow light travels an extra 570 nanometers, its little arrow will turn all the way around and end up back where it started.

So either Feynman's picture is of a very tiny mirror, or he is talking about some very big photons, when you look at how fast the little arrows seem to be rotating. Relative to the wavelength of visible light, a human being is a lot bigger than the level at which you can see quantum effects.

You'll recall that the first key to recovering the classical hallucination from the reality of quantum physics, was the possibility of approximate independence in the amplitude distribution. (Where the distribution roughly factorizes, it can look like a subsystem of particles is evolving on its own, without being entangled with every other particle in the universe.)

The second key to re-deriving the classical hallucination, is the kind of behavior that we see in this mirror. Most of the possible paths cancel each other out, and only a small group of neighboring paths add up. Most of the amplitude comes from a small neighborhood of histories - the sort of history where, for example, the photon's angle of incidence is equal to its angle of reflection. And so too with many other things you are pleased to regard as "normal".

My first posts on QM showed amplitude flowing in crude chunks from discrete situation to discrete situation. In real life there are continuous amplitude flows between continuous configurations, like we saw with Feynman's mirror. But by the time you climb all the way up from a few hundred nanometers to the size scale of human beings, most of the amplitude contributions have canceled out except for a narrow neighborhood around one path through history.

Mind you, this is not the reason why a photon only seems to be in one place at a time. That's a different story, which we won't get to today.

The more massive things are - actually the more energetic they are, mass being a form of energy - the faster the little arrows rotate. Shorter wavelengths of light having more energy is a special case of this. Compound objects, like a neutron made of three quarks, can be treated as having a collective amplitude that is the multiplicative product of the component amplitudes - at least to the extent that the amplitude distribution factorizes, so that you can treat the neutron as an individual.

Thus the relation between energy and wavelength holds for more than photons and electrons; atoms, molecules, and human beings can be regarded as having a wavelength.

But by the time you move up to a human being - or even a single biological cell - the mass-energy is really, really large relative to a yellow photon. So the clock is rotating really, really fast. The wavelength is really, really short. Which means that the neighborhood of paths where things don't cancel out is really, really narrow.

By and large, a human experiences what seems like a single path through configuration space - the classical hallucination.

This is not how Schrödinger's Cat works, but it is how a regular cat works.

Just remember that this business of single paths through time is not fundamentally true. It's merely a good approximation for modeling a sofa. The classical hallucination breaks down completely by the time you get to the atomic level. It can't handle quantum computers at all. It would fail you even if you wanted a sufficiently precise prediction of a brick. A billiard ball taking a single path through time is not how the universe really, really works - it is just what human beings have evolved to easily visualize, for the sake of throwing rocks.

(PS: I'm given to understand that the Feynman path integral may be more fundamental than the Schrödinger equation: that is, you can derive Schrödinger from Feynman. But as far as I can tell from examining the equations, Feynman is still differentiating the amplitude distribution, and so reality doesn't yet break down into point amplitude flows between point configurations. Some physicist please correct me if I'm wrong about this, because it is a matter on which I am quite curious.)

Sequence: Quantum Physics

The Quantum Arena

No Individual Particles

Referenced by: No Individual Particles • Decoherence • The So-Called Heisenberg Uncertainty Principle • On Being Decoherent • Spooky Action at a Distance: The No-Communication Theorem • Many Worlds, One Best Guess • Timeless Identity • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Feynman Paths

No Individual Particles

Eliezer Yudkowsky, 18 April 2008 04:40AM

This post is part of the Quantum Physics Sequence.
Followup to: Can You Prove Two Particles Are Identical?, Feynman Paths

Even babies think that objects have individual identities. If you show an infant a ball rolling behind a screen, and then a moment later, two balls roll out, the infant looks longer at the expectation-violating event. Long before we're old enough to talk, we have a parietal cortex that does spatial modeling: that models individual animals running or rocks flying through 3D space.

And this is just not the way the universe works. The difference is experimentally knowable, and known. Grasping this fact, being able to see it at a glance, is one of the fundamental bridges to cross in understanding quantum mechanics.

If you shouldn't start off by talking to your students about wave/particle duality, where should a quantum explanation start? I would suggest taking, as your first goal in teaching, explaining how quantum physics implies that a simple experimental test can show that two electrons are entirely indistinguishable - not just indistinguishable according to known measurements of mass and electrical charge.

To grasp on a gut level how this is possible, it is necessary to move from thinking in billiard balls to thinking in configuration spaces; and then you have truly entered into the true and quantum realm.

In previous posts such as Joint Configurations and The Quantum Arena, we've seen that the physics of our universe takes place in a multi-particle configuration space.

The illusion of individual particles arises from approximate factorizability of a multi-particle distribution, as shown at left for a classical configuration space.

If the probability distribution over this 2D configuration space of two classical 1D particles, looks like a rectangular plaid pattern, then it will factorize into a distribution over A times a distribution over B.

In classical physics, the particles A and B are the fundamental things, and the configuration space is just an isomorphic way of looking at them.

In quantum physics, the configuration space is the fundamental thing, and you get the appearance of an individual particle when the amplitude distribution factorizes enough to let you look at a subspace of the configuration space, and see a factor of the amplitude distribution - a factor that might look something like this:

This isn't an amplitude distribution, mind you. It's a factor in an amplitude distribution, which you'd have to multiply by the subspace for all the other particles in the universe, to approximate the physically real amplitude distribution.

Most mathematically possible amplitude distributions won't factor this way. Quantum entanglement is not some extra, special, additional bond between two particles. "Quantum entanglement" is the general case. The special and unusual case is quantum independence.

Reluctant tourists in a quantum universe talk about the bizarre phenomenon of quantum entanglement. Natives of a quantum universe talk about the special case of quantum independence. Try to think like a native, because you are one.

I've previously described a configuration as a mathematical object whose identity is "A photon here, a photon there; an electron here, an electron there." But this is not quite correct. Whenever you see a real-world electron, caught in a little electron trap or whatever, you are looking at a blob of amplitude, not a point mass. In fact, what you're looking at is a blob of amplitude-factor in a subspace of a global distribution that happens to factorize.

Clearly, then, an individual point in the configuration space does not have an identity of "blob of amplitude-factor here, blob of amplitude-factor there"; so it doesn't make sense to say that a configuration has the identity "A photon here, a photon there."

But what is an individual point in the configuration space, then?

Well, it's physics, and physics is math, and you've got to come to terms with thinking in pure mathematical objects. A single point in quantum configuration space, is the product of multiple point positions per quantum field; multiple point positions in the electron field, in the photon field, in the quark field, etc.

When you actually see an electron trapped in a little electron trap, what's really going on, is that the cloud of amplitude distribution that includes you and your observed universe, can at least roughly factorize into a subspace that corresponds to that little electron, and a subspace that corresponds to everything else in the universe. So that the physically real amplitude distribution is roughly the product of a little blob of amplitude-factor in the subspace for that electron, and the amplitude-factor for everything else in the universe. Got it?

One commenter reports attaining enlightenment on reading in Wikipedia:

'From the point of view of quantum field theory, particles are identical if and only if they are excitations of the same underlying quantum field. Thus, the question "why are all electrons identical?" arises from mistakenly regarding individual electrons as fundamental objects, when in fact it is only the electron field that is fundamental.'

Okay, but that doesn't make the basic jump into a quantum configuration space that is inherently over multiple particles. It just sounds like you're talking about individual disturbances in the aether, or something. As I understand it, an electron isn't an excitation of a quantum electron field, like a wave in the aether; the electron is a blob of amplitude-factor in a subspace of a configuration space whose points correspond to multiple point positions in quantum fields, etc.

The difficult jump from classical to quantum is not thinking of an electron as an excitation of a field. Then you could just think of a universe made up of "Excitation A in electron field over here" + "Excitation B in electron field over there" + etc. You could factorize the universe into individual excitations of a field. Your parietal cortex would have no trouble with that one - it doesn't care whether you call the little billiard balls "excitations of an electron field" so long as they still behave like little billiard balls.

The difficult jump is thinking of a configuration space that is the product of many positions in many fields, without individual identities for the positions. A configuration space whose points are "a position here in this field, a position there in this field, a position here in that field, and a position there in that field". Not, "A positioned here in this field, B positioned there in this field, C positioned here in that field" etc.

You have to reduce the appearance of individual particles to a regularity in something that is different from the appearance of particles, something that is not itself a little billiard ball.

Oh, sure, thinking of photons as individual objects will seem to work out, as long as the amplitude distribution happens t factorize. But what happens when you've got your "individual" photon A and your "individual" photon B, and you're in a situation where, a la Feynman paths, it's possible for photon A to end up in position 1 and photon B to end up in position 2, or for A to end up in 2 and B to end up in 1? Then the illusion of classicality breaks down, because the amplitude flows overlap:

In that triangular region where the distribution overlaps itself, no fact exists as to which particle is which, even in principle - and in the real world, we often get a lot more overlap than that.

I mean, imagine that I take a balloon full of photons, and shake it up.

Amplitude's gonna go all over the place. If you label all the original apparent-photons, there's gonna be Feynman paths for photons A, B, C ending up at positions 1, 2, 3 via a zillion different paths and permutations.

The amplitude-factor that corresponds to the "balloon full of photons" subspace, which contains bulges of amplitude-subfactor at various different locations in the photon field, will undergo a continuously branching evolution that involves each of the original bulges ending up in many different places by all sorts of paths, and the final configuration will have amplitude contributed from many different permutations.

It's not that you don't know which photon went where. It's that no fact of the matter exists. The illusion of individuality, the classical hallucination, has simply broken down.

And the same would hold true of a balloon full of quarks or a balloon full of electrons. Or even a balloon full of helium. Helium atoms can end up in the same places, via different permutations, and have their amplitudes add just like photons.

Don't be tempted to look at the balloon, and think, "Well, helium atom A could have gone to 1, or it could have gone to 2; and helium atom B could have gone to 1 or 2; quantum physics says the atoms both sort of split, and each went both ways; and now the final helium atoms at 1 and 2 are a mixture of the identities of A and B." Don't torture your poor parietal cortex so. It wasn't built for such usage.

Just stop thinking in terms of little billiard balls, with or without confused identities. Start thinking in terms of amplitude flows in configuration space. That's all there ever is.

And then it will seem completely intuitive that a simple experiment can tell you whether two blobs of amplitude-factor are over the same quantum field.

Just perform any experiment where the two blobs end up in the same positions, via different permutations, and see if the amplitudes add.

Sequence: Quantum Physics

Feynman Paths

Identity Isn't In Specific Atoms

Referenced by: Identity Isn't In Specific Atoms • Decoherence • Decoherent Essences • Bell's Theorem: No EPR "Reality" • Mach's Principle: Anti-Epiphenomenal Physics • Relative Configuration Space • Timeless Identity • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: No Individual Particles

Identity Isn't In Specific Atoms

Eliezer Yudkowsky, 19 April 2008 04:55AM

This post is part of the Quantum Physics Sequence.
Continuation of: No Individual Particles
Followup to: The Generalized Anti-Zombie Principle

Suppose I take two atoms of helium-4 in a balloon, and swap their locations via teleportation. I don't move them through the intervening space; I just click my fingers and cause them to swap places. Afterward, the balloon looks just the same, but two of the helium atoms have exchanged positions.

Now, did that scenario seem to make sense? Can you imagine it happening?

If you looked at that and said, "The operation of swapping two helium-4 atoms produces an identical configuration - not a similar configuration, an identical configuration, the same mathematical object - and particles have no individual identities per se - so what you just said is physical nonsense," then you're starting to get quantum mechanics.

If you furthermore had any thoughts about a particular "helium atom" being a factor in a subspace of an amplitude distribution that happens to factorize that way, so that it makes no sense to talk about swapping two identical multiplicative factors, when only the combined amplitude distribution is real, then you're seriously starting to get quantum mechanics.

If you thought about two similar billiard balls changing places inside a balloon, but nobody on the outside being able to notice a difference, then... oh, hell, I don't know, go back to the beginning of the series and try rereading the whole thing over the course of one day. If that still doesn't work, read an actual book on quantum mechanics. Feynman's QED is a great place to start - though not a good place to finish, and it's not written from a pure realist perspective.

But if you did "get" quantum physics, then, as promised, we have now come to the connection between the truth of quantum mechanics, the lies of human intuitions, and the Generalized Anti-Zombie Principle.

Stirling Westrup previously commented, on the GAZP post:

I found the previous articles on Zombies somewhat tedious... Still, now I'm glad I read through it all as I can see why you were so careful to lay down the foundations you did.

The question of what changes one can make to the brain while maintaining 'identity' has been been discussed many times on the Extropians list, and seldom with any sort of constructive results.

Today's article has already far exceeded the signal to noise ratio of any other discussion on the same topic that I've ever seen...

The Extropians email list that Westrup refers to, is the oldest online gathering place of transhumanists. It is where I made my debut as a writer, and it is where the cofounders of the Singularity Institute met. Though the list is not what it once was...

There are certain topics, on the Extropians list, that have been discussed over and over again, for years and years, without making any progress. Just the same arguments and counterarguments, over and over again.

The worst of those infinite loops concerns the question of personal identity. For example, if you build an exact physical replica of a human, using different atoms, but atoms of the same kind in the same places, is it the same person or just a copy?

This question has flared up at least once a year, always with the same arguments and counterarguments, every year since I joined the Extropians mailing list in 1996. And I expect the Personal Identity Wars started well before then.

I did try remarking, "Quantum mechanics says there isn't any such thing as a 'different particle of the same kind', so wherever your personal identity is, it sure isn't in particular atoms, because there isn't any such thing as a 'particular atom'."

It didn't work, of course. I didn't really expect it to. Without a long extended explanation, a remark like that doesn't actually mean anything.

The concept of reality as a sum of independent individual billiard balls, seems to be built into the human parietal cortex - the parietal cortex being the part of our brain that does spatial modeling: navigating rooms, grasping objects, throwing rocks.

Even very young children, infants, look longer at a scene that violates expectations - for example, a scene where a ball rolls behind a screen, and then two balls roll out.

People try to think of a person, an identity, an awareness, as though it's an awareness-ball located inside someone's skull. Even nonsophisticated materialists tend to think that, since the consciousness ball is made up of lots of little billiard balls called "atoms", if you swap the atoms, why, you must have swapped the consciousness.

Now even without knowing any quantum physics - even in a purely classical universe - it is possible to refute this idea by applying the Generalized Anti-Zombie Principle. There are many possible formulations of the GAZP, but one of the simpler ones says that, if alleged gigantic changes are occurring in your consciousness, you really ought to notice something happening, and be able to say so.

The equivalent of the Zombie World, for questions of identity/continuity, is the Soul Swap World. The allegation is that the Soul Swap World is microphysically identical to our own; but every five minutes, each thread of consciousness jumps to a random new brain, without the brains changing in any third-party experimentally detectable way. One second you're yourself, the next second you're Britney Spears. And neither of you say that you've noticed anything happening - by hypothesis, since you're microphysically identical down to the motion of your lips.

(Let me know if the Soul Swap World has been previously invented in philosophy, and has a standard name - so far as I presently know, this is my own idea.)

We can proceed to demolish the Soul Swap World by an argument exactly analogous to the one that demolished the Zombie World: Whatever-it-is which makes me feel that I have a consciousness that continues through time, that whatever-it-is was physically potent enough to make me type this sentence. Should I try to make the phrase "consciousness continuing through time" refer to something that has nothing to do with the cause of my typing those selfsame words, I will have problems with the meaning of my arguments, not just their plausibility.

Whatever it is that makes me say, aloud, that I have a personal identity, a causally closed world physically identical to our own, has captured that source - if there is any source at all.

And we can proceed, again by an exactly analogous argument, to a Generalized Anti-Swapping Principle: Flicking a disconnected light switch shouldn't switch your personal identity, even though the motion of the switch has an in-principle detectable gravitational effect on your brain, because the switch flick can't disturb the true cause of your talking about "the experience of subjective continuity".

So even in a classical universe, if you snap your fingers and swap an atom in the brain for a physically similar atom outside; and the brain is not disturbed, or not disturbed any more than the level of thermal noise; then whatever causes the experience of subjective continuity, should also not have been disturbed. Even if you swap all the classical atoms in a brain at the same time, if the person doesn't notice anything happen, why, it probably didn't.

And of course there's the classic (and classical) argument, "Well, your body's turnover time for atoms is seven years on average."

But it's a moot argument.

We don't live in a classical universe.

We live in a quantum universe where the notion of "same hydrogen atom vs. different hydrogen atom" is physical nonsense.

We live in a universe where the whole notion of billiard balls bopping around is fundamentally wrong.

This can be a disorienting realization, if you formerly thought of yourself as an awareness ball that moves around.

Sorry. Your parietal cortex is fooling you on this one.

But wait! It gets even worse!

The brain doesn't exactly repeat itself; the state of your brain one second from now is not the state of your brain one second ago. The neural connections don't all change every second, of course. But there are enough changes every second that the brain's state is not cyclic, not over the course of a human lifetime. With every fragment of memory you lay down - and every thought that pops in and out of short-term memory - and every glance of your eyes that changes the visual field of your visual cortex - you ensure that you never repeat yourself exactly.

Over the course of a single second - not seven years, but one second - the joint position of all the atoms in your brain, will change far enough away from what it was before, that there is no overlap with the previous joint amplitude distribution. The brain doesn't repeat itself. Over the course of one second, you will end up being comprised of a completely different, nonoverlapping volume of configuration space.

And the quantum configuration space is the most fundamental known reality, according to our best current theory, remember. Even if quantum theory turns out not to be really truly fundamental, it has already finished superseding the hallucination of individual particles. We're never going back to billiard balls, any more than we're going back to Newtonian mechanics or phlogiston theory. The ratchet of science turns, but it doesn't turn backward.

And actually, the time for you to be comprised of a completely different volume of configuration space, is way less than a second. That time is the product of all the individual changes in your brain put together. It'll be less than a millisecond, less than a femtosecond, less than the time it takes light to cross a neutron diameter. It works out to less than the Planck time, if that turns out to make physical sense.

And then there's the point to consider that the physically real amplitude distribution is over a configuration space of all the particles in the universe. "You" are just a factored subspace of that distribution.

Yes, that's right, I'm calling you a factored subspace.

None of this should be taken as saying that you are somehow independent of the quantum physics comprising you. If an anvil falls on your head, you will stop talking about consciousness. This is experimentally testable. Don't try it at home.

But the notion that you can equate your personal continuity, with the identity of any physically real constituent of your existence, is absolutely and utterly hopeless.

You are not "the same you, because you are made of the same atoms". You have zero overlap with the fundamental constituents of yourself from even one nanosecond ago. There is continuity of information, but not equality of parts.

The new factor over the subspace looks a whole lot like the old you, and not by coincidence: The flow of time is lawful, there are causes and effects and preserved commonalities. Look to the regularity of physics, if you seek a source of continuity. Do not ask to be composed of the same objects, for this is hopeless.

Whatever makes you feel that your present is connected to your past, it has nothing to do with an identity of physically fundamental constituents over time.

Which you could deduce a priori, even in a classical universe, using the Generalized Anti-Zombie Principle. The imaginary identity-tags that read "This is electron #234,567..." don't affect particle motions or anything else; they can be swapped without making a difference because they're epiphenomenal. But since this final conclusion happens to be counterintuitive to a human parietal cortex, it helps to have the brute fact of quantum mechanics to crush all opposition.

Damn, have I waited a long time to be able to say that.

And no, this isn't the only point I have to make on how counterintuitive physics rules out intuitive conceptions of personal identity. I've got even stranger points to make. But those will take more physics first.

Sequence: Quantum Physics

No Individual Particles

Decoherence

Referenced by: Three Dialogues on Identity • When Science Can't Help • Timeless Identity • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity

Original with comments: Identity Isn't In Specific Atoms

Zombies: The Movie

Eliezer Yudkowsky, 20 April 2008 05:53AM

FADE IN around a serious-looking group of uniformed military officers. At the head of the table, a senior, heavy-set man, GENERAL FRED, speaks.

GENERAL FRED: The reports are confirmed. New York has been overrun... by zombies.

COLONEL TODD: Again? But we just had a zombie invasion 28 days ago!

GENERAL FRED: These zombies... are different. They're... philosophical zombies.

CAPTAIN MUDD: Are they filled with rage, causing them to bite people?

COLONEL TODD: Do they lose all capacity for reason?

GENERAL FRED: No. They behave... exactly like we do... except that they're not conscious.

(Silence grips the table.)

COLONEL TODD: Dear God.

GENERAL FRED moves over to a computerized display.

GENERAL FRED: This is New York City, two weeks ago.

The display shows crowds bustling through the streets, people eating in restaurants, a garbage truck hauling away trash.

GENERAL FRED: This... is New York City... now.

The display changes, showing a crowded subway train, a group of students laughing in a park, and a couple holding hands in the sunlight.

COLONEL TODD: It's worse than I imagined.

CAPTAIN MUDD: How can you tell, exactly?

COLONEL TODD: I've never seen anything so brutally ordinary.

A lab-coated SCIENTIST stands up at the foot of the table.

SCIENTIST: The zombie disease eliminates consciousness without changing the brain in any way. We've been trying to understand how the disease is transmitted. Our conclusion is that, since the disease attacks dual properties of ordinary matter, it must, itself, operate outside our universe. We're dealing with an epiphenomenal virus.

GENERAL FRED: Are you sure?

SCIENTIST: As sure as we can be in the total absence of evidence.

GENERAL FRED: All right. Compile a report on every epiphenomenon ever observed. What, where, and who. I want a list of everything that hasn't happened in the last fifty years.

CAPTAIN MUDD: If the virus is epiphenomenal, how do we know it exists?

SCIENTIST: The same way we know we're conscious.

CAPTAIN MUDD: Oh, okay.

GENERAL FRED: Have the doctors made any progress on finding an epiphenomenal cure?

SCIENTIST: They've tried every placebo in the book. No dice. Everything they do has an effect.

GENERAL FRED: Have you brought in a homeopath?

SCIENTIST: I tried, sir! I couldn't find any!

GENERAL FRED: Excellent. And the Taoists?

SCIENTIST: They refuse to do anything!

GENERAL FRED: Then we may yet be saved.

COLONEL TODD: What about David Chalmers? Shouldn't he be here?

GENERAL FRED: Chalmers... was one of the first victims.

COLONEL TODD: Oh no.

(Cut to the INTERIOR of a cell, completely walled in by reinforced glass, where DAVID CHALMERS paces back and forth.)

DOCTOR: David! David Chalmers! Can you hear me?

CHALMERS: Yes.

NURSE: It's no use, doctor.

CHALMERS: I'm perfectly fine. I've been introspecting on my consciousness, and I can't detect any difference. I know I would be expected to say that, but -

The DOCTOR turns away from the glass screen in horror.

DOCTOR: His words, they... they don't mean anything.

CHALMERS: This is a grotesque distortion of my philosophical views. This sort of thing can't actually happen!

DOCTOR: Why not?

NURSE: Yes, why not?

CHALMERS: Because -

(Cut to two POLICE OFFICERS, guarding a dirt road leading up to the imposing steel gate of a gigantic concrete complex. On their uniforms, a badge reads "BRIDGING LAW ENFORCEMENT AGENCY".)

OFFICER 1: You've got to watch out for those clever bastards. They look like humans. They can talk like humans. They're identical to humans on the atomic level. But they're not human.

OFFICER 2: Scumbags.

The huge noise of a throbbing engine echoes over the hills. Up rides the MAN on a white motorcycle. The MAN is wearing black sunglasses and a black leather business suit with a black leather tie and silver metal boots. His white beard flows in the wind. He pulls to a halt in front of the gate.

The OFFICERS bustle up to the motorcycle.

OFFICER 1: State your business here.

MAN: Is this where you're keeping David Chalmers?

OFFICER 2: What's it to you? You a friend of his?

MAN: Can't say I am. But even zombies have rights.

OFFICER 1: All right, buddy, let's see your qualia.

MAN: I don't have any.

OFFICER 2 suddenly pulls a gun, keeping it trained on the MAN. OFFICER 2: Aha! A zombie!

OFFICER 1: No, zombies claim to have qualia.

OFFICER 2: So he's an ordinary human?

OFFICER 1: No, they also claim to have qualia.

The OFFICERS look at the MAN, who waits calmly.

OFFICER 2: Um...

OFFICER 1: Who are you?

MAN: I'm Daniel Dennett, bitches.

Seemingly from nowhere, DENNETT pulls a sword and slices OFFICER 2's gun in half with a steely noise. OFFICER 1 begins to reach for his own gun, but DENNETT is suddenly standing behind OFFICER 1 and chops with a fist, striking the junction of OFFICER 1's shoulder and neck. OFFICER 1 drops to the ground.

OFFICER 2 steps back, horrified.

OFFICER 2: That's not possible! How'd you do that?

DENNETT: I am one with my body.

DENNETT drops OFFICER 2 with another blow, and strides toward the gate. He looks up at the imposing concrete complex, and grips his sword tighter.

DENNETT (quietly to himself): There is a spoon.

(Cut back to GENERAL FRED and the other military officials.)

GENERAL FRED: I've just received the reports. We've lost Detroit.

CAPTAIN MUDD: I don't want to be the one to say "Good riddance", but -

GENERAL FRED: Australia has been... reduced to atoms.

COLONEL TODD: The epiphenomenal virus is spreading faster. Civilization itself threatens to dissolve into total normality. We could be looking at the middle of humanity.

CAPTAIN MUDD: Can we negotiate with the zombies?

GENERAL FRED: We've sent them messages. They sent only a single reply.

CAPTAIN MUDD: Which was...?

GENERAL FRED: It's on its way now.

An orderly brings in an envelope, and hands it to GENERAL FRED.

GENERAL FRED opens the envelope, takes out a single sheet of paper, and reads it.

Silence envelops the room.

CAPTAIN MUDD: What's it say?

GENERAL FRED: It says... that we're the ones with the virus.

(A silence falls.)

COLONEL TODD raises his hands and stares at them.

COLONEL TODD: My God, it's true. It's true. I...

(A tear rolls down COLONEL TODD's cheek.)

COLONEL TODD: I don't feel anything.

The screen goes black.

The sound goes silent.

The movie continues exactly as before.

PS: This is me being attacked by zombie nurses at Penguicon.

Only at a combination science fiction and open-source convention would it be possible to attend a session on knife-throwing, cry "In the name of Bayes, die!", throw the knife, and then have a fellow holding a wooden shield say, "Yes, but how do you determine the prior for where the knife hits?"

Sequence: Zombies

Belief in the Implied Invisible

Referenced by: Continuous Improvement • (Moral) Truth in Fiction? • I'm Not Saying People Are Stupid

Original with comments: Zombies: The Movie

Three Dialogues on Identity

Eliezer Yudkowsky, 21 April 2008 06:13AM

This post is part of the Quantum Physics Sequence.
Followup to: Identity Isn't In Specific Atoms

It is widely said that some primitive tribe or other once feared that photographs could steal their souls.

Ha ha! How embarrassing. Silly tribespeople.

I shall now present three imaginary conversations along such lines - the common theme being frustration.

The first conversation:

Foolishly leaving the world of air-conditioning, you traveled to the Godforsaken Outback, and of course, got lost in the woods. A more primitive tribe than yours, the Hu'wha, saved your butt. Although the Hu'wha have told you how to reach an outpost of Internet access, that is, civilization, you've stayed with them a while longer; you've become their friend, and they yours.

One custom of the Hu'wha does seem strange to you, coming as you do from a more civilized culture: They don't hold with lies, even small ones. They consider a lie as an infringement upon the soul of the listener. They have a saying, "It is better to die than to be lied to." Though this is a very strange and primitive custom, you have come to respect it.

Late one night, the shaman calls you to his tent. His face is grave. "I have heard the most disturbing news," he says, "from the Tribe That Lives Across The Water. They say that your people, the People of the Net, have a most terrible custom: they paint images of others, and thereby steal their souls, for a person cannot be in two places at once. It is even said that you have weapons called 'cameras', for doing this automatically; and that the cameras of your folk can be very small, or disguised as other things."

"Um," you say, "I think you may be laboring under certain basic misconceptions. Cameras are not weapons; they make images, but they don't steal souls."

The grey-bearded shaman smiles, and shakes his head. "Young fellow, I am the shaman of the Hu'wha, and I hold the tradition passed down from my father through many generations; the true and original wisdom granted by the gods to the first shaman. I think I know what steals a soul and what does not, young fellow! Even to you it should be obvious."

And you think: Foolish mortal, how little you understand the power of Science. But this is beyond the conception of this man who thinks himself above you, and so you say nothing.

"I understand," the shaman says, "that your people may be so utterly ignorant of magic that they don't realize their cameras are dangerous. But that makes it all the more urgent that I ask you, Net-user, upon your honor: Have you by any means whatever, in your time among us, whether yourself, or by any device, produced an image of anyone here? If you have, we will do no violence to you - for I know there is no malice in you - but you will no longer be welcome among us."

You pause. The Hu'wha set great store on the literal truth of words, as well as their intent. And though you have no camera or paintbrushes, the answer to the question just asked, is literally yes. Your eyes, retina, and optic nerve are constantly painting images in your visual cortex.

"I haven't made any pictures the way you mean it," you say.

The shaman frowns. "I was looking for a simple No. Why the hesitation?"

Oh, dear. "The knowledge of my own people, the Net-folk, is not like your own knowledge," you say, "and you asked a... deeper question than you know, according to the beliefs of my own people."

"This is a very simple matter," the shaman says sharply, "and it has to do with what you have done. Have you made any pictures, or not?"

"I've painted no picture, and used no camera."

"Have you caused a picture to be made by any other means?" demands the shaman.

Dammit. "Not the way you mean it. I've done nothing that the Hu'wha do not also do."

"Explain yourself!"

You sigh. "It is a teaching of my people, which you are welcome to believe or not as it suits you, that pictures are constantly being created of all of us, all the time."

"What?" says the shaman.

"When you look at someone," you explain, "or even when an animal looks at you, that creates an image on the inside of the skull... that is how you see. Indeed, it is what you see - everything you see is a picture your eyes create."

"That's nonsense," says the shaman. "You're right there! I'm seeing you, not an image of you! Now I ask you again, on your honor: Do we Hu'wha still have our souls since you came among us, or not?"

Oh, bloody hell. "It is a teaching of my people," you say, "that what you call a 'soul' is... a confused idea."

"You are being evasive," says the shaman sternly. "The soul is not complicated, and it would be very hard to mistake a soul for something else, like a shoe or something. Our souls are breathed into us by Great Ghu at birth, and stays with us our whole lives, unless someone steals it; and if no one has photographed us, our souls go to the Happy Gaming Room when we die. Now I ask you again: Do I have my soul, or not? Give me the truth!"

"The truth," you say, "is that the way my people see the world is so different from yours, that you can't even imagine what I think is the truth. I've painted no pictures, taken no photographs; all I've done is look at you, and nothing happens when I look at you, that doesn't happen when anyone else looks at you. But you are being constantly photographed, all the time, and you never had any soul to begin with: this is the truth."

"Horse output," says the shaman. "Go away; we never want to see you again."

The second conversation:

John Smith still looked a little pale. This was quite understandable. Going to a pleasant dinner with your family, having a sudden heart attack, riding to the hospital by ambulance, dying, being cryonically suspended by Alcor, spending decades in liquid nitrogen, and then awakening, all in the span of less than 24 subjective hours, will put a fair amount of stress on anyone.

"Look," said John, "I accept that there are things you're not allowed to tell me -"

"Not right away," you say. "We've found that certain pieces of information are best presented in a particular order."

John nods. "Fine, but I want to be very clear that I don't want to be told any comforting lies. Not for the sake of my 'psychological health', and not for anything. If you can't tell me, just say nothing. Please."

You raise your hand to your chest, two fingers out and the others folded. "That, I can promise: I cannot tell you everything, but what I say to you will be true. In the name of Richard Feynman, who is dead but not forgotten."

John is giving you a very strange look. "How long did you say I was suspended?"

"Thirty-five years," you say.

"I was thinking," said John, "that things surely wouldn't have changed all that much in thirty-five years."

You say nothing, thus keeping your promise.

"But if things have changed that much," John says, "I want to know something. Have I been uploaded?"

You frown. "Uploaded? I'm sorry, I don't understand. The word 'upload' used to apply to computer files, right?"

"I mean," says John, "have I been turned into a program? An algorithm somewhere?"

Huh? "Turned into an algorithm? What were you before, a constant integer?"

"Aargh," says John. "Okay, yes, I'm a program, you're a program. Every human in the history of humanity has been a program running on their brain. I understand that. What I want to know is whether me, this John Smith, the one talking to you right now, is a program on the same hardware as the John Smith who got cryonically suspended."

You pause. "What do you mean, 'same hardware'?"

John starts to look worried. "I was hoping for a simple 'Yes', there. Am I made of the same atoms as before, or not?"

Oh, dear. "I think you may be laboring under certain basic misconceptions," you say.

"I understand," John said, "that your people may have the cultural belief that uploading preserves personal identity - that a human is memories and personality, not particular atoms. But I happen to believe that my identity is bound up with the atoms that make me. It's not as if there's an experiment you can do to prove that I'm wrong, so my belief is just as valid as yours."

Foolish child, you think, how little you understand the power of Science. "You asked a deeper question than you know," you say, "and the world does not work the way you think it does. An atom is... not what you imagine."

"Look," John says sharply, "I'm not asking you about this time's theories of personal identity, or your beliefs about consciousness - that's all outside the realm of third-party scientific investigation anyway. I'm asking you a simple question that is experimentally testable. Okay, you found something new underneath the quarks. That's not surprising. I'm asking, whatever stuff I am made of, is it the same stuff as before? Yes or no?"

The third conversation:

Your question is itself confused. Whatever is, is real.

"Look," Eliezer said, "I know I'm not being misunderstood, so I'm not going to try and phrase this the elaborately correct way: Is this thing that I'm holding an old-fashioned banana, or does it only have the appearance of a banana?"

You wish to know if the accustomed state of affairs still holds. In which it merely appears that there is a banana in your hand, but actually, there is something very different behind the appearance: a configuration of particles, held together by electromagnetic fields and other laws that humans took centuries to discover.

"That's right. I want to know if the lower levels of organization underlying the banana have a substantially different structure than before, and whether the causal relation between that structure and my subjective experience has changed in style."

Well then. Rest assured that you are not holding the mere appearance of a banana. There really is a banana there, not just a collection of atoms.

There was a long pause.

"WHAT?"

Or perhaps that was only a joke. Let it stand that the place in which you find yourself is at least as real as anywhere you ever thought you were, and the things you see are even less illusionary than your subjective experiences of them.

"Oh, come on! I'm not some hunter-gatherer worried about a photographer stealing his soul! If I'm running on a computer somewhere, and this is a virtual environment, that's fine! I was just curious, that's all."

Some of what you believe is true, and some of what you believe is false: this may also be said of the hunter-gatherer. But there is a true difference between yourself and the hunter-gatherer, which is this: You have a concept of what it means for a fundamental assumption to be mistaken. The hunter-gatherer has no experience with other cultures that believe differently, no history that tells of past scientific revolutions. But you know what is meant, whether or not you accept it, you understand the assertion itself: Some of your fundamental assumptions are mistaken.

Referenced by: The Quantum Physics Sequence • Quantum Mechanics and Personal Identity

Original with comments: Three Dialogues on Identity

Decoherence

Eliezer Yudkowsky, 22 April 2008 06:41AM

This post is part of the Quantum Physics Sequence.
Previously in series: Feynman Paths

To understand the quantum process called "decoherence", we first need to look at how the special case of quantum independence can be destroyed - how the evolution of a quantum system can produce entanglement where there was formerly independence.

Quantum independence, as you'll recall, is a special case of amplitude distributions that approximately factorize - amplitude distributions that can be treated as a product of sub-distributions over subspaces.

Reluctant tourists visiting quantum universes think as if the absence of a rectangular plaid pattern is some kind of special ghostly link between particles. Hence the unfortunate term, "quantum entanglement".

The evolution of a quantum system can produce entanglement where there was formerly independence - turn a rectangular plaid pattern into something else. Quantum independence, being a special case, is easily lost.

Let's pretend for a moment that we're looking at a classical system, which will make it easier to see what kind of physical process leads to entanglement.

At right is a system in which a positively charged light thingy is on a track, far above a negatively charged heavy thingy on a track.

At the beginning, the two thingies are far enough apart that they're not significantly interacting.

But then we lower the top track, bringing the two thingies into the range where they can easily attract each other. (Opposite charges attract.)

So the light thingy on top rolls toward the heavy thingy on the bottom. (And the heavy thingy on the bottom rolls a little toward the top thingy, just like an apple attracts the Earth as it falls.)

Now switch to the Feynman path integral view. That is, imagine the evolution of a quantum system as a sum over all the paths through configuration space the initial conditions could take.

Suppose the bottom heavy thingy and the top thingy started out in a state of quantum independence, so that we can view the amplitude distribution over the whole system as the product of a "bottom thingy distribution" and a "top thingy distribution".

The bottom thingy distribution starts with bulges in three places - which, in the Feynman path view, we might think of as three possible starting configurations from which amplitude will flow.

When we lower the top track, the light thingy on top is attracted toward the heavy bottom thingy -

- except that the bottom thingy has a sub-distribution with three bulges in three different positions.

So the end result is a joint distribution in which there are three bulges in the amplitude distribution over joint configuration space, corresponding to three different joint positions of the top thingy and bottom thingy.

I've been trying very carefully to avoid saying things like "The bottom thingy is in three places at once" or "in each possibility, the top thingy is attracted to wherever the bottom thingy is".

Still, you're probably going to visualize it that way, whether I say it or not. To be honest, that's how I drew the diagram - I visualized three possibilities and three resulting outcomes. Well, that's just how a human brain tends to visualize a Feynman path integral.

But this doesn't mean there are actually three possible ways the universe could be, etc. That's just a trick for visualizing the path integral. All the amplitude flows actually happen, they are not possibilities.

Now imagine that, in the starting state, the bottom thingy has an amplitude-factor that is smeared out over the whole bottom track; and the top thingy has an amplitude-factor in one place. Then the joint distribution over "top thingy, bottom thingy" would start out looking like the plaid pattern at left, and develop into the non-plaid pattern at right:

Here the horizontal coordinate corresponds to the top thingy, and the vertical coordinate corresponds to the bottom thingy. So we start with the top thingy localized and the bottom thingy spread out, and then the system develops into a joint distribution where the top thingy and the bottom thingy are in the same place, but their mutual position is spread out. Very loosely speaking.

So an initially factorizable distribution, evolved into an "entangled system" - a joint amplitude distribution that is not viewable as a product of distinct factors over subspaces.

(Important side note: You'll note that, in the diagram above, system evolution obeyed the second law of thermodynamics, aka Liouville's Theorem. Quantum evolution conserved the "size of the cloud", the volume of amplitude, the total amount of grey area in the diagram.

If instead we'd started out with a big light-gray square - meaning that both particles had amplitude-factors widely spread - then the second law of thermodynamics would prohibit the combined system from developing into a tight dark-gray diagonal line.

A system has to start in a low-entropy state to develop into a state of quantum entanglement, as opposed to just a diffuse cloud of amplitude.

Mutual information is also negentropy, remember. Quantum amplitudes aren't information per se, but the rule is analogous: Amplitude must be highly concentrated to look like a neatly entangled diagonal line, instead of just a big diffuse cloud. If you imagine amplitude distributions as having a "quantum entropy", then an entangled system has low quantum entropy.)

Okay, so now we're ready to discuss decoherence.

The system at left is highly entangled - it's got a joint distribution that looks something like, "There's two particles, and either they're both over here, or they're both over there."

Yes, I phrased this as if there were two separate possibilities, rather than a single physically real amplitude distribution. Seriously, there's no good way to use a human brain to talk about quantum physics in English.

But if you can just remember the general rule that saying "possibility" is shorthand for "physically real blob within the amplitude distribution", then I can describe amplitude distributions a lot faster by using the language of uncertainty. Just remember that it is language. "Either the particle is over here, or it's over there" means a physically real amplitude distribution with blobs in both places, not that the particle is in one of those places but we don't know which.

Anyway. Dealing with highly entangled systems is often annoying - for human physicists, not for reality, of course. It's not just that you've got to calculate all the possible outcomes of the different possible starting conditions. (I.e., add up a lot of physically real amplitude flows in a Feynman path integral.) The possible outcomes may interfere with each other. (Which actual possible outcomes would never do, but different blobs in an amplitude distribution do.) Like, maybe the two particles that are both over here, or both over there, meet twenty other particles and do a little dance, and at the conclusion of the path integral, many of the final configurations have received amplitude flows from both initial blobs.

But that kind of extra-annoying entanglement only happens when the blobs in the initial system are close enough that their evolutionary paths can slop over into each other. Like, if the particles were either both here, or both there, but here and there were two light-years apart, then any system evolution taking less than a year, couldn't have the different possible outcomes overlapping.

Okay, so let's talk about three particles now.

This diagram shows a blob of amplitude that factors into the product of a 2D subspace and a 1D subspace. That is, two entangled particles and one independent particle.

The vertical dimension is the one independent particle, the length and breadth are the two entangled particles.

The independent particle is in one definite place - the cloud of amplitude is vertically narrow. The two entangled particles are either both here, or both there. (Again I'm using that wrong language of uncertainty, words like "definite" and "either", but you see what I mean.)

Now imagine that the third independent particle interacts with the two entangled particles in a sensitive way. Maybe the third particle is balanced on the top of a hill; and the two entangled particles pass nearby, and attract it magnetically; and the third particle falls off the top of the hill and rolls to the bottom, in that particular direction.

Afterward, the new amplitude distribution might look like this. The third particle is now entangled with the other two particles. And the amplitude distribution as a whole consists of two more widely separated blobs.

Loosely speaking, in the case where the two entangled particles were over here, the third particle went this way, and in the case where the two entangled particles were over there, the third particle went that way.

So now the final amplitude distribution is fully entangled - it doesn't factor into subspaces at all.

But the two blobs are more widely separated in the configuration space. Before, each blob of amplitude had two particles in different positions; now each blob of amplitude has three particles in different positions.

Indeed, if the third particle interacted in an especially sensitive way, like being tipped off a hill and sliding down, the new separation could be much larger than the old one.

Actually, it isn't necessary for a particle to get tipped off a hill. It also works if you've got twenty particles interacting with the first two, and ending up entangled with them. Then the new amplitude distribution has got two blobs, each with twenty-two particles in different places. The distance between the two blobs in the joint configuration space is much greater.

And the greater the distance between blobs, the less likely it is that their amplitude flows will intersect each other and interfere with each other.

That's decoherence. Decoherence is the third key to recovering the classical hallucination, because it makes the blobs behave independently; it lets you treat the whole amplitude distribution as a sum of separated non-interfering blobs.

Indeed, once the blobs have separated, the pattern within a single blob may look a lot more plaid and rectangular - I tried to show that in the diagram above as well.

Thus, the big headache in quantum computing is preventing decoherence. Quantum computing relies on the amplitude distributions staying close enough together in configuration space to interfere with each other. And the environment contains a zillion particles just begging to accidentally interact with your fragile qubits, teasing apart the pieces of your painstakingly sculpted amplitude distribution.

And you can't just magically make the pieces of the scattered amplitude distribution jump back together - these are blobs in the joint configuration, remember. You'd have to put the environmental particles in the same places, too.

(Sounds pretty irreversible, doesn't it? Like trying to unscramble an egg? Well, that's a very good analogy, in fact.

This is why I emphasized earlier that entanglement happens starting from a condition of low entropy. Decoherence is irreversible because it is an essentially thermodynamic process.

It is a fundamental principle of the universe - as far as we can tell - that if you "run the film backward" all the fundamental laws are still obeyed. If you take a movie of an egg falling onto the floor and smashing, and then play the film backward and see a smashed egg leaping off the floor and into a neat shell, you will not see the known laws of physics violated in any particular. All the molecules will just happen to bump into each other in just the right way to make the egg leap off the floor and reassemble. It's not impossible, just unbelievably improbable.

Likewise with a smashed amplitude distribution suddenly assembling many distantly scattered blobs into mutual coherence - it's not impossible, just extremely improbable that many distant starting positions would end up sending amplitude flows to nearby final locations. You are far more likely to see the reverse.

Actually, in addition to running the film backward, you've got to turn all the positive charges to negative, and reverse left and right (or some other single dimension - essentially you have to turn the universe into its mirror image).

This is known as CPT symmetry, for Charge, Parity, and Time.

CPT symmetry appears to be a really, really, really deep principle of the universe. Trying to violate CPT symmetry doesn't sound quite as awful to a modern physicist as trying to throw a baseball so hard it travels faster than light. But it's almost that awful. I'm told that General Relativity Quantum Field Theory requires CPT symmetry, for one thing.

So the fact that decoherence looks like a one-way process, but is only thermodynamically irreversible rather than fundamentally asymmetrical, is a very important point. It means quantum physics obeys CPT symmetry.

It is a universal rule in physics - according to our best current knowledge - that every apparently irreversible process is a special case of the second law of thermodynamics, not the result of time-asymmetric fundamental laws.)

To sum up:

Decoherence is a thermodynamic process of ever-increasing quantum entanglement, which, through an amazing sleight of hand, masquerades as increasing quantum independence: Decoherent blobs don't interfere with each other, and within a single blob but not the total distribution, the blob is more factorizable into subspaces.

Thus, decoherence is the third key to recovering the classical hallucination. Decoherence lets a human physicist think about one blob at a time, without worrying about how blobs interfere with each other; and the blobs themselves, considered as isolated individuals, are less internally entangled, hence easier to understand. This is a fine thing if you want to pretend the universe is classical, but not so good if you want to factor a million-digit number before the Sun burns out.

Sequence: Quantum Physics

Identity Isn't In Specific Atoms

The So-Called Heisenberg Uncertainty Principle

Referenced by: The So-Called Heisenberg Uncertainty Principle • Where Physics Meets Experience • On Being Decoherent • Spooky Action at a Distance: The No-Communication Theorem • Collapse Postulates • Timeless Physics • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Decoherence

The So-Called Heisenberg Uncertainty Principle

Eliezer Yudkowsky, 23 April 2008 06:36AM

This post is part of the Quantum Physics Sequence.
Previously in series: Decoherence

As touched upon earlier, Heisenberg's "Uncertainty Principle" is horribly misnamed.

Amplitude distributions in configuration space evolve over time. When you specify an amplitude distribution over joint positions, you are also necessarily specifying how the distribution will evolve. If there are blobs of position, you know where the blobs are going.

In classical physics, where a particle is, is a separate fact from how fast it is going. In quantum physics this is not true. If you perfectly know the amplitude distribution on position, you necessarily know the evolution of any blobs of position over time.

So there is a theorem which should have been called the Heisenberg Certainty Principle, or the Heisenberg Necessary Determination Principle; but what does this theorem actually say?

At left is an image I previously used to illustrate a possible amplitude distribution over positions of a 1-dimensional particle.

Suppose that, instead, the amplitude distribution is actually a perfect helix. (I.e., the amplitude at each point has a constant modulus, but the complex phase changes linearly with the position.) And neglect the effect of potential energy on the system evolution; i.e., this is a particle out in intergalactic space, so it's not near any gravity wells or charged particles.

If you started with an amplitude distribution that looked like a perfect spiral helix, the laws of quantum evolution would make the helix seem to rotate / move forward at a constant rate. Like a corkscrew turning at a constant rate.

This is what a physicist views as a single particular momentum.

And you'll note that a "single particular momentum" corresponds to an amplitude distribution that is fully spread out - there's no bulges in any particular position.

Let me emphasize that I have not just described a real situation you could find a particle in.

The physicist's notion of "a single particular momentum" is a mathematical tool for analyzing quantum amplitude distributions.

The evolution of the amplitude distribution involves things like taking the second derivative in space and multiplying by i to get (one component of) the first derivative in time. Which turns out to give rise to a wave mechanics - blobs that can propagate themselves across space, over time.

One of the basic tools in wave mechanics is taking apart complicated waves into a sum of simpler waves.

If you've got a wave that bulges in particular places, and thus changes in pitch and diameter, then you can take apart that ugly wave into a sum of prettier waves.

A sum of simpler waves whose individual behavior is easy to calculate; and then you just add those behaviors back together again.

A sum of nice neat waves, like, say, those perfect spiral helices corresponding to precise momenta.

A physicist can, for mathematical convenience, decompose a position distribution into an integral over (infinitely many) helices of different pitches, phases, and diameters.

Which integral looks like assigning a complex number to each possible pitch of the helix. And each pitch of the helix corresponds to a different momentum. So you get a complex distribution over momentum-space.

It happens to be a fact that, when the position distribution is more concentrated - when the position distribution bulges more sharply - the integral over momentum-helices gets more widely distributed.

Which has the physical consequence, that anything which is very sharply in one place, tends to soon spread itself out. Narrow bulges don't last.

Alternatively, you might find it convenient to think, "Hm, a narrow bulge has sharp changes in its second derivative, and I know the evolution of the amplitude distribution depends on the second derivative, so I can sorta imagine how a narrow bulge might tend to propagate off in all directions."

Technically speaking, the distribution over momenta is the Fourier transform of the distribution over positions. And it so happens that, to go back from momenta to positions, you just do another Fourier transform. So there's a precisely symmetrical argument which says that anything moving at a very definite speed, has to occupy a very spread-out place. Which goes back to what was shown before, about a perfect helix having a "definite momentum" (corkscrewing at a constant speed) but being equally distributed over all positions.

That's Heisenberg's Necessary Relation Between Position Distribution And Position Evolution Which Prevents The Position Distribution And The Momentum Viewpoint From Both Being Sharply Concentrated At The Same Time Principle in a nutshell.

So now let's talk about some of the assumptions, issues, and common misinterpretations of Heisenberg's Misnamed Principle.

The effect of observation on the observed

Here's what actually happens when you "observe a particle's position":

Decoherence, as discussed yesterday, can take apart a formerly coherent amplitude distribution into noninteracting blobs.

Let's say you have a particle X with a fairly definite position and fairly definite momentum, the starting stage shown at left above. And then X comes into the neighborhood of another particle S, or set of particles S, where S is highly sensitive to X's exact location - in particular, whether X's position is on the left or right of the black line in the middle. For example, S might be poised at the top of a knife-edge, and X could tip it off to the left or to the right.

The result is to decohere X's position distribution into two noninteracting blobs, an X-to-the-left blob and an X-to-the-right blob. Well, now the position distribution within each blob, has become sharper. (Remember: Decoherence is a process of increasing quantum entanglement that masquerades as increasing quantum independence.)

So the Fourier transform of the more definite position distribution within each blob, corresponds to a more spread-out distribution over momentum-helices.

Running the particle X past a sensitive system S, has decohered X's position distribution into two noninteracting blobs. Over time, each blob spreads itself out again, by Heisenberg's Sharper Bulges Have Broader Fourier Transforms Principle.

All this gives rise to very real, very observable effects.

In the system shown at right, there is a light source, a screen blocking the light source, and a single slit in the screen.

Ordinarily, light seems to go in straight lines (for less straightforward reasons). But in this case, the screen blocking the light source decoheres the photon's amplitude. Most of the Feynman paths hit the screen.

The paths that don't hit the screen, are concentrated into a very narrow range. All positions except a very narrow range have decohered away from the blob of possibilities for "the photon goes through the slit", so, within this blob, the position-amplitude is concentrated very narrowly, and the spread of momenta is vey large.

Way up at the level of human experimenters, we see that when photons strike the second screen, they strike over a broad range - they don't just travel in a straight line from the light source.

Wikipedia, and at least some physics textbooks, claim that it is misleading to ascribe Heisenberg effects to an "observer effect", that is, perturbing interactions between the measuring apparatus and the measured system:

"Sometimes it is a failure to measure the particle that produces the disturbance. For example, if a perfect photographic film contains a small hole, and an incident photon is not observed, then its momentum becomes uncertain by a large amount. By not observing the photon, we discover that it went through the hole."

However, the most technical treatment I've actually read was by Feynman, and Feynman seemed to be saying that, whenever measuring the position of a particle increases the spread of its momentum, the measuring apparatus must be delivering enough of a "kick" to the particle to account for the change.

In other words, Feynman seemed to assert that the decoherence perspective actually was dual to the observer-effect perspective - that an interaction which produced decoherence would always be able to physically account for any resulting perturbation of the particle.

Not grokking the math, I'm inclined to believe the Feynman version. It sounds pretty, and physics has a known tendency to be pretty.

The alleged effect of conscious knowledge on particles

One thing that the Heisenberg Student Confusion Principle DEFINITELY ABSOLUTELY POSITIVELY DOES NOT SAY is that KNOWING ABOUT THE PARTICLE or CONSCIOUSLY SEEING IT will MYSTERIOUSLY MAKE IT BEHAVE DIFFERENTLY because THE UNIVERSE CARES WHAT YOU THINK.

Decoherence works exactly the same way whether a system is decohered by a human brain or by a rock. Yes, physicists tend to construct very sensitive instruments that slice apart amplitude distributions into tiny little pieces, whereas a rock isn't that sensitive. That's why your camera uses photographic film instead of mossy leaves, and why replacing your eyeballs with grapes will not improve your vision. But any sufficiently sensitive physical system will produce decoherence, where "sensitive" means "developing to widely different final states depending on the interaction", where "widely different" means "the blobs of amplitude don't interact".

Does this description say anything about beliefs? No, just amplitude distributions. When you jump up to a higher level and talk about cognition, you realize that forming accurate beliefs requires sensors. But the decohering power of sensitive interactions can be analyzed on a purely physical level.

There is a legitimate "observer effect", and it is this: Brains that see, and pebbles that are seen, are part of a unified physics; they are both built out of atoms. To gain new empirical knowledge about a thingy, the particles in you have to interact with the particles in the thingy. It so happens that, in our universe, the laws of physics are pretty symmetrical about how particle interactions work - conservation of momentum and so on: if you pull on something, it pulls on you.

So you can't, in fact, observe a rock without affecting it, because to observe something is to depend on it - to let it affect you, and shape your beliefs. And, in our universe's laws of physics, any interaction in which the rock affects your brain, tends to have consequences for the rock as well.

Even if you're looking at light that left a distant star 500 years ago, then 500 years ago, emitting the light affected the star.

That's how the observer effect works. It works because everything is particles, and all the particles obey the same unified mathematically simple physics.

It does not mean the physical interactions we happen to call "observations" have a basic, fundamental, privileged effect on reality.

To suppose that physics contains a basic account of "observation" is like supposing that physics contains a basic account of being Republican. It projects a complex, intricate, high-order biological cognition onto fundamental physics. It sounds like a simple theory to humans, but it's not simple.

Linearity

One of the foundational assumptions physicists used to figure out quantum theory, is that time evolution is linear. If you've got an amplitude distribution X1 that evolves into X2, and an amplitude distribution Y1 that evolves into Y2, then the amplitude distribution (X1 + Y1) should evolve into (X2 + Y2).

(To "add two distributions" means that we just add the complex amplitudes at every point. Very simple.)

Physicists assume you can take apart an amplitude distribution into a sum of nicely behaved individual waves, add up the time evolution of those individual waves, and get back the actual correct future of the total amplitude distribution.

Linearity is why we can take apart a bulging blob of position-amplitude into perfect momentum-helices, without the whole model degenerating into complete nonsense.

The linear evolution of amplitude distributions is a theorem in the Standard Model of physics. But physicists didn't just stumble over the linearity principle; it was used to invent the hypotheses, back when quantum physics was being figured out.

I talked earlier about taking the second derivative of position; well, taking the derivative of a differentiable distribution is a linear operator. F'(x) + G'(x) = (F + G)'(x). Likewise, integrating the sum of two integrable distributions gets you the sum of the integrals. So the amplitude distribution evolving in a way that depends on the second derivative - or the equivalent view in terms of integrating over Feynman paths - doesn't mess with linearity.

Any "non-linear system" you've ever heard of is linear on a quantum level. Only the high-level simplifications that we humans use to model systems are nonlinear. (In the same way, the lightspeed limit requires physics to be local, but if you're thinking about the Web on a very high level, it looks like any webpage can link to any other webpage, even if they're not neighbors.)

Given that quantum physics is strictly linear, you may wonder how the hell you can build any possible physical instrument that detects a ratio of squared moduli of amplitudes, since the squared modulus operator is not linear: the squared modulus of the sum is not the sum of the squared moduli of the parts.

This is a very good question.

We'll get to it shortly.

Meanwhile, physicists, in their daily mathematical practice, assume that quantum physics is linear. It's one of those important little assumptions, like CPT invariance.

Sequence: Quantum Physics

Decoherence

Which Basis Is More Fundamental?

Referenced by: Which Basis Is More Fundamental? • On Being Decoherent • Decoherent Essences • The Born Probabilities • Decoherence as Projection • Spooky Action at a Distance: The No-Communication Theorem • Collapse Postulates • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: The So-Called Heisenberg Uncertainty Principle

Which Basis Is More Fundamental?

Eliezer Yudkowsky, 24 April 2008 04:17AM

This post is part of the Quantum Physics Sequence.
Followup to: The So-Called Heisenberg Uncertainty Principle

For decades, quantum physics was vehemently asserted to be nothing but a convenience of calculation. The equations were not to be interpreted as describing reality, though they made good predictions for reasons that it was mere philosophy to question. This being the case, any quantity you could define seemed as fundamentally real as any other quantity, which is to say, not real at all.

Physicists have invented, for convenience of calculation, something called a momentum basis of quantum mechanics. Instead of having a complex amplitude distribution over the positions of particles, you had a complex amplitude distribution over their momenta.

The "momentum basis" contains all the information that is in the "position basis", and the "position basis" contains all the information that is in the "momentum basis". Physicists use the word "basis" for both, suggesting that they are on the same footing: that positions are no better than momenta, or vice versa.

But, in my humble opinion, the two representations are not on an equal footing when it comes to being "fundamental".

Physics in the position basis can be computed locally. To determine the instantaneous change of amplitude at a configuration, you only need to look at its infinitesimal neighborhood.

The momentum basis cannot be computed locally. Quantum evolution depends on potential energy. Potential energy depends on how far apart things are from each other, like how high an apple is off the ground. To figure out how far apart things are from each other, you have to look at the entire momentum basis to recover the positions.

The "momentum basis" is in some ways like a description of the chessboard in which you have quantities like "the queen's position minus the rook's position" and "the queen's position plus the rook's position". You can get back a description of the entire chessboard - but the rules of the game are much harder to phrase. Each rule has to take into account many more facts, and there's no longer an elegant local structure to the board.

Now the above analogy is not really fair, because the momentum basis is not that inelegant. The momentum basis is the Fourier transform of the position basis, and symmetrically, the position basis is the Fourier transform of the momentum basis. They're equally easy to extract from each other. Even so, the momentum basis has no local physics.

So if you think that the nature of reality seems to tend toward local relations, local causality, or local anything, then the position basis is a better candidate for being fundamentally real.

What is this "nature of reality" that I'm talking about?

I sometimes talk about the Tao as being the distribution from which our laws of physics were drawn - the alphabet in which our physics was generated. This is almost certainly a false concept, but it is a useful one.

It was a very important discovery, in human history, that the Tao wrote its laws in the language of mathematics, rather than heroic mythology. We had to discover the general proposition that equations were better explanations for natural phenomena than "Thor threw a lightning bolt". (Even though Thor sounds simpler to humans than Maxwell's Equations.)

Einstein seems to have discovered General Relativity almost entirely on the basis of guessing what language the laws should be written in, what properties they should have, rather than by distilling vast amounts of experimental evidence into an empirical regularity. This is the strongest evidence I know of for the pragmatic usefulness of the "Tao of Physics" concept. If you get one law, like Special Relativity, you can look at the language it's written in, and infer what the next law ought to look like. If the laws are not being generated from the same language, they surely have something in common; and this I refer to as the Tao.

Why "Tao"? Because no matter how I try to describe the whole business, when I look over the description, I'm pretty sure it's wrong. Therefore I call it the Tao.

One of the aspects of the Tao of Physics seems to be locality. (Markov neighborhoods, to be precise.) Discovering this aspect of the Tao was part of the great transition from Newtonian mechanics to relativity. Newton thought that gravity and light propagated at infinite speed, action-at-a-distance. Now that we know that everything obeys a speed limit, we know that what happens at a point in spacetime only depends on an immediate neighborhood of the immediate past.

Ever since Einstein figured out that the Tao prefers its physics local, physicists have successfully used the heuristic of prohibiting all action-at-a-distance in their hypotheses. We've figured out that the Tao doesn't like it. You can see how local physics would be easier to compute... though the Tao has no objection to wasting incredible amounts of computing power on things like quarks and quantum mechanics.

The Standard Model includes many fields and laws. Our physical models require many equations and postulates to write out. To the best of our current knowledge, the laws still appear, if not complicated, then not perfectly simple.

Why should every known behavior in physics be linear in quantum evolution, local in space and time, Charge-Parity-Time symmetrical, and conservative of probability density? I don't know, but you'd have to be pretty stupid not to notice the pattern. A single exception, in any individual behavior of physics, would destroy the generalization. It seems like too much coincidence.

So, yes, the position basis includes all the information of the momentum basis, and the momentum basis includes all the information of the position basis, and they give identical predictions.

But the momentum basis looks like... well, it looks like humans took the real laws and rewrote them in a mathematically convenient way that destroys the Tao's beloved locality.

That may be a poor way of putting it, but I don't know how else to do so.

In fact, the position basis is also not a good candidate for being fundamentally real, because it doesn't obey the relativistic spirit of the Tao. Talking about any particular position basis, involves choosing an arbitrary space of simultaneity. Of course, transforming your description of the universe to a different space of simultaneity, will leave all your experimental predictions exactly the same. But however the Tao of Physics wrote the real laws, it seems really unlikely that they're written to use Greenwich's space of simultaneity as the arbitrary standard, or whatever. Even if you can formulate a mathematically equivalent representation that uses Greenwich space, it doesn't seem likely that the Tao actually wrote it that way... if you see what I mean.

I wouldn't be surprised to learn that there is some known better way of looking at quantum mechanics than the position basis, some view whose mathematical components are relativistically invariant and locally causal.

But, for now, I'm going to stick with the observation that the position basis is local, and the momentum basis is not, regardless of how pretty they look side-by-side. It's not that I think the position basis is fundamental, but that it seems fundamentaler.

The notion that every possible way of slicing up the amplitude distribution is a "basis", and every "basis" is on an equal footing, is a habit of thought from those dark ancient ages when quantum amplitudes were thought to be states of partial information.

You can slice up your information any way you like. When you're reorganizing your beliefs, the only question is whether the answers you want are easy to calculate.

But if a model is meant to describe reality, then I would tend to suspect that a locally causal model probably gets closer to fundamentals, compared to a nonlocal model with action-at-a-distance. Even if the two give identical predictions.

This is admittedly a deep philosophical issue that gets us into questions I can't answer, like "Why does the Tao of Physics like math and CPT symmetry?" and "Why should a locally causal isomorph of a structural essence, be privileged over nonlocal isomorphs when it comes to calling it 'real'?", and "What the hell is the Tao?"

Good questions, I agree.

This talk about the Tao is messed-up reasoning. And I know that it's messed up. And I'm not claiming that just because it's a highly useful heuristic, that is an excuse for it being messed up.

But I also think it's okay to have theories that are in progress, that are not even claimed to be in a nice neat finished state, that include messed-up elements clearly labeled as messed-up, which are to be resolved as soon as possible rather than just tolerated indefinitely.

That, I think, is how you make incremental progress on these kinds of problems - by working with incomplete theories that have wrong elements clearly labeled "WRONG!" Academics, it seems to me, have a bias toward publishing only theories that they claim to be correct - or even worse, complete - or worse yet, coherent. This, of course, rules out incremental progress on really difficult problems.

When using this methodology, you should, to avoid confusion, choose labels that clearly indicate that the theory is wrong. For example, the "Tao of Physics". If I gave that some kind of fancy technical-sounding formal name like "metaphysical distribution", people might think it was a name for a coherent theory, rather than a name for my own confusion.

I accept the possibility that this whole blog post is merely stupid. After all, the question of whether the position basis or the momentum basis is "more fundamental" should never make any difference as to what we anticipate. If you ever find that your anticipations come out one way in the position basis, and a different way in the momentum basis, you are surely doing something wrong.

But Einstein (and others!) seem to have comprehended the Tao of Physics to powerfully predictive effect. The question "What kind of laws does the Tao favor writing?" has paid more than a little rent.

The position basis looks noticeably more... favored.

Added: When I talk about "locality", I mean locality in the abstract, computational sense: mathematical objects talking only to their immediate neigbors. In particular, quantum physics is local in the configuration space.

This also happens to translate into physics that is local in what humans think of "space": it is impossible to send signals faster than light. But this isn't immediately obvious. It is an additional structure of the neighborhoods in configuration space. A configuration only neighbors configurations where positions didn't change faster than light.

A view that made both forms of locality explicit, in a relativistically invariant way, would be much more fundamentalish than the position basis. Unfortunately I don't know what such a view might be.

Sequence: Quantum Physics

The So-Called Heisenberg Uncertainty Principle

Where Physics Meets Experience

Referenced by: Decoherent Essences • Spooky Action at a Distance: The No-Communication Theorem • Collapse Postulates • The Quantum Physics Sequence

Original with comments: Which Basis Is More Fundamental?

Where Physics Meets Experience

Eliezer Yudkowsky, 25 April 2008 04:58AM

This post is part of the Quantum Physics Sequence.
Followup to: Decoherence, Where Philosophy Meets Science

Once upon a time, there was an alien species, whose planet hovered in the void of a universe with laws almost like our own. They would have been alien to us, but of course they did not think of themselves as alien. They communicated via rapid flashes of light, rather than sound. We'll call them the Ebborians.

Ebborians reproduce by fission, an adult dividing into two new individuals. They share genetic material, but not through sexual recombination; Ebborian adults swap genetic material with each other. They have two eyes, four legs, and two hands, letting a fissioned Ebborian survive long enough to regrow.

Human DNA is built in a double helix; unzipping the helix a little at a time produces two stretches of single strands of DNA. Each single strand attracts complementary bases, producing a new double strand. At the end of the operation, a DNA double helix has turned into two double helices. Hence earthly life.

Ebborians fission their brains, as well as their bodies, by a process something like how human DNA divides.

Imagine an Ebborian brain as a flat sheet of paper, computing in a way that is more electrical than chemical - charges flowing down conductive pathways.

When it's time for an Ebborian to fission, the brain-paper splits down its thickness into two sheets of paper. Each new sheet is capable of conducting electricity on its own. Indeed, the Ebborian(s) stays conscious throughout the whole fissioning process. Over time, the brain-paper grows thick enough to fission again.

Electricity flows through Ebborian brains faster than human neurons fire. But the Ebborian brain is constrained by its two-dimensionality. An Ebborian brain-paper must split down its thickness while retaining the integrity of its program. Ebborian evolution took the cheap way out: the brain-paper computes in a purely two-dimensional way. The Ebborians have much faster neuron-equivalents, but they are far less interconnected.

On the whole, Ebborians think faster than humans and remember less. They are less susceptible to habit; they recompute what we would cache. They would be incredulous at the idea that a human neuron might be connected to a thousand neighbors, and equally incredulous at the idea that our axons and dendrites propagate signals at only a few meters per second.

The Ebborians have no concept of parents, children, or sexuality. Every adult Ebborian remembers fissioning many times. But Ebborian memories quickly fade if not used; no one knows the last common ancestor of those now alive.

In principle, an Ebborian personality can be immortal. Yet an Ebborian remembers less life than a seventy-year-old human. They retain only the most important highlights of their last few millennia. Is this immortality? Is it death?

The Ebborians had to rediscover natural selection from scratch, because no one retained their memories of being a fish.

But I digress from my tale.

Today, the Ebborians have gathered to celebrate a day which all present will remember for hundreds of years. They have discovered (they believe) the Ultimate Grand Unified Theory of Everything for their universe. The theory which seems, at last, to explain every known fundamental physical phenomenon - to predict what every instrument will measure, in every experiment whose initial conditions are exactly known, and which can be calculated on available computers.

"But wait!" cries an Ebborian. (We'll call this one Po'mi.) "But wait!", cries Po'mi, "There are still questions the Unified Theory can't answer! During the fission process, when exactly does one Ebborian consciousness become two separate people?"

The gathered Ebborians look at each other. Finally, there speaks the moderator of the gathering, the second-foremost Ebborian on the planet: the much-respected Nharglane of Ebbore, who achieved his position through consistent gentleness and courtesy.

"Well," Nharglane says, "I admit I can't answer that one - but is it really a question of fundamental physics?"

"I wouldn't even call that a 'question'," snorts De'da the Ebborian, "seeing as how there's no experimental test whose result depends on the answer."

"On the contrary," retorts Po'mi, "all our experimental results ultimately come down to our experiences. If a theory of physics can't predict what we'll experience, what good is it?"

De'da shrugs. "One person, two people - how does that make a difference even to experience? How do you tell even internally whether you're one person or two people? Of course, if you look over and see your other self, you know you're finished dividing - but by that time your brain has long since finished splitting."

"Clearly," says Po'mi, "at any given point, whatever is having an experience is one person. So it is never necessary to tell whether you are one person or two people. You are always one person. But at any given time during the split, does there exist another, different consciousness as yet, with its own awareness?"

De'da performs an elaborate quiver, the Ebborian equivalent of waving one's hands. "When the brain splits, it splits fast enough that there isn't much time where the question would be ambiguous. One instant, all the electrical charges are moving as a whole. The next instant, they move separately."

"That's not true," says Po'mi. "You can't sweep the problem under the rug that easily. There is a quite appreciable time - many picoseconds - when the two halves of the brain are within distance for the moving electrical charges in each half to tug on the other. Not quite causally separated, and not quite the same computation either. Certainly there is a time when there is definitely one person, and a time when there is definitely two people. But at which exact point in between are there two distinct conscious experiences?"

"My challenge stands," says De'da. "How does it make a difference, even a difference of first-person experience, as to when you say the split occurs? There's no third-party experiment you can perform to tell you the answer. And no difference of first-person experience, either. Your belief that consciousness must 'split' at some particular point, stems from trying to model consciousness as a big rock of awareness that can only be in one place at a time. There's no third-party experiment, and no first-person experience, that can tell you when you've split; the question is meaningless."

"If experience is meaningless," retorts Po'mi, "then so are all our scientific theories, which are merely intended to explain our experiences."

"If I may," says another Ebborian, named Yu'el, "I think I can refine my honorable colleague Po'mi's dilemma. Suppose that you anesthetized one of us -"

(Ebborians use an anesthetic that effectively shuts off electrical power to the brain - no processing or learning occurs while an Ebborian is anesthetized.)

"- and then flipped a coin. If the coin comes up heads, you split the subject while they are unconscious. If the coin comes up tails, you leave the subject as is. When the subject goes to sleep, should they anticipate a 2/3 probability of seeing the coin come up heads, or anticipate a 1/2 probability of seeing the coin come up heads? If you answer 2/3, then there is a difference of anticipation that could be made to depend on exactly when you split."

"Clearly, then," says De'da, "the answer is 1/2, since answering 2/3 gets us into paradoxical and ill-defined issues."

Yu'el looks thoughtful. "What if we split you into 512 parts while you were anesthetized? Would you still answer a probability of 1/2 for seeing the coin come up heads?"

De'da shrugs. "Certainly. When I went to sleep, I would figure on a 1/2 probability that I wouldn't get split at all."

"Hmm..." Yu'el says. "All right, suppose that we are definitely going to split you into 16 parts. 3 of you will wake up in a red room, 13 of you will wake up in a green room. Do you anticipate a 13/16 probability of waking up in a green room?"

"I anticipate waking up in a green room with near-1 probability," replies De'da, "and I anticipate waking up in a red room with near-1 probability. My future selves will experience both outcomes."

"But I'm asking about your personal anticipation," Yu'el persists. "When you fall asleep, how much do you anticipate seeing a green room? You can't see both room colors at once - that's not an experience anyone will have - so which color do you personally anticipate more?"

De'da shakes his head. "I can see where this is going; you plan to ask what I anticipate in cases where I may or may not be split. But I must deny that your question has an objective answer, precisely because of where it leads. Now, I do say to you, that I care about my future selves. If you ask me whether I would like each of my green-room selves, or each of my red-room selves, to receive ten dollars, I will of course choose the green-roomers - but I don't care to follow this notion of 'personal anticipation' where you are taking it."

"While you are anesthetized," says Yu'el, "I will flip a coin; if the coin comes up heads, I will put 3 of you into red rooms and 13 of you into green rooms. If the coin comes up tails, I will reverse the proportion. If you wake up in a green room, what is your posterior probability that the coin came up heads?"

De'da pauses. "Well..." he says slowly, "Clearly, some of me will be wrong, no matter which reasoning method I use - but if you offer me a bet, I can minimize the number of me who bet poorly, by using the general policy, of each self betting as if the posterior probability of their color dominating is 13/16. And if you try to make that judgment depend on the details of the splitting process, then it just depends on how whoever offers the bet counts Ebborians."

Yu'el nods. "I can see what you are saying, De'da. But I just can't make myself believe it, at least not yet. If there were to be 3 of me waking up in red rooms, and a billion of me waking up in green rooms, I would quite strongly anticipate seeing a green room when I woke up. Just the same way that I anticipate not winning the lottery. And if the proportions of three red to a billion green, followed from a coin coming up heads; but the reverse proportion, of a billion red to three green, followed from tails; and I woke up and saw a red room; why, then, I would be nearly certain - on a quite personal level - that the coin had come up tails."

"That stance exposes you to quite a bit of trouble," notes De'da.

Yu'el nods. "I can even see some of the troubles myself. Suppose you split brains only a short distance apart from each other, so that they could, in principle, be fused back together again? What if there was an Ebborian with a brain thick enough to be split into a million parts, and the parts could then re-unite? Even if it's not biologically possible, we could do it with a computer-based mind, someday. Now, suppose you split me into 500,000 brains who woke up in green rooms, and 3 much thicker brains who woke up in red rooms. I would surely anticipate seeing the green room. But most of me who see the green room will see nearly the same thing - different in tiny details, perhaps, enough to differentiate our experience, but such details are soon forgotten. So now suppose that my 500,000 green selves are reunited into one Ebborian, and my 3 red selves are reunited into one Ebborian. Have I just sent nearly all of my "subjective probability" into the green future self, even though it is now only one of two? With only a little more work, you can see how a temporary expenditure of computing power, or a nicely refined brain-splitter and a dose of anesthesia, would let you have a high subjective probability of winning any lottery. At least any lottery that involved splitting you into pieces."

De'da furrows his eyes. "So have you not just proved your own theory to be nonsense?"

"I'm not sure," says Yu'el. "At this point, I'm not even sure the conclusion is wrong."

"I didn't suggest your conclusion was wrong," says De'da, "I suggested it was nonsense. There's a difference."

"Perhaps," says Yu'el. "Perhaps it will indeed turn out to be nonsense, when I know better. But if so, I don't quite know better yet. I can't quite see how to eliminate the notion of subjective anticipation from my view of the universe. I would need something to replace it, something to re-fill the role that anticipation currently plays in my worldview."

De'da shrugs. "Why not just eliminate 'subjective anticipation' outright?"

"For one thing," says Yu'el, "I would then have no way to express my surprise at the orderliness of the universe. Suppose you claimed that the universe was actually made up entirely of random experiences, brains temporarily coalescing from dust and experiencing all possible sensory data. Then if I don't count individuals, or weigh their existence somehow, that chaotic hypothesis would predict my existence as strongly as does science. The realization of all possible chaotic experiences would predict my own experience with probability 1. I need to keep my surprise at having this particular orderly experience, to justify my anticipation of seeing an orderly future. If I throw away the notion of subjective anticipation, then how do I differentiate the chaotic universe from the orderly one? Presumably there are Yu'els, somewhere in time and space (for the universe is spatially infinite) who are about to have a really chaotic experience. I need some way of saying that these Yu'els are rare, or weigh little - some way of mostly anticipating that I won't sprout wings and fly away. I'm not saying that my current way of doing this is good bookkeeping, or even coherent bookkeeping; but I can't just delete the bookkeeping without a more solid understanding to put in its place. I need some way to say that there are versions of me who see one thing, and versions of me who see something else, but there's some kind of different weight on them. Right now, what I try to do is count copies - but I don't know exactly what constitutes a copy."

Po'mi clears his throat, and speaks again. "So, Yu'el, you agree with me that there exists a definite and factual question as to exactly when there are two conscious experiences, instead of one."

"That, I do not concede," says Yu'el. "All that I have said may only be a recital of my own confusion. You are too quick to fix the language of your beliefs, when there are words in it that, by your own admission, you do not understand. No matter how fundamental your experience feels to you, it is not safe to trust that feeling, until experience is no longer something you are confused about. There is a black box here, a mystery. Anything could be inside that box - any sort of surprise - a shock that shatters everything you currently believe about consciousness. Including upsetting your belief that experience is fundamental. In fact, that strikes me as a surprise you should anticipate - though it will still come as a shock."

"But then," says Po'mi, "do you at least agree that if our physics does not specify which experiences are experienced, or how many of them, or how much they 'weigh', then our physics must be incomplete?"

"No," says Yu'el, "I don't concede that either. Because consider that, even if a physics is known - even if we construct a universe with very simple physics, much simpler than our own Unified Theory - I can still present the same split-brain dilemmas, and they will still seem just as puzzling. This suggests that the source of the confusion is not in our theories of fundamental physics. It is on a higher level of organization. We can't compute exactly how proteins will fold up; but this is not a deficit in our theory of atomic dynamics, it is a deficit of computing power. We don't know what makes sharkras bloom only in spring; but this is not a deficit in our Unified Theory, it is a deficit in our biology - we don't possess the technology to take the sharkras apart on a molecular level to find out how they work. What you are pointing out is a gap in our science of consciousness, which would present us with just the same puzzles even if we knew all the fundamental physics. I see no work here for physicists, at all."

Po'mi smiles faintly at this, and is about to reply, when a listening Ebborian shouts, "What, have you begun to believe in zombies? That when you specify all the physical facts about a universe, there are facts about consciousness left over?"

"No!" says Yu'el. "Of course not! You can know the fundamental physics of a universe, hold all the fundamental equations in your mind, and still not have all the physical facts. You may not know why sharkras bloom in the summer. But if you could actually hold the entire fundamental physical state of the sharkra in your mind, and understand all its levels of organization, then you would necessarily know why it blooms - there would be no fact left over, from outside physics. When I say, 'Imagine running the split-brain experiment in a universe with simple known physics,' you are not concretely imagining that universe, in every detail. You are not actually specifying the entire physical makeup of an Ebborian in your imagination. You are only imagining that you know it. But if you actually knew how to build an entire conscious being from scratch, out of paperclips and rubberbands, you would have a great deal of knowledge that you do not presently have. This is important information that you are missing! Imagining that you have it, does not give you the insights that would follow from really knowing the full physical state of a conscious being."

"So," Yu'el continues, "We can imagine ourselves knowing the fundamental physics, and imagine an Ebborian brain splitting, and find that we don't know exactly when the consciousness has split. Because we are not concretely imagining a complete and detailed description of a conscious being, with full comprehension of the implicit higher levels of organization. There are knowledge gaps here, but they are not gaps of physics. They are gaps in our understanding of consciousness. I see no reason to think that fundamental physics has anything to do with such questions."

"Well then," Po'mi says, "I have a puzzle I should like you to explain, Yu'el. As you know, it was discovered not many years ago, that our universe has four spatial dimensions, rather than three dimensions, as it first appears."

"Aye," says Nharglane of Ebbore, "this was a key part in our working-out of the Unified Theory. Our models would be utterly at a loss to account for observed experimental results, if we could not model the fourth dimension, and differentiate the fourth-dimensional density of materials."

"And we also discovered," continues Po'mi, "that our very planet of Ebbore, including all the people on it, has a four-dimensional thickness, and is constantly fissioning along that thickness, just as our brains do. Only the fissioned sides of our planet do not remain in contact, as our new selves do; the sides separate into the fourth-dimensional void."

Nharglane nods. "Yes, it was rather a surprise to realize that the whole world is duplicated over and over. I shall remember that realization for a long time indeed. It is a good thing we Ebborians had our experience with self-fissioning, to prepare us for the shock. Otherwise we might have been driven mad, and embraced absurd physical theories."

"Well," says Po'mi, "when the world splits down its four-dimensional thickness, it does not always split exactly evenly. Indeed, it is not uncommon to see nine-tenths of the four-dimensional thickness in one side."

"Really?" says Yu'el. "My knowledge of physics is not so great as yours, but -"

"The statement is correct," says the respected Nharglane of Ebbore.

"Now," says Po'mi, "if fundamental physics has nothing to do with consciousness, can you tell me why the subjective probability of finding ourselves in a side of the split world, should be exactly proportional to the square of the thickness of that side?"

There is a great terrible silence.

"WHAT?" says Yu'el.

"WHAT?" says De'da.

"WHAT?" says Nharglane.

"WHAT?" says the entire audience of Ebborians.

To be continued...

Sequence: Quantum Physics

Which Basis Is More Fundamental?

Where Experience Confuses Physicists

Referenced by: Where Experience Confuses Physicists • On Being Decoherent • The Conscious Sorites Paradox • Decoherent Essences • Bell's Theorem: No EPR "Reality" • Many Worlds, One Best Guess • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity • For The People Who Are Still Alive • The Anthropic Trilemma

Original with comments: Where Physics Meets Experience

Where Experience Confuses Physicists

Eliezer Yudkowsky, 26 April 2008 05:05AM

This post is part of the Quantum Physics Sequence.
Continuation of: Where Physics Meets Experience

When we last met our heroes, the Ebborians, they were discussing the known phenomenon in which the entire planet of Ebbore and all its people splits down its fourth-dimensional thickness into two sheets, just like an individual Ebborian brain-sheet splitting along its third dimension.

And Po'mi has just asked:

"Why should the subjective probability of finding ourselves in a side of the split world, be exactly proportional to the square of the thickness of that side?"

When the initial hubbub quiets down, the respected Nharglane of Ebbore asks: "Po'mi, what is it exactly that you found?"

"Using instruments of the type we are all familiar with," Po'mi explains, "I determined when a splitting of the world was about to take place, and in what proportions the world would split. I found that I could not predict exactly which world I would find myself in -"

"Of course not," interrupts De'da, "you found yourself in both worlds, every time -"

"- but I could predict probabilistically which world I would find myself in. Out of all the times the world was about to split 2:1, into a side of two-thirds width and a side of one-third width, I found myself on the thicker side around 4 times out of 5, and on the thinner side around 1 time out of 5. When the world was about to split 3:1, I found myself on the thicker side 9 times out of 10, and on the thinner side 1 time out of 10."

"Are you very sure of this?" asks Nharglane. "How much data did you gather?"

Po'mi offers an overwhelming mountain of experimental evidence.

"I guess that settles that," mutters Nharglane.

"So you see," Po'mi says, "you were right after all, Yu'el, not to eliminate 'subjective probability' from your worldview. For if we do not have a 4/5 subjective anticipation of continuing into the thicker side of a 2:1 split, then how could we even describe this rule?"

"A good question," says De'da. "There ought to be some way of phrasing your discovery, which eliminates this problematic concept of 'subjective continuation'..."

The inimitable Ha'ro speaks up: "You might say that we find ourselves in a world in which the remembered splits obey the squared-thickness rule, to within the limits of statistical expectation."

De'da smiles. "Yes, excellent! That describes the evidence in terms of recorded experimental results, which seems less problematic than this 'subjective anticipation' business."

"Does that really buy us anything...?" murmurs Yu'el. "We're not limited to memories; we could perform the experiment again. What, on that next occasion, would you anticipate as your experimental result? If the thickness is split a hundred to one? Afterward it will be only a memory... but what about beforehand?"

"I think," says De'da, "that you have forgotten one of your own cardinal rules, Yu'el. Surely, what you anticipate is part of your map, not the territory. Your degree of anticipation is partial information you possess; it is not a substance of the experiment itself."

Yu'el pauses. "Aye, that is one of my cardinal rules... but I like my partial information to be about something. Before I can distinguish the map and the territory, I need a concept of the territory. What is my subjective anticipation about, in this case? I will in fact end up in both world-sides. I can calculate a certain probability to five decimal places, and verify it experimentally - but what is it a probability of?"

"I know!" shouts Bo'ma. "It's the probability that your original self ends up on that world-side! The other person is just a copy!"

A great groan goes up from the assembled Ebborians. "Not this again," says De'da. "Didn't we settle this during the Identity Wars?"

"Yes," Yu'el says. "There is no copy: there are two originals."

De'da shakes his head in disgust. "And what are the odds that, out of umpteen billion split Ebbores, we would be the originals at this point?"

"But you can't deny," Bo'ma says smugly, "that my theory produces good experimental predictions! It explains our observations, and that's all you can ask of any theory. And so science vindicates the Army of Original Warriors - we were right all along!"

"Hold on," says Yu'el. "That theory doesn't actually explain anything. At all."

"What?" says Bo'ma. "Of course it does. I use it daily to make experimental predictions; though you might not understand that part, not being a physicist."

Yu'el raises an eye. "Failure to explain anything is a hard-to-notice phenomenon in scientific theories. You have to pay close attention, or you'll miss it. It was once thought that phlogiston theory predicted that wood, when burned, would lose phlogiston and transform into ash; and predicted that candles, burning in an enclosed space, would saturate the air with phlogiston and then go out. But these were not advance predictions of phlogiston theory. Rather, phlogiston theorists saw those results, and then said 'Phlogiston did it.' Now why didn't people notice this right away? Because that sort of thing is actually surprisingly hard to notice."

"In this case," continues Yu'el, "you have given us a rule that the original Ebborian has a probability of ending up in a world-side, which is proportional to the squared thickness of the side. We originally had the mystery of where the squared-thickness rule came from. And now that you've offered us your rule, we have the exact same mystery as before. Why would each world have a squared-thickness probability of receiving the original? Why wouldn't the original consciousness always go to the thicker world? Or go with probability directly proportional to thickness, instead of the square? And what does it even mean to be the original?"

"That doesn't matter," retorts Bo'ma. "Let the equation mean anything it likes, so long as it gives good experimental predictions. What is the meaning of an electrical charge? Why is it an electrical charge? That doesn't matter; only the numbers matter. My law that the original ends up in a particular side, with probability equaling the square of its thickness, gives good numbers. End of story."

Yu'el shakes his head. "When I look over the raw structure of your theory - the computer program that would correspond to this model - it contains a strictly superfluous element. You have to compute the square of the thickness, and turn it into a probability, in order to get the chance that the original self goes there. Why not just keep that probability as the experimental prediction? Why further specify that this is the probability of original-ness? Adding that last rule doesn't help you compute any better experimental predictions; and it leaves all the original mysteries intact. Including Po'mi's question as to when exactly a world splits. And it adds the new mystery of why original-ness should only end up in one world-side, with probability equal to the square of the thickness." Yu'el pauses. "You might as well just claim that all the split world-sides except one vanish from the universe."

Bo'ma snorts. "For a world-side to 'just vanish' would outright violate the laws of physics. Why, if it all vanished in an instant, that would mean the event occurred non-locally - faster than light. My suggestion about 'originals' and 'copies' doesn't postulate unphysical behavior, whatever other criticisms you may have."

Yu'el nods. "You're right, that was unfair of me. I apologize."

"Well," says Bo'ma, "how about this, then? What if 'fourth-dimensional thickness', as we've been calling it, is actually a degree of partial information about who we really are? And then when the world splits, we find out."

"Um... what?" says Yu'el. "Are you sure you don't want to rephrase that, or something?"

Bo'ma shakes his head. "No, you heard me the first time."

"Okay," says Yu'el, "correct me if I'm wrong, but I thought I heard Nharglane say that you had to do things like differentiate the fourth-dimensional density in order to do your experimental calculations. That doesn't sound like probability theory to me. It sounds like physics."

"Right," Bo'ma says, "it's a quantity that propagates around with wave mechanics that involve the differential of the density, but it's also a degree of partial information."

"Look," Yu'el says, "if this 4D density business works the way you say it does, it should be easy to set up a situation where there's no possible 'fact as to who you really are' that is fixed in advance but unknown to you, because the so-called 'answer' will change depending on the so-called 'question' - "

"Okay," Bo'ma says, "forget the 'probability' part. What if 4D thickness is the very stuff of reality itself? So how real something is, equals the 4D thickness - no, pardon me, the square of the 4D thickness. Thus, some world-sides are quantitatively realer than others, and that's why you're more likely to find yourself in them."

"Why," says Yu'el, "is the very stuff of reality itself manifesting as a physical quantity with its own wave mechanics? What's next, electrical charge as a degree of possibility? And besides, doesn't that violate -"

Then Yu'el pauses, and falls silent.

"What is it?" inquires Po'mi.

"I was about to say, wouldn't that violate the Generalized Anti-Zombie Principle," Yu'el replies slowly. "Because then you could have a complete mathematical model of our world, to be looked over by the Flying Spaghetti Monster, and then afterward you would need to tell the Flying Spaghetti Monster an extra postulate: Things are real in proportion to the square of their fourth-dimensional thickness. You could change that postulate, and leave everything microphysically the same, but people would find... different proportions of themselves?... in different places. The difference would be detectable internally... sort of... because the inhabitants would experience the results in different proportions, whatever that means. They would see different things, or at least see the same things in different relative amounts. But any third-party observer, looking over the universe, couldn't tell which internal people were more real, and so couldn't discover the statistics of experience."

De'da laughs. "Sounds like a crushing objection to me."

"Only," says Yu'el, "is that really so different from believing that you can have the whole mathematical structure of a world, and then an extra fact as to whether that world happens to exist or not exist? Shouldn't that be ruled out by the Anti-Zombie Principle too? Shouldn't the Anti-Zombie Principle say that it was logically impossible to have had a world physically identical to our own, except that it doesn't exist? Otherwise there could be an abstract mathematical object structurally identical to this world, but with no experiences in it, because it doesn't exist. And papers that philosophers wrote about subjectivity wouldn't prove they were conscious, because the papers would also 'not exist'."

"Um..." says an Ebborian in the crowd, "correct me if I'm mistaken, but didn't you just solve the mystery of the First Cause?"

"You are mistaken," replies Yu'el. "I can tell when I have solved a mystery, because it stops being mysterious. To cleverly manipulate my own confusion is not to dissolve a problem. It is an interesting argument, and I may try to follow it further - but it's not an answer until the confusion goes away."

"Nonetheless," says Bo'ma, "if you're allowed to say that some worlds exist, and some worlds don't, why not have a degree of existence that's quantitative? And propagates around like a wave, and then we have to square it to get an answer."

Yu'el snorts. "Why not just let the 'degree of existence' be a complex number, while you're at it?"

Bo'ma rolls his eyes. "Please stop mocking me. I can't even imagine any possible experimental evidence which would point in the direction of that conclusion. You'd need a case where two events that were real in opposite directions canceled each other out."

"I'm sorry," says Yu'el, "I need to learn to control my tendency to attack straw opponents. But still, where would the squaring rule come from?"

An Ebborian named Ev'Hu suggests, "Well, you could have a rule that world-sides whose thickness tends toward zero, must have a degree of reality that also tends to zero. And then the rule which says that you square the thickness of a world-side, would let the probability tend toward zero as the world-thickness tended toward zero. QED."

"That's not QED," says Po'mi. "That's a complete non-sequitur. Logical fallacy of affirming the consequent. You could have all sorts of rules that would let the reality tend toward zero as the world-thickness tended toward zero, not just the squaring rule. You could approach the limit from many different directions. And in fact, all our world-sides have a thickness that 'tends toward zero' because they keep splitting. Furthermore, why would an indefinite tendency in the infinite future have any impact on what we do now?"

"The frequentist heresy," says Yu'el. "It sounds like some of their scriptures. But let's move on. Does anyone have any helpful suggestions? Ones that don't just shuffle the mystery around?"

Ha'ro speaks. "I've got one."

"Okay," Yu'el says, "this should be good."

"Suppose that when a world-side gets thin enough," Ha'ro says, "it cracks to pieces and falls apart. And then, when you did the statistics, it would turn out that the vast majority of surviving worlds have splitting histories similar to our own."

There's a certain unsettled pause.

"Ha'ro," says Nharglane of Ebbore, "to the best of my imperfect recollection, that is the most disturbing suggestion any Ebborian physicist has ever made in the history of time."

"Thank you very much," says Ha'ro. "But it could also be that a too-small world-side just sheds off in flakes when it splits, rather than containing actual sentient beings who get to experience a moment of horrified doom. The too-small worlds merely fail to exist, as it were. Or maybe sufficiently small world-sides get attracted to larger world-sides, and merge with them in a continuous process, obliterating the memories of anything having happened differently. But that's not important, the real question is whether the numbers would work out for the right size limit, and in fact," Ha'ro waves some calculations on a piece of paper, "all you need is for the minimum size of a cohesive world to be somewhere around the point where half the total fourth-dimensional mass is above the limit -"

"Eh?" says Yu'el.

"I figured some numbers and they don't look too implausible and we might be able to prove it, either from first-principles of 4D physics showing that a cracking process occurs, or with some kind of really clever experiment," amplifies Ha'ro.

"Sounds promising," says Yu'el. "So if I get what you're saying, there would be a completely physical explanation for why, when a typical bunch of worlds split 2:1, there's around 4 times as many cohesive worlds left that split from the thicker side, as from the thinner side."

"Yes," says Ha'ro, "you just count the surviving worlds."

"And if the Flying Spaghetti Monster ran a simulation of our universe's physics, the simulation would automatically include observers that experienced the same things we did, with the same statistical probabilities," says Yu'el. "No extra postulates required. None of the quantities in the universe would need additional characteristics beyond their strictly physical structure. Running any mathematically equivalent computer program would do the trick - you wouldn't need to be told how to interpret it a particular way."

Ha'ro nods. "That's the general idea."

"Well, I don't know if that's correct," says Yu'el. "There's some potential issues, as you know. But I've got to say it's the first suggestion I've heard that's even remotely helpful in making all this seem any less mysterious."

Sequence: Quantum Physics

Where Physics Meets Experience

On Being Decoherent

Referenced by: The Conscious Sorites Paradox • The Born Probabilities • Bell's Theorem: No EPR "Reality" • Timeless Causality • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Where Experience Confuses Physicists

On Being Decoherent

Eliezer Yudkowsky, 27 April 2008 04:59AM

This post is part of the Quantum Physics Sequence.
Previously in series: The So-Called Heisenberg Uncertainty Principle

"A human researcher only sees a particle in one place at one time." At least that's what everyone goes around repeating to themselves. Personally, I'd say that when a human researcher looks at a quantum computer, they quite clearly see particles not behaving like they're in one place at a time. In fact, you have never in your life seen a particle "in one place at a time" because they aren't.

Nonetheless, when you construct a big measuring instrument that is sensitive to a particle's location - say, the measuring instrument's behavior depends on whether a particle is to the left or right of some dividing line - then you, the human researcher, see the screen flashing "LEFT", or "RIGHT", but not a mixture like "LIGFT".

As you might have guessed from reading about decoherence and Heisenberg, this is because we ourselves are governed by the laws of quantum mechanics and subject to decoherence.

The standpoint of the Feynman path integral suggests viewing the evolution of a quantum system as a sum over histories, an integral over ways the system "could" behave - though the quantum evolution of each history still depends on things like the second derivative of that component of the amplitude distribution; it's not a sum over classical histories. And "could" does not mean possibility in the logical sense; all the amplitude flows are real events...

Nonetheless, a human being can try to grasp a quantum system by imagining all the ways that something could happen, and then adding up all the little arrows that flow to identical outcomes. That gets you something of the flavor of the real quantum physics, of amplitude flows between volumes of configuration space.

Now apply this mode of visualization to a sensor measuring an atom - say, a sensor measuring whether an atom is to the left or right of a dividing line.

Which is to say: The sensor and the atom undergo some physical interaction in which the final state of the sensor depends heavily on whether the atom is to the left or right of a dividing line. (I am reusing some previous diagrams, so this is not an exact depiction; but you should be able to use your own imagination at this point.)

You may recognize this as the entangling interaction described in "Decoherence". A quantum system that starts out highly factorizable, looking plaid and rectangular, that is, independent, can evolve into an entangled system as the formerly-independent parts interact among themselves.

So you end up with an amplitude distribution that contains two blobs of amplitude - a blob of amplitude with the atom on the left, and the sensor saying "LEFT"; and a blob of amplitude with the atom on the right, and the sensor saying "RIGHT".

For a sensor to measure an atom is to become entangled with it - for the state of the sensor to depend on the state of the atom - for the two to become correlated. In a classical system, this is true only on a probabilistic level. In quantum physics it is a physically real state of affairs.

To observe a thing is to entangle yourself with it. You may recall my having previously said things that sound a good deal like this, in describing how cognition obeys the laws of thermodynamics, and, much earlier, talking about how rationality is a phenomenon within causality. It is possible to appreciate this in a purely philosophical sense, but quantum physics helps drive the point home.

Let's say you've got an Atom, whose position has amplitude bulges on the left and on the right. We can regard the Atom's distribution as a sum (addition, not multiplication) of the left bulge and the right bulge:

Atom = (Atom-LEFT + Atom-RIGHT)

Also there's a Sensor in a ready-to-sense state, which we'll call BLANK:

Sensor = Sensor-BLANK

By hypothesis, the system starts out in a state of quantum independence - the Sensor hasn't interacted with the Atom yet. So:

System = (Sensor-BLANK) * (Atom-LEFT + Atom-RIGHT)

Sensor-BLANK is an amplitude sub-distribution, or sub-factor, over the joint positions of all the particles in the sensor. Then you multiply this distribution by the distribution (Atom-LEFT + Atom-RIGHT), which is the sub-factor for the Atom's position. Which gets you the joint configuration space over all the particles in the system, the Sensor and the Atom.

Quantum evolution is linear, which means that Evolution(A + B) = Evolution(A) + Evolution(B). We can understand the behavior of this whole distribution by understanding its parts. Not its multiplicative factors, but its additive components. So now we use the distributive rule of arithmetic, which, because we're just adding and multiplying complex numbers, works just as usual:

System = (Sensor-BLANK) * (Atom-LEFT + Atom-RIGHT)
= (Sensor-BLANK * Atom-LEFT) + (Sensor-BLANK * Atom-RIGHT)

Now, the volume of configuration space corresponding to (Sensor-BLANK * Atom-LEFT) evolves into (Sensor-LEFT * Atom-LEFT).

Which is to say: Particle positions for the sensor being in its initialized state and the Atom being on the left, end up sending their amplitude flows to final configurations in which the Sensor is in a LEFT state, and the Atom is still on the left.

So we have the evolution:

(Sensor-BLANK * Atom-LEFT) + (Sensor-BLANK * Atom-RIGHT)
=>
(Sensor-LEFT * Atom-LEFT) + (Sensor-RIGHT * Atom-RIGHT)

By hypothesis, Sensor-LEFT is a different state from Sensor-RIGHT - otherwise it wouldn't be a very sensitive Sensor. So the final state doesn't factorize any further; it's entangled.

But this entanglement is not likely to manifest in difficulties of calculation. Suppose the Sensor has a little LCD screen that's flashing "LEFT" or "RIGHT". This may seem like a relatively small difference to a human, but it involves avogadros of particles - photons, electrons, entire molecules - occupying different positions.

So, since the states Sensor-LEFT and Sensor-RIGHT are widely separated in the configuration space, the volumes (Sensor-LEFT * Atom-LEFT) and (Sensor-RIGHT * Atom-RIGHT) are even more widely separated.

The LEFT blob and the RIGHT blob in the amplitude distribution can be considered separately; they won't interact. There are no plausible Feynman paths that end up with both LEFT and RIGHT sending amplitude to the same joint configuration. There would have to be a Feynman path from LEFT, and a Feynman path from RIGHT, in which all the quadrillions of differentiated particles ended up in the same places. So the amplitude flows from LEFT and RIGHT don't intersect, and don't interfere.

You may recall this principle from "Decoherence", for how a sensitive interaction can decohere two interacting blobs of amplitude, into two noninteracting blobs.

Formerly, the Atom-LEFT and Atom-RIGHT states were close enough in configuration space, that the blobs could interact with each other - there would be Feynman paths where an atom on the left ended up on the right. Or Feynman paths for both an atom on the left, and an atom on the right, to end up in the middle.

Now, however, the two blobs are decohered. For LEFT to interact with RIGHT, it's not enough for just the Atom to end up on the right. The Sensor would have to spontaneously leap into a state where it was flashing "RIGHT" on screen. Likewise with any particles in the environment which previously happened to be hit by photons for the screen flashing "LEFT". Trying to reverse decoherence is like trying to unscramble an egg.

And when a human being looks at the Sensor's little display screen... or even just stands nearby, with quintillions of particles slightly influenced by gravity... then, under exactly the same laws, the system evolves into:

(Human-LEFT * Sensor-LEFT * Atom-LEFT) + (Human-RIGHT * Sensor-RIGHT * Atom-RIGHT)

Thus, any particular version of yourself only sees the sensor registering one result.

That's it - the big secret of quantum mechanics. As physical secrets go, it's actually pretty damn big. Discovering that the Earth was not the center of the universe, doesn't hold a candle to realizing that you're twins.

That you, yourself, are made of particles, is the fourth and final key to recovering the classical hallucination. It's why you only ever see the universe from within one blob of amplitude, and not the vastly entangled whole.

Asking why you can't see Schrodinger's Cat as simultaneously dead and alive, is like an Ebborian asking: "But if my brain really splits down the middle, why do I only ever remember finding myself on either the left or the right? Why don't I find myself on both sides?"

Because you're not outside and above the universe, looking down. You're in the universe.

Your eyes are not an empty window onto the soul, through which the true state of the universe leaks in to your mind. What you see, you see because your brain represents it: because your brain becomes entangled with it: because your eyes and brain are part of a continuous physics with the rest of reality.

You only see nearby objects, not objects light-years away, because photons from those objects can't reach you, therefore you can't see them. By a similar locality principle, you don't interact with distant configurations.

When you open your eyes and see your shoelace is untied, that event happens within your brain. A brain is made up of interacting neurons. If you had two separate groups of neurons that never interacted with each other, but did interact among themselves, they would not be a single computer. If one group of neurons thought "My shoelace is untied", and the other group of neurons thought "My shoelace is tied", it's difficult to see how these two brains could possibly contain the same consciousness.

And if you think all this sounds obvious, note that, historically speaking, it took more than two decades after the invention of quantum mechanics for a physicist to publish that little suggestion. People really aren't used to thinking of themselves as particles.

The Ebborians have it a bit easier, when they split. They can see the other sides of themselves, and talk to them.

But the only way for two widely separated blobs of amplitude to communicate - to have causal dependencies on each other - would be if there were at least some Feynman paths leading to identical configurations from both starting blobs.

Once one entire human brain thinks "Left!", and another entire human brain thinks "Right!", then it's extremely unlikely for all of the particles in those brains, and all of the particles in the sensors, and all of the nearby particles that interacted, to coincidentally end up in approximately the same configuration again.

It's around the same likelihood as your brain spontaneously erasing its memories of seeing the sensor and going back to its exact original state; while nearby, an egg unscrambles itself and a hamburger turns back into a cow.

So the decohered amplitude-blobs don't interact. And we never get to talk to our other selves, nor can they speak to us.

Of course, this doesn't mean that the other amplitude-blobs aren't there any more, any more than we should think that a spaceship suddenly ceases to exist when it travels over the cosmological horizon (relative to us) of an expanding universe.

(Oh, you thought that post on belief in the implied invisible was part of the Zombie sequence? No, that was covert preparation for the coming series on quantum mechanics.

You can go through line by line and substitute the arguments, in fact.

Remember that the next time some commenter snorts and says, "But what do all these posts have to do with your Artificial Intelligence work?")

Disturbed by the prospect of there being more than one version of you? But as Max Tegmark points out, living in a spatially infinite universe already implies that an exact duplicate of you exists somewhere, with probability 1. In all likelihood, that duplicate is no more than 10^(10^29) lightyears away. Or 10^(10^29) meters away, with numbers of that magnitude it's pretty much the same thing.

(Stop the presses! Shocking news! Scientists have announced that you are actually the duplicate of yourself 10^(10^29) lightyears away! What you thought was "you" is really just a duplicate of you.)

You also get the same Big World effect from the inflationary scenario in the Big Bang, which buds off multiple universes. And both spatial infinity and inflation are more or less standard in the current model of physics. So living in a Big World, which contains more than one person who resembles you, is a bullet you've pretty much got to bite - though none of the guns are certain, physics is firing that bullet at you from at least three different directions.

Maybe later I'll do a post about why you shouldn't panic about the Big World. You shouldn't be drawing many epistemic implications from it, let alone moral implications. As Greg Egan put it, "It all adds up to normality." Indeed, I sometimes think of this as Egan's Law.

Sequence: Quantum Physics

Where Experience Confuses Physicists

The Conscious Sorites Paradox

Referenced by: The Conscious Sorites Paradox • Decoherence is Pointless • The Born Probabilities • Decoherence as Projection • Bell's Theorem: No EPR "Reality" • Spooky Action at a Distance: The No-Communication Theorem • Collapse Postulates • The Dilemma: Science or Bayes? • Timeless Physics • Timeless Identity • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: On Being Decoherent

The Conscious Sorites Paradox

Eliezer Yudkowsky, 28 April 2008 02:58AM

This post is part of the Quantum Physics Sequence.
Followup to: On Being Decoherent

Decoherence is implicit in quantum physics, not an extra postulate on top of it, and quantum physics is continuous. Thus, "decoherence" is not an all-or-nothing phenomenon - there's no sharp cutoff point. Given two blobs, there's a quantitative amount of amplitude that can flow into identical configurations between them. This quantum interference diminishes down to an exponentially tiny infinitesimal as the two blobs separate in configuration space.

Asking exactly when decoherence takes place, in this continuous process, is like asking when, if you keep removing grains of sand from a pile, it stops being a "heap".

The sand-heap dilemma is known as the Sorites Paradox, after the Greek soros, for heap. It is attributed to Eubulides of Miletus, in the 4th century BCE. The moral I draw from this very ancient tale: If you try to draw sharp lines in a continuous process and you end up looking silly, it's your own darn fault.

(Incidentally, I once posed the Sorites Paradox to Marcello Herreshoff, who hadn't previously heard of it; and Marcello answered without the slightest hesitation, "If you remove all the sand, what's left is a 'heap of zero grains'." Now that's a computer scientist.)

Ah, but what about when people become decoherent? What of the Conscious Sorites Paradox?

What about the case where two blobs of amplitude containing people are interacting, but only somewhat - so that there is visibly a degree of causal influence, and visibly a degree of causal independence?

Okay, this interval may work out to less than the Planck time for objects the size of a human brain. But I see that as no excuse to evade the question. In principle we could build a brain that would make the interval longer.

Shouldn't there be some definite fact of the matter as to when one person becomes two people?

Some folks out there would just say "No". I suspect Daniel Dennett would just say "No". Personally, I wish I could just say "No", but I'm not that advanced yet. I haven't yet devised a way to express my appreciation of the orderliness of the universe, which doesn't involve counting people in orderly states as compared to disorderly states.

Yet if you insist on an objective population count, for whatever reason, you have Soritic problems whether or not you delve into quantum physics.

What about the Ebborians? The Ebborians, you recall, have brains like flat sheets of conducting polymer, and when they reproduce, the brain-sheet splits down its thickness. In the beginning, there is definitely one brain; in the end, there is definitely two brains; in between, there is a continuous decrease of causal influence and synchronization. When does one Ebborian become two?

Those who insist on an objective population count in a decoherent universe, must confront exactly analogous people-splitting problems in classical physics!

Heck, you could simulate quantum physics the way we currently think it works, and ask exactly the same question! At the beginning there is one blob, at the end there are two blobs, in this universe we have constructed. So when does the consciousness split, if you think there's an objective answer to that?

Demanding an objective population count is not a reason to object to decoherence, as such. Indeed, the last fellow I argued with, ended up agreeing that his objection to decoherence was in fact a fully general objection to functionalist theories of consciousness.

You might be tempted to try sweeping the Conscious Sorites Paradox under a rug, by postulating additionally that the Quantum Spaghetti Monster eats certain blobs of amplitude at exactly the right time to avoid a split.

But then (1) you have to explain exactly when the QSM eats the amplitude, so you aren't avoiding any burden of specification.

And (2) you're requiring the Conscious Sorites Paradox to get answered by fundamental physics, rather than being answered or dissolved by a better understanding of consciousness. It's hard to see why taking this stance advances your position, rather than just closing doors.

In fact (3) if you think you have a definite answer to "When are there two people?", then it's hard to see why you can't just give that same answer within the standard quantum theory instead. The Quantum Spaghetti Monster isn't really helping here! For every definite theory with a QSM, there's an equally definite theory with no QSM. This is one of those occasions you have to pay close attention to see the superfluous element of your theory that doesn't really explain anything - it's harder when the theory as a whole does explain something, as quantum physics certainly does.

Above all, (4) you would still have to explain afterward what happens with the Ebborians, or what happens to decoherent people in a simulation of quantum physics the way we currently think it works. So you really aren't avoiding any questions!

It's also worth noting that, in any physics that is continuous (or even any physics that has a very fine-grained discrete cellular level underneath), there are further Conscious Sorites Parodoxes for when people are born and when they die. The bullet plows into your brain, crushing one neuron after another - when exactly are there zero people instead of one?

Does it still seem like the Conscious Sorites Paradox is an objection to decoherent quantum mechanics, in particular?

A reductionist would say that the Conscious Sorites Paradox is not a puzzle for physicists, because it is a puzzle you get even after the physicists have done their duty, and told us the true laws governing every fundamental event.

As previously touched on, this doesn't imply that consciousness is a matter of nonphysical knowledge. You can know the fundamental laws, and yet lack the computing power to do protein folding. So, too, you can know the fundamental laws; and yet lack the empirical knowledge of the brain's configuration, or miss the insight into higher levels of organization, which would give you a compressed understanding of consciousness.

Or so a materialist would assume. A non-epiphenomenal dualist would say, "Ah, but you don't know the true laws of fundamental physics, and when you do know them, that is where you will find the thundering insight that also resolves questions of consciousness and identity."

It's because I actually do acknowledge the possibility that there is some thundering insight in the fundamental physics we don't know yet, that I am not quite willing to say that the Conscious Sorites puzzle is not a puzzle for physicists. Or to look at it another way, the problem might not be their responsibility, but that doesn't mean they can't help. The physicists might even swoop in and solve it, you never know.

In one sense, there's a clear gap in our interpretation of decoherence: we don't know exactly how quantum-mechanical states correspond to the experiences that are (from a Cartesian standpoint) our final experimental results.

But this is something you could say about all current scientific theories (at least that I've heard of). And I, for one, am betting that the puzzle-cracking insight comes from a cognitive scientist.

I'm not just saying tu quoque (i.e., "Your theory has that problem too!") I'm saying that "But you haven't explained consciousness!" doesn't reasonably seem like the responsibility of physicists, or an objection to a theory of fundamental physics.

An analogy: When a doctor says, "Hey, I think that virus X97 is causing people to drip green slime," you don't respond: "Aha, but you haven't explained the exact chain of causality whereby this merely physical virus leads to my experience of dripping green slime... so it's probably not a virus that does it, but a bacterium!"

This is another of those sleights-of-hand that you have to pay close attention to notice. Why does a non-viral theory do any better than a viral theory at explaining which biological states correspond to which conscious experiences? There is a puzzle here, but how is it a puzzle that provides evidence for one epidemiological theory over another?

It can reasonably seem that, however consciousness turns out to work, getting infected with virus X97 eventually causes your experience of dripping green slime. You've solved the medical part of the problem, as it were, and the remaining mystery is a matter for cognitive science.

Likewise, when a physicist has said that two objects attract each other with a force that goes as the product of the masses and the inverse square of the distance between them, that looks pretty much consistent with the experience of an apple falling on your head. If you have an experience of the apple floating off into space, that's a problem for the physicist. But that you have any experience at all, is not a problem for that particular theory of gravity.

If two blobs of amplitude are no longer interacting, it seems reasonable to regard this as consistent with there being two different brains that have two different experiences, however consciousness turns out to work. Decoherence has a pretty reasonable explanation of why you experience a single world rather than an entangled one, given that you experience anything at all.

However the whole debate over consciousness turns out, it seems that we see pretty much what we should expect to see given decoherent physics. What's left is a puzzle, but it's not a physicist's responsibility to answer.

...is what I would like to say.

But unfortunately there's that whole thing with the squared modulus of the complex amplitude giving the apparent "probability" of "finding ourselves in a particular blob".

That part is a serious puzzle with no obvious answer, which I've discussed already in analogy. I'll shortly be doing an explanation of how the problem looks from within actual quantum theory.

Just remember, if someone presents you with an apparent "answer" to this puzzle, don't forget to check whether the phenomenon still seems mysterious, whether the answer really explains anything, and whether every part of the hypothesis is actively helping.

Sequence: Quantum Physics

On Being Decoherent

Decoherence is Pointless

Referenced by: Decoherence is Pointless • Decoherent Essences • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity • For The People Who Are Still Alive

Original with comments: The Conscious Sorites Paradox

Decoherence is Pointless

Eliezer Yudkowsky, 29 April 2008 06:38AM

This post is part of the Quantum Physics Sequence.
Previously in series: On Being Decoherent

Yesterday's post argued that continuity of decoherence is no bar to accepting it as an explanation for our experienced universe, insofar as it is a physicist's responsibility to explain it. This is a good thing, because the equations say decoherence is continuous, and the equations get the final word.

Now let us consider the continuity of decoherence in greater detail...

On Being Decoherent talked about the decoherence process,

(Human-BLANK) * (Sensor-BLANK) * (Atom-LEFT + Atom-RIGHT)
=>
(Human-BLANK) * ((Sensor-LEFT * Atom-LEFT) + (Sensor-RIGHT * Atom-RIGHT))
=>
(Human-LEFT * Sensor-LEFT * Atom-LEFT) + (Human-RIGHT * Sensor-RIGHT * Atom-RIGHT)

At the end of this process, it may be that your brain in LEFT and your brain in RIGHT are, in a technical sense, communicating - that they have intersecting, interfering amplitude flows.

But the amplitude involved in this process, is the amplitude for a brain (plus all entangled particles) to leap into the other brain's state. This influence may, in a quantitative sense, exist; but it's exponentially tinier than the gravitational influence upon your brain of a mouse sneezing on Pluto.

By the same token, decoherence always entangles you with a blob of amplitude density, not a point mass of amplitude. A point mass of amplitude would be a discontinuous amplitude distribution, hence unphysical. The distribution can be very narrow, very sharp - even exponentially narrow - but it can't actually be pointed (nondifferentiable), let alone a point mass.

Decoherence, you might say, is pointless.

If a measuring instrument is sensitive enough to distinguish 10 positions with 10 separate displays on a little LCD screen, it will decohere the amplitude into at least 10 parts, almost entirely noninteracting. In all probability, the instrument is physically quite a bit more sensitive (in terms of evolving into different configurations) than what it shows on screen. You would find experimentally that the particle was being decohered (with consequences for momentum, etc.) more than the instrument was designed to measure from a human standpoint.

But there is no such thing as infinite sensitivity in a continuous quantum physics: If you start with blobs of amplitude density, you don't end up with point masses. Liouville's Theorem, which generalizes the second law of thermodynamics, guarantees this: you can't compress probability.

What about if you measure the position of an Atom using an analog Sensor whose dial shows a continuous reading?

Think of probability theory over classical physics:

When the Sensor's dial appears in a particular position, that gives us evidence corresponding to the likelihood function for the Sensor's dial to be in that place, given that the Atom was originally in a particular position. If the instrument is not infinitely sensitive (which it can't be, for numerous reasons), then the likelihood function will be a density distribution, not a point mass. A very sensitive Sensor might have a sharp spike of a likelihood distribution, with density falling off rapidly. If the Atom is really at position 5.0121, the likelihood of the Sensor's dial ending up in position 5.0123 might be very small. And so, unless we had overwhelming prior knowledge, we'd conclude a tiny posterior probability that the Atom was so much as 0.0002 millimeters from the Sensor's indicated position. That's probability theory over classical physics.

Similarly in quantum physics:

The blob of amplitude in which you find yourself, where you see the Sensor's dial in some particular position, will have a sub-distribution over actual Atom positions that falls off according to (1) the initial amplitude distribution for the Atom, analogous to the prior; and (2) the amplitude for the Sensor's dial (and the rest of the Sensor!) to end up in our part of configuration space, if the Atom started out in that position. (That's the part analogous to the likelihood function.) With a Sensor at all sensitive, the amplitude for the Atom to be in a state noticeably different from what the Sensor shows, will taper off very sharply.

(All these amplitudes I'm talking about are actually densities, N-dimensional integrals over dx dy dz..., rather than discrete flows between discrete states; but you get the idea.)

If there's not a lot of amplitude flowing from initial particle position 5.0150 +/- 0.0001 to configurations where the sensor's LED display reads '5.0123', then the joint configuration of (Sensor=5.0123 * Atom=5.0150) ends up with very tiny amplitude.

Sequence: Quantum Physics

The Conscious Sorites Paradox

Decoherent Essences

Referenced by: Decoherent Essences • The Born Probabilities • Collapse Postulates • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • And the Winner is... Many-Worlds!

Original with comments: Decoherence is Pointless

Decoherent Essences

Eliezer Yudkowsky, 30 April 2008 06:32AM

This post is part of the Quantum Physics Sequence.
Followup to: Decoherence is Pointless

In "Decoherence is Pointless", we talked about quantum states such as

(Human-BLANK) * ((Sensor-LEFT * Atom-LEFT) + (Sensor-RIGHT * Atom-RIGHT))

which describes the evolution of a quantum system just after a sensor has measured an atom, and right before a human has looked at the sensor - or before the human has interacted gravitationally with the sensor, for that matter. (It doesn't take much interaction to decohere objects the size of a human.)

But this is only one way of looking at the amplitude distribution - a way that makes it easy to see objects like humans, sensors, and atoms. There are other ways of looking at this amplitude distribution - different choices of basis - that will make the decoherence less obvious.

Suppose that you have the "entangled" (non-independent) state:

(Sensor-LEFT * Atom-LEFT) + (Sensor-RIGHT * Atom-RIGHT)

considering now only the sensor and the atom.

This state looks nicely diagonalized - separated into two distinct blobs. But by linearity, we can take apart a quantum amplitude distribution any way we like, and get the same laws of physics back out. So in a different basis, we might end up writing (Sensor-LEFT * Atom-LEFT) as:

(0.5(Sensor-LEFT + Sensor-RIGHT) + 0.5(Sensor-LEFT - Sensor-RIGHT)) * (0.5(Atom-RIGHT + Atom-LEFT) - 0.5(Atom-RIGHT - Atom-LEFT))

(Don't laugh. There are legitimate reasons for physicists to reformulate their quantum representations in weird ways.)

The result works out the same, of course. But if you view the entangled state in a basis made up of linearly independent components like (Sensor-LEFT - Sensor-RIGHT) and (Atom-RIGHT - Atom-LEFT), you see a differently shaped amplitude distribution, and it may not look like the blobs are separated.

Oh noes! The decoherence has disappeared!

...or that's the source of a huge academic literature asking, "Doesn't the decoherence interpretation require us to choose a preferred basis?"

To which the short answer is: Choosing a basis is an isomorphism; it doesn't change any experimental predictions. Decoherence is an experimentally visible phenomenon or we would not have to protect quantum computers from it. You can't protect a quantum computer by "choosing the right basis" instead of using environmental shielding. Likewise, looking at splitting humans from another angle won't make their decoherence go away.

But this is an issue that you're bound to encounter if you pursue quantum mechanics, especially if you talk to anyone from the Old School, and so it may be worth expanding on this reply.

After all, if the short answer is as obvious as I've made it sound, then why, oh why, would anyone ever think you could eliminate an experimentally visible phenomenon like decoherence, by isomorphically reformulating the mathematical representation of quantum physics?

That's a bit difficult to describe in one mere blog post. It has to do with history. You know the warning I gave about dragging history into explanations of QM... so consider yourself warned: Quantum mechanics is simpler than the arguments we have about quantum mechanics. But here, then, is the history:

Once upon a time,

Long ago and far away, back when the theory of quantum mechanics was first being developed,

No one had ever thought of decoherence. The question of why a human researcher only saw one thing at a time, was a Great Mystery with no obvious answer.

You had to interpret quantum mechanics to get an answer back out of it. Like reading meanings into an oracle. And there were different, competing interpretations. In one popular interpretation, when you "measured" a system, the Quantum Spaghetti Monster would eat all but one blob of amplitude, at some unspecified time that was exactly right to give you whatever experimental result you actually saw.

Needless to say, this "interpretation" wasn't in the quantum equations. You had to add in the extra postulate of a Quantum Spaghetti Monster on top, additionally to the differential equations you had fixed experimentally for describing how an amplitude distribution evolved.

Along came Hugh Everett and said, "Hey, maybe the formalism just describes the way the universe is, without any need to 'interpret' it."

But people were so used to adding extra postulates to interpret quantum mechanics, and so unused to the idea of amplitude distributions as real, that they couldn't see this new "interpretation" as anything except an additional Decoherence Postulate which said:

"When clouds of amplitude become separated enough, the Quantum Spaghetti Monster steps in and creates a new world corresponding to each cloud of amplitude."

So then they asked:

"Exactly how separated do two clouds of amplitude have to be, quantitatively speaking, in order to invoke the instantaneous action of the Quantum Spaghetti Monster? And in which basis does the Quantum Spaghetti Monster measure separation?"

But, in the modern view of quantum mechanics - which is accepted by everyone except for a handful of old fogeys who may or may not still constitute a numerical majority - well, as David Wallace puts it:

"If I were to pick one theme as central to the tangled development of the Everett interpretation of quantum mechanics, it would probably be: the formalism is to be left alone."

Decoherence is not an extra phenomenon. Decoherence is not something that has to be proposed additionally. There is no Decoherence Postulate on top of standard QM. It is implicit in the standard rules. Decoherence is just what happens by default, given the standard quantum equations, unless the Quantum Spaghetti Monster intervenes.

Some still claim that the quantum equations are unreal - a mere model that just happens to give amazingly good experimental predictions. But then decoherence is what happens to the particles in the "unreal model", if you apply the rules universally and uniformly. It is denying decoherence that requires you to postulate an extra law of physics, or an act of the Quantum Spaghetti Monster.

(Needless to say, no one has ever observed a quantum system behaving coherently, when the untouched equations say it should be decoherent; nor observed a quantum system behaving decoherently, when the untouched equations say it should be coherent.)

If you're talking about anything that isn't in the equations, you must not be talking about "decoherence". The standard equations of QM, uninterpreted, do not talk about a Quantum Spaghetti Monster creating new worlds. So if you ask when the Quantum Spaghetti Monster creates a new world, and you can't answer the question just by looking at the equations, then you must not be talking about "decoherence". QED.

Which basis you use in your calculations makes no difference to standard QM. "Decoherence" is a phenomenon implicit in standard QM. Which basis you use makes no difference to "decoherence". QED.

Changing your view of the configuration space can change your view of the blobs of amplitude, but ultimately the same physical events happen for the same causal reasons. Momentum basis, position basis, position basis with a different relativistic space of simultaneity - it doesn't matter to QM, ergo it doesn't matter to decoherence.

If this were not so, you could do an experiment to find out which basis was the right one! Decoherence is an experimentally visible phenomenon - that's why we have to protect quantum computers from it.

Ah, but then where is the decoherence in

(0.5(Sensor-LEFT + Sensor-RIGHT) + 0.5(Sensor-LEFT - Sensor-RIGHT)) * (0.5(Atom-RIGHT + Atom-LEFT) - 0.5(Atom-RIGHT - Atom-LEFT)) + (0.5(Sensor-LEFT + Sensor-RIGHT) - 0.5(Sensor-LEFT - Sensor-RIGHT)) * (0.5(Atom-RIGHT + Atom-LEFT) + 0.5(Atom-RIGHT - Atom-LEFT))

?

The decoherence is still there. We've just made it harder for a human to see, in the new representation.

The main interesting fact I would point to, about this amazing new representation, is that we can no longer calculate its evolution with local causality. For a technical definition of what I mean by "causality" or "local", see Judea Pearl's Causality. Roughly, to compute the evolution of an amplitude cloud in a locally causal basis, each point in configuration space only has to look at its infinitesimal neighborhood to determine its instantaneous change. As I understand quantum physics - I pray to some physicist to correct me if I'm wrong - the position basis is local in this sense.

(Note: It's okay to pray to physicists, because physicists actually exist and can answer prayers.)

However, once you start breaking down the amplitude distribution into components like (Sensor-RIGHT - Sensor-LEFT), then the flow of amplitude, and the flow of causality, is no longer local within the new configuration space. You can still calculate it, but you have to use nonlocal calculations.

In essence, you've obscured the chessboard by subtracting the queen's position from the king's position. All the information is still there, but it's harder to see.

When it comes to talking about whether "decoherence" has occurred in the quantum state of a human brain, what should intuitively matter is questions like, "Does the event of a neuron firing in Human-LEFT have a noticeable influence on whether a corresponding neuron fires in Human-RIGHT?" You can choose a basis that will mix up the amplitude for Human-LEFT and Human-RIGHT, in your calculations. You cannot, however, choose a basis that makes a human neuron fire when it would not otherwise have fired; any more than you can choose a basis that will protect a quantum computer without the trouble of shielding, or choose a basis that will make apples fall upward instead of down, etcetera.

The formalism is to be left alone! If you're talking about anything that isn't in the equations, you're not talking about decoherence! Decoherence is part of the invariant essence that doesn't change no matter how you spin your basis - just like the physical reality of apples and quantum computers and brains.

There may be a kind of Mind Projection Fallacy at work here. A tendency to see the basis itself as real - something that a Quantum Spaghetti Monster might come in and act upon - because you spend so much time calculating with it.

In a strange way, I think, this sort of jump is actively encouraged by the Old School idea that the amplitude distributions aren't real. If you were told the amplitude distributions were physically real, you would (hopefully) get in the habit of looking past mere representations, to see through to some invariant essence inside - a reality that doesn't change no matter how you choose to represent it.

But people are told the amplitude distribution is not real. The calculation itself is all there is, and has no virtue save its mysteriously excellent experimental predictions. And so there is no point in trying to see through the calculations to something within.

Then why not interpret all this talk of "decoherence" in terms of an arbitrarily chosen basis? Isn't that all there is to interpret - the calculation that you did in some representation or another? Why not complain, if - having thus interpreted decoherence - the separatedness of amplitude blobs seems to change, when you change the basis? Why try to see through to the neurons, or the flows of causality, when you've been told that the calculations are all?

(This notion of seeing through - looking for an essence, and not being distracted by surfaces - is one that pops up again and again, and again and again and again, in the Way of Rationality.)

Another possible problem is that the calculations are crisp, but the essences inside them are not. Write out an integral, and the symbols are digitally distinct. But an entire apple, or an entire brain, is larger than anything you can handle formally.

Yet the form of that crisp integral will change when you change your basis; and that sloppy real essence will remain invariant. Reformulating your equations won't remove a dagger, or silence a firing neuron, or shield a quantum computer from decoherence.

The phenomenon of decoherence within brains and sensors, may not be any more crisply defined than the brains and sensors themselves. Brains, as high-level phenomena, don't always make a clear appearance in fundamental equations. Apples aren't crisp, you might say.

For historical reasons, some Old School physicists are accustomed to QM being "interpreted" using extra postulates that involve crisp actions by the Quantum Spaghetti Monster - eating blobs of amplitude at a particular instant, or creating worlds as a particular instant. Since the equations aren't supposed to be real, the sloppy borders of real things are not looked for, and the crisp calculations are primary. This makes it hard to see through to a real (but uncrisp) phenomenon among real (but uncrisp) brains and apples, invariant under changes of crisp (but arbitrary) representation.

Likewise, any change of representation that makes apples harder to see, or brains harder to see, will make decoherence within brains harder to see. But it won't change the apple, the brain, or the decoherence.

As always, any philosophical problems that result from "brain" or "person" or "consciousness" not being crisply defined, are not the responsibility of physicists or of any fundamental physical theory. Nor are they limited to decoherent quantum physics particularly, appearing likewise in splitting brains constructed under classical physics, etcetera.

Coming tomorrow (hopefully): The Born Probabilities, aka, that mysterious thing we do with the squared modulus to get our experimental predictions.

Sequence: Quantum Physics

Decoherence is Pointless

The Born Probabilities

Referenced by: The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity

Original with comments: Decoherent Essences

The Born Probabilities

Eliezer Yudkowsky, 01 May 2008 05:50AM

This post is part of the Quantum Physics Sequence.
Previously in series: Decoherence is Pointless
Followup to: Where Experience Confuses Physicists

One serious mystery of decoherence is where the Born probabilities come from, or even what they are probabilities of. What does the integral over the squared modulus of the amplitude density have to do with anything?

This was discussed by analogy in "Where Experience Confuses Physicists", and I won't repeat arguments already covered there. I will, however, try to convey exactly what the puzzle is, in the real framework of quantum mechanics.

A professor teaching undergraduates might say: "The probability of finding a particle in a particular position is given by the squared modulus of the amplitude at that position."

This is oversimplified in several ways.

First, for continuous variables like position, amplitude is a density, not a point mass. You integrate over it. The integral over a single point is zero.

(Historical note: If "observing a particle's position" invoked a mysterious event that squeezed the amplitude distribution down to a delta point, or flattened it in one subspace, this would give us a different future amplitude distribution from what decoherence would predict. All interpretations of QM that involve quantum systems jumping into a point/flat state, which are both testable and have been tested, have been falsified. The universe does not have a "classical mode" to jump into; it's all amplitudes, all the time.)

Second, a single observed particle doesn't have an amplitude distribution. Rather the system containing yourself, plus the particle, plus the rest of the universe, may approximately factor into the multiplicative product of (1) a sub-distribution over the particle position and (2) a sub-distribution over the rest of the universe. Or rather, the particular blob of amplitude that you happen to be in, can factor that way.

So what could it mean, to associate a "subjective probability" with a component of one factor of a combined amplitude distribution that happens to factorize?

Recall the physics for:

(Human-BLANK * Sensor-BLANK) * (Atom-LEFT + Atom-RIGHT)
=>
(Human-LEFT * Sensor-LEFT * Atom-LEFT) + (Human-RIGHT * Sensor-RIGHT * Atom-RIGHT)

Think of the whole process as reflecting the good-old-fashioned distributive rule of algebra. The initial state can be decomposed - note that this is an identity, not an evolution - into:

(Human-BLANK * Sensor-BLANK) * (Atom-LEFT + Atom-RIGHT)
=
(Human-BLANK * Sensor-BLANK * Atom-LEFT) + (Human-BLANK * Sensor-BLANK * Atom-RIGHT)

We assume that the distribution factorizes. It follows that the term on the left, and the term on the right, initially differ only by a multiplicative factor of Atom-LEFT vs. Atom-RIGHT.

If you were to immediately take the multi-dimensional integral over the squared modulus of the amplitude density of that whole system,

Then the ratio of the all-dimensional integral of the squared modulus over the left-side term, to the all-dimensional integral over the squared modulus of the right-side term,

Would equal the ratio of the lower-dimensional integral over the squared modulus of the Atom-LEFT, to the lower-dimensional integral over the squared modulus of Atom-RIGHT,

For essentially the same reason that if you've got (2 * 3) * (5 + 7), the ratio of (2 * 3 * 5) to (2 * 3 * 7) is the same as the ratio of 5 to 7.

Doing an integral over the squared modulus of a complex amplitude distribution in N dimensions doesn't change that.

There's also a rule called "unitary evolution" in quantum mechanics, which says that quantum evolution never changes the total integral over the squared modulus of the amplitude density.

So if you assume that the initial left term and the initial right term evolve, without overlapping each other, into the final LEFT term and the final RIGHT term, they'll have the same ratio of integrals over etcetera as before.

What all this says is that,

If some roughly independent Atom has got a blob of amplitude on the left of its factor, and a blob of amplitude on the right,

Then, after the Sensor senses the atom, and you look at the Sensor,

The integrated squared modulus of the whole LEFT blob, and the integrated squared modulus of the whole RIGHT blob,

Will have the same ratio,

As the ratio of the squared moduli of the original Atom-LEFT and Atom-RIGHT components.

This is why it's important to remember that apparently individual particles have amplitude distributions that are multiplicative factors within the total joint distribution over all the particles.

If a whole gigantic human experimenter made up of quintillions of particles,

Interacts with one teensy little atom whose amplitude factor has a big bulge on the left and a small bulge on the right,

Then the resulting amplitude distribution, in the joint configuration space,

Has a big amplitude blob for "human sees atom on the left", and a small amplitude blob of "human sees atom on the right".

And what that means, is that the Born probabilities seem to be about finding yourself in a particular blob, not the particle being in a particular place.

But what does the integral over squared moduli have to do with anything? On a straight reading of the data, you would always find yourself in both blobs, every time. How can you find yourself in one blob with greater probability? What are the Born probabilities, probabilities of? Here's the map - where's the territory?

I don't know. It's an open problem. Try not to go funny in the head about it.

This problem is even worse than it looks, because the squared-modulus business is the only non-linear rule in all of quantum mechanics. Everything else - everything else - obeys the linear rule that the evolution of amplitude distribution A, plus the evolution of the amplitude distribution B, equals the evolution of the amplitude distribution A + B.

When you think about the weather in terms of clouds and flapping butterflies, it may not look linear on that higher level. But the amplitude distribution for weather (plus the rest of the universe) is linear on the only level that's fundamentally real.

Does this mean that the squared-modulus business must require additional physics beyond the linear laws we know - that it's necessarily futile to try to derive it on any higher level of organization?

But even this doesn't follow.

Let's say I have a computer program which computes a sequence of positive integers that encode the successive states of a sentient being. For example, the positive integers might describe a Conway's-Game-of-Life universe containing sentient beings (Life is Turing-complete) or some other cellular automaton.

Regardless, this sequence of positive integers represents the time series of a discrete universe containing conscious entities. Call this sequence Sentient(n).

Now consider another computer program, which computes the negative of the first sequence: -Sentient(n). If the computer running Sentient(n) instantiates conscious entities, then so too should a program that computes Sentient(n) and then negates the output.

Now I write a computer program that computes the sequence {0, 0, 0...} in the obvious fashion.

This sequence happens to be equal to the sequence Sentient(n) + -Sentient(n).

So does a program that computes {0, 0, 0...} necessarily instantiate as many conscious beings as both Sentient programs put together?

Admittedly, this isn't an exact analogy for "two universes add linearly and cancel out". For that, you would have to talk about a universe with linear physics, which excludes Conway's Life. And then in this linear universe, two states of the world both containing conscious observers - world-states equal but for their opposite sign - would have to cancel out.

It doesn't work in Conway's Life, but it works in our own universe! Two quantum amplitude distributions can contain components that cancel each other out, and this demonstrates that the number of conscious observers in the sum of two distributions, need not equal the sum of conscious observers in each distribution separately.

So it actually is possible that we could pawn off the only non-linear phenomenon in all of quantum physics onto a better understanding of consciousness. The question "How many conscious observers are contained in an evolving amplitude distribution?" has obvious reasons to be non-linear.

(!)

Robin Hanson has made a suggestion along these lines.

(!!)

Decoherence is a physically continuous process, and the interaction between LEFT and RIGHT blobs may never actually become zero.

So, Robin suggests, any blob of amplitude which gets small enough, becomes dominated by stray flows of amplitude from many larger worlds.

A blob which gets too small, cannot sustain coherent inner interactions - an internally driven chain of cause and effect - because the amplitude flows are dominated from outside. Too-small worlds fail to support computation and consciousness, or are ground up into chaos, or merge into larger worlds.

Hence Robin's cheery phrase, "mangled worlds".

The cutoff point will be a function of the squared modulus, because unitary physics preserves the squared modulus under evolution; if a blob has a certain total squared modulus, future evolution will preserve that integrated squared modulus so long as the blob doesn't split further. You can think of the squared modulus as the amount of amplitude available to internal flows of causality, as opposed to outside impositions.

The seductive aspect of Robin's theory is that quantum physics wouldn't need interpreting. You wouldn't have to stand off beside the mathematical structure of the universe, and say, "Okay, now that you're finished computing all the mere numbers, I'm furthermore telling you that the squared modulus is the 'degree of existence'." Instead, when you run any program that computes the mere numbers, the program automatically contains people who experience the same physics we do, with the same probabilities.

A major problem with Robin's theory is that it seems to predict things like, "We should find ourselves in a universe in which lots of very few decoherence events have already taken place," which tendency does not seem especially apparent.

The main thing that would support Robin's theory would be if you could show from first principles that mangling does happen; and that the cutoff point is somewhere around the median amplitude density (the point where half the total amplitude density is in worlds above the point, and half beneath it), which is apparently what it takes to reproduce the Born probabilities in any particular experiment.

What's the probability that Hanson's suggestion is right? I'd put it under fifty percent, which I don't think Hanson would disagree with. It would be much lower if I knew of a single alternative that seemed equally... reductionist.

But even if Hanson is wrong about what causes the Born probabilities, I would guess that the final answer still comes out equally non-mysterious. Which would make me feel very silly, if I'd embraced a more mysterious-seeming "answer" up until then. As a general rule, it is questions that are mysterious, not answers.

When I began reading Hanson's paper, my initial thought was: The math isn't beautiful enough to be true.

By the time I finished processing the paper, I was thinking: I don't know if this is the real answer, but the real answer has got to be at least this normal.

This is still my position today.

Sequence: Quantum Physics

Decoherent Essences

Decoherence as Projection

Referenced by: Decoherence as Projection • Spooky Action at a Distance: The No-Communication Theorem • Collapse Postulates • If Many-Worlds Had Come First • Many Worlds, One Best Guess • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity • Lawful Creativity • The Anthropic Trilemma • Privileging the Hypothesis

Original with comments: The Born Probabilities

Decoherence as Projection

Eliezer Yudkowsky, 02 May 2008 06:32AM

This post is part of the Quantum Physics Sequence.
Previously in series: The Born Probabilities

In "The So-Called Heisenberg Uncertainty Principle" we got a look at how decoherence can affect the apparent surface properties of objects: By measuring whether a particle is to the left or right of a dividing line, you can decohere the part of the amplitude distribution on the left with the part on the right. Separating the amplitude distribution into two parts affects its future evolution (within each component) because the two components can no longer interfere with each other.

Yet there are more subtle ways to take apart amplitude distributions than by splitting the position basis down the middle. And by exploring this, we rise further up the rabbit hole.

(Remember, the classical world is Wonderland, the quantum world is reality. So when you get deeper into quantum physics, you are going up the rabbit hole, not down the rabbit hole.)

Light has a certain quantum property called "polarization". Of course, all known physical properties are "quantum properties", but in this case I mean that polarization neatly exhibits fundamental quantum characteristics. I mention this, because polarization is often considered part of "classical" optics. Why? Because the quantum nature of polarization is so simple that it was accidentally worked out as part of classical mechanics, back when light was thought to be a wave.

(Nobody tell the marketers, though, or we'll be wearing "quantum sunglasses".)

I don't usually begin by discussing the astronomically high-level phenomena of macroscopic physics, but in this case, I think it will be helpful to begin with a human-world example...

I hand you two little sheets of semi-transparent material, looking perhaps like dark plastic, with small arrows drawn in marker along the sides. When you hold up one of the sheets in front of you, the scene through it is darker - it blocks some of the light.

Now you hold up the second sheet in front of the first sheet...

When the two arrows are aligned, pointing in the same direction, the scene is no darker than before - that is, the two sheets in series block the same amount of light as the first sheet alone.

But as you rotate the second sheet, so that the two arrows point in increasingly different directions, the world seen through both sheets grows darker. When the arrows are at 45° angles, the world is half as bright as when you were only holding up one sheet.

When the two arrows are perpendicular (90°) the world is completely black.

Then, as you continue rotating the second sheet, the world gets lighter again. When the two arrows point in opposite directions, again the lightness is the same as for only one sheet.

Clearly, the sheets are selectively blocking light. Let's call the sheets "polarized filters".

Now, you might reason something like this: "Light is built out of two components, an up-down component and a left-right component. When you hold up a single filter, with the arrow pointing up, it blocks out the left-right component of light, and lets only the up-down component through. When you hold up another filter in front of the first one, and the second filter has the arrow pointing to the left (or the right), it only allows the left-right component of light, and we already blocked that out, so the world is completely dark. And at intermediate angles, it, um, blocks some of the light that wasn't blocked already."

So I ask, "Suppose you've already put the second filter at a 45° angle to the first filter. Now you put up the third filter at a 45° angle to the second filter. What do you expect to see?"

"That's ambiguous," you say. "Do you mean the third filter to end up at a 0° angle to the first filter, or a 90° angle to the first filter?"

"Good heavens," I say, "I'm surprised I forgot to specify that! Tell me what you expect either way."

"If the third filter is at a 0° angle to the first filter," you say, "It won't block out anything the first filter hasn't blocked already. So we'll be left with the half-light world, from the second filter being at a 45° angle to the first filter. And if the third filter is at a 90° angle to the first filter, it will block out everything that the first filter didn't block, and the world will be completely dark."

I hand you a third filter. "Go ahead," I say, "Try it."

First you set the first filter at 0° and the second filter at 45°, as your reference point. Half the light gets through.

Then you set the first filter at 0°, the second filter at 45°, and the third filter at 0°. Now one quarter of the light gets through.

"Huh?" you say.

"Keep going," I reply.

With the first filter at 0°, the second filter at 45°, and the third filter at 90°, one quarter of the light goes through. Again.

"Umm..." you say. You quickly take out the second filter, and find that the world goes completely dark. Then you put in the second filter, again at 45°, and the world resumes one-quarter illumination.

Further investigation quickly verifies that all three filters seem to have the same basic properties - it doesn't matter what order you put them in.

"All right," you say, "that just seems weird." You pause. "So it's probably something quantum."

Indeed it is.

Though light may seem "dim" or "bright" at the macroscopic level, you can't split it up indefinitely; you can always send a single photon into the series of filters, and ask what happens to that single photon.

As you might suspect, if you send a single photon through the succession of three filters, you will find that - assuming the photon passes the first filter (at 0°) - the photon is observed to pass the second filter (at 45°) with 50% probability, and, if the photon does pass the second filter, then it seems to pass the third filter (at 90°) with 50% probability.

The appearance of "probability" in deterministic amplitude evolutions, as we now know, is due to decoherence. Each time a photon was blocked, some other you saw it go through. Each time a photon went through, some other you saw it blocked.

But what exactly is getting decohered? And why does an intervening second filter at 45°, let some photons pass that would otherwise be blocked by the 0° filter plus the 90° filter?

First: We can represent the polarization of light as a complex amplitude for up-down plus a complex amplitude for left-right. So polarizations might be written as (1 ; 0) or (0 ; -i) or (√.5 ; √.5), with the units (up-down ; left-right). It is more customary to write these as column vectors, but row vectors are easier to type.

(Note that I say that this is a way to "represent" the polarization of light. There's nothing magical about picking up-down vs. left-right, instead of upright-downleft vs. upleft-downright. The vectors above are written in an arbitrary but convenient basis. This will become clearer.)

Let's say that the first filter has its little arrow pointing right. This doesn't mean that the filter blocks any photon whose polarization is not exactly (0 ; 1) or a multiple thereof. But it nonetheless happens that all the photons which we see leave the first filter, will have a polarization of (0 ; 1) or some irrelevantly complex multiple thereof. Let's just take this for granted, for the moment. Past the first filter at 0°, we're looking at a stream of photons purely polarized in the left-right direction.

Now the photons hit a second filter. Let's say the second filter is at a 30° angle to the first - so the arrow written on the second filter is pointing 30° above the horizontal.

Then each photon has a 25% probability of being blocked at the second filter, and a 75% probability of going through.

How about if the second filter points to 20° above the horizontal? 12% probability of blockage, 88% probability of going through.

45°, 50/50.

The general rule is that the probability of being blocked is the squared sine of the angle, and the probability of going through is the squared cosine of the angle.

Why?

First, remember two rules we've picked up about quantum mechanics: The evolution of quantum systems is linear and unitary. When an amplitude distribution breaks into parts that then evolve separately, the components must (1) add to the original distribution and (2) have squared moduli adding to the squared modulus of the original distribution.

So now let's consider the photons leaving the first filter, with "polarizations", quantum states, of (0 ; 1).

To understand what happens when the second filter is set at a 45° angle, we observe... and think of this as a purely abstract statement about 2-vectors... that:

(0 ; 1) = (.5 ; .5) + (-.5 ; .5)

Okay, so the two vectors on the right-hand-side sum to (0 ; 1) on the left-hand-side.

But what about the squared modulus? Just because two vectors sum to a third, doesn't mean that the squares of the first two vectors' lengths sum to the square of the third vector's length.

The squared length of the vector (.5 ; .5) is (.5)2 + (.5)2 = .25 + .25 = 0.5. And likewise the squared length of the vector (-.5 ; .5) is (-.5)2 + (.5)2 = 0.5. The sum of the squares is 0.5 + 0.5 = 1. Which matches the squared length of the vector (0 ; 1).

So when you decompose (0 ; 1) into (.5 ; .5) + (-.5 ; .5), this obeys both linearity and unitarity: The two parts sum to the original, and the squared modulus of the parts sums to the squared modulus of the original.

When you interpose the second filter at an angle of 45° from the first, it decoheres the incoming amplitude of (0 ; 1) into an amplitude of (.5 ; .5) for being transmitted and an amplitude of (-.5 ; .5) for being blocked. Taking the squared modulus of the amplitudes gives us the observed Born probabilities, i.e. fifty-fifty.

What if you interposed the second filter at an angle of 30° from the first? Then that would decohere the incoming amplitude vector of (0 ; 1) into the vectors (.433 ; .75) and (-.433, .25). The squared modulus of the first vector is .75, and the squared modulus of the second vector is .25, again summing to one.

A polarized filter projects the incoming amplitude vector into the two sides of a right triangle that sums to the original vector, and decoheres the two components. And so, under Born's rule, the transmission and absorption probabilities are given by the Pythagorean Theorem.

(!)

A filter set at 0° followed by a filter set at 90° will block all light - any photon that emerges from the first filter will have an amplitude vector of (0 ; 1), and the component in the direction of (1 ; 0) will be 0. But suppose that instead you put an intermediate filter at 45°. This will decohere the vector of (0 ; 1) into a transmission vector of (.5 ; .5) and an absorption amplitude of (-.5 ; .5).

A photon that is transmitted through the 45° filter will have a polarization amplitude vector of (.5 ; .5). (The (-.5 ; .5) component is decohered into another world where you see the photon absorbed.)

This photon then hits the 90° filter, whose transmission amplitude is the component in the direction of (1 ; 0), and whose absorption amplitude is the component in the direction of (0 ; 1). (.5 ; .5) has a component of (.5 ; 0) in the direction of (1 ; 0) and a component of (0 ; .5) in the direction of (0 ; 1). So it has an amplitude of (.5 ; 0) to make it through both filters, which translates to a Born probability of .25.

Likewise if the second filter is at -45°. Then it decoheres the incoming (0 ; 1) into a transmission amplitude of (-.5 ; .5) and an absorption amplitude of (.5 ; .5). When (-.5 ; .5) hits the third filter at 90°, it has a component of (-.5 ; 0) in the direction of (1 ; 0), and because these are complex numbers we're talking about, (-.5 ; 0) has a squared modulus of 0.25, that is, 25% probability to go through both filters.

It may seem surprising that putting in an extra filter causes more photons to go through, even when you send them one at a time; but that's quantum physics for you.

"But wait," you say, "Who needs the second filter? Why not just use math? The initial amplitude of (0 ; 1) breaks into an amplitude of (-.5 ; .5) + (.5 ; .5) whether or not you have the second filter there. By linearity, the evolution of the parts should equal the evolution of the whole."

Yes, indeed! So, with no second filter - just the 0° filter and the 90° filter - here's how we'd do that analysis:

First, the 0° filter decoheres off all amplitude of any incoming photons except the component in the direction of (0 ; 1). Now we look at the photon - which has some amplitude (0 ; x) that we've implicitly been renormalizing to (0 ; 1) - and, in a purely mathematical sense, break it up into (.5x ; .5x) and (-.5x ; .5x) whose squared moduli will sum to x^2.

Now first we consider the (.5x ; .5x) component; it strikes the 90° filter which transmits the component (.5x ; 0) and absorbs the (0 ; .5x) component.

Next we consider the (-.5x ; .5x) component. It also strikes the 90° filter, which transmits the component (-.5x ; 0) and absorbs the component (0 ; .5x).

Since no other particles are entangled, we have some identical configurations here: Namely, the two configurations where the photon is transmitted, and the two configurations where the photon is absorbed.

Summing the amplitude vectors of (.5x ; 0) and (-.5x ; 0) for transmission, we get a total amplitude vector of (0 ; 0).

Summing the amplitude vectors of (0 ; .5x) and (0 ; .5x) for absorption, we get an absorption amplitude of (0 ; x).

So all photons that make it through the first filter are blocked.

Remember Experiment 2 from way back when? Opening up a new path to a detector can cause fewer photons to be detected, because the new path has an amplitude of opposite sign to some existing path, and they cancel out.

In an exactly analogous manner, having a filter that sometimes absorbs photons, can cause more (individual) photons to get through a series of filters. Think of it as decohering off a component of the amplitude that would otherwise destructively interfere with another component.

A word about choice of basis:

You could just as easily create a new basis in which (1 ; 0) = (.707 ; .707) and (0 ; 1) = (.707 ; -.707). This is the upright-downleft and upleft-downright basis of which I spoke before. .707 = √.5, so the basis vectors individually have length 1; and the dot product of the two vectors is 0, so they are orthogonal. That is, they are "orthonormal".

The new basis is just as valid as a compass marked NW, NE, SE, SW instead of N, E, S, W. There isn't an absolute basis of the photon's polarization amplitude vector, any more than there's an absolute three-coordinate system that describes your location in space. Ideally, you should see the photon's polarization as a purely abstract 2-vector in complex space.

(One of my great "Ahas!" while reading the Feynman Lectures was the realization that, rather than a 3-vector being made out of an ordered list of 3 scalars, a 3-vector was just a pure mathematical object in a vector algebra. If you wanted to take the 3-vector apart for some reason, you could generate an arbitrary orthonormal basis and get 3 scalars that way. In other words, you didn't build the vector space by composing scalars - you built the decomposition from within the vector space. I don't know if that makes any sense to my readers out there, but it was the great turning point in my relationship with linear algebra.)

Oh, yes, and what happens if you have a complex polarization in the up-down/left-right basis, like (.707i ; .707)? Then that corresponds to "circular polarization" or "elliptical polarization". All the polarizations I've been talking about are "linear polarizations", where the amplitudes in the up-down/left-right basis happen to be real numbers.

When things decohere, they decohere into pieces that add up to the original (linearity) and whose squared moduli add up to the original squared modulus (unitarity). If the squared moduli of the pieces add up to the original squared modulus, this implies the pieces are orthogonal - that the components have inner products of zero with each other. That is why the title of this blog post is "Decoherence as Projection".

A word about how not to see this whole business of polarization:

Some ancient textbooks will say that when you send a photon through a 0° filter, and it goes through, you've learned that the photon is polarized left-right rather than up-down. Now you measure it with another filter at a 45° angle, and it goes through, so you've learned that the photon is polarized upright-downleft rather than upleft-downright. And (says the textbook) this second measurement "destroys" the first, so that if you want to know the up-down / left-right polarization, you'll have to measure it all over again.

Because you can't know both at the same time.

And some of your more strident ancient textbooks will say something along the lines of: the up-down / left-right polarization no longer exists after the photon goes through the 45° filter. It's not just unknown, it doesn't exist, and -

(you might think that wasn't too far from the truth)

- it is meaningless to even talk about it.

Okay. That's going a bit too far.

There are ways to use a polarizer to split a beam into two components, rather than absorbing a component and transmitting a component.

Suppose you first send the photons through a 0° filter. Then you send them through a 45° splitter. Then you recombine the beams. Then you send the photons through a 0° filter again. All the photons that made it past the first filter, will make it past the third filter as well. Because, of course, you've put the components back together again, and (.5 ; .5) + (-.5 ; .5) = (0 ; 1).

This doesn't seem to square with the idea that measuring the 45° polarization automatically destroys the up-down/left-right polarization, that it isn't even meaningful to talk about it.

Of course the one will say, "Ah, but now you no longer know which path the photon took past the splitter. When you recombined the beams, you unmeasured the photon's 45° polarization, and the original 0° polarization popped back into existence again, and it was always meaningful to talk about it."

O RLY?

Anyway, that's all talk about classical surface appearances, and you've seen the underlying quantum reality. A photon with polarization of (-.707 ; .707) has a component of (.707 ; 0) in the up-down direction and a component of (0 ; .707) in the left-right direction. If you happened to feed it into an apparatus that decohered these two components - like a polarizing filter - then you would be able to predict the decoherent evolution as a deterministic fact about the amplitude distribution, and the Born probabilities would (deterministically if mysteriously) come out to 50/50.

Now someone comes along and says that the result of this measurement you may or may not perform, doesn't exist or, better yet, isn't meaningful.

It's hard to see what this startling statement could mean, let alone how it could improve your experimental predictions. How would you falsify it?

Sequence: Quantum Physics

The Born Probabilities

Entangled Photons

Referenced by: Entangled Photons • Bell's Theorem: No EPR "Reality" • Spooky Action at a Distance: The No-Communication Theorem • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds!

Original with comments: Decoherence as Projection

Entangled Photons

Eliezer Yudkowsky, 03 May 2008 07:20AM

This post is part of the Quantum Physics Sequence.
Previously in series: Decoherence as Projection

Today we shall analyze the phenomenon of "entangled particles". We're going to make heavy use of polarized photons here, so you'd better have read yesterday's post.

If a particle at rest decays into two other particles, their net momentum must add up to 0. The two new particles may have amplitudes to head off in all directions, but in each joint configuration, the directions will be opposite.

By a similar method you can produce two entangled photons which head off in opposite directions, and are guaranteed to be polarized oppositely (at right angles to each other), but with a 50% prior probability of going through any given polarized filter.

You might think that this would involve amplitudes over a continuous spectrum of opposite configurations - an amplitude for photon A to be polarized at 0° and for photon B to be polarized at 90°, an amplitude for A to be 1° polarized and for B to be 91° polarized, etc. But in fact it's possible to describe the quantum state "unknown but opposite polarizations" much more compactly.

First, note that the two photons are heading off in opposite directions. This justifies calling one photon A and one photon B; they aren't likely to get their identities mixed up.

As with yesterday, the polarization state (1 ; 0) is what passes a 90° filter. The polarization state (0 ; 1) is what passes a 0° filter. (1 ; 0) is polarized up-down, (0 ; 1) is polarized left-right.

If A is in the polarization state (1 ; 0), we'll write that as A=(1 ; 0).

If A=(1 ; 0) and B=(0 ; 1), we'll write that as

[ A=(1 ; 0) ∧ B=(0 ; 1) ]

The state for "unknown opposite polarization" can be written as:

√(1/2) * ( [ A=(1 ; 0) ∧ B=(0 ; 1) ] - [ A=(0 ; 1) ∧ B=(1; 0) ] )

Note that both terms are being multiplied by the square root of 1/2. This ensures that the squared modulus of both terms sums to 1. Also, don't overlook the minus sign in the center, we'll need it.

If you measure the A photon's polarization in the up-down/left-right basis, the result is pretty straightforward. Your measurement decoheres the entanglement, creating one evolution out of the A=(1 ; 0) ∧ B=(0 ; 1) configuration, and a second, noninteracting evolution out of the A=(0 ; 1) ∧ B=(1; 0) configuration.

If you find that the A photon is polarized up-down, i.e., (1 ; 0), then you know you're in the A=(1 ; 0) ∧ B=(0 ; 1) blob of amplitude. So you know that if you or anyone else measures B, they'll report to you that they found B in the (0 ; 1) or left-right polarization. The version of you that finds A=(1 ; 0), and the version of your friend that finds B=(0 ; 1), always turn out to live in the same blob of amplitude.

On the other side of configuration space, another version of you finds themselves in the A=(0 ; 1) ∧ B=(1; 0) blob. If a friend measures B, the other you will expect to hear that B was polarized up-down, just as you expect to meet the version of your friend that measured B left-right.

But what if you measure the system in a slanted basis - test a photon with a 30° polarized filter? Given the specified starting state, in the up-down / left-right basis, what happens if we measure in the 30° basis instead? Will we still find the photons having opposite polarizations? Can this be demonstrated?

Yes, but the math gets a little more interesting.

Let's review, from yesterday, the case where a photon previously polarized in the up-down/left-right basis encounters a 30° filter.

A 30-60-90 triangle has a hypotenuse of 1, a small side of 1/2, and a longer side of (√3)/2, in accordance with the Pythagorean Theorem.

If a photon passes a 0° filter, coming out with polarization (0 ; 1), and then encounters another filter at 30°, the vector that would be transmitted through the 30° filter is

(√3)/2 * (1/2 ; (√3)/2) = (.433 ; .75)

and the polarization vector that would be absorbed is

1/2 * (-(√3)/2 ; 1/2) = (-.433 ; .25)

Note that the polarization states (1/2 ; (√3)/2) and (-(√3)/2 ; 1/2) form an orthonormal basis: The inner product of each vector with itself is 1, and the inner product of the two vectors with each other is 0.

Then we had (√3)/2 of one basis vector plus 1/2 of the other, guaranteeing the squared moduli would sum to 1. ((√3)/2)2 + (1/2)2 = 3/4 + 1/4 = 1.

So we can say that in the 30° basis, the incoming (0 ; 1) photon had a (√3)/2 amplitude to be transmitted, and a 1/2 amplitude to be absorbed.

Symmetrically, suppose a photon had passed a 90° filter, coming out with polarization (1 ; 0), and then encountered the same 30° filter. Then the transmitted vector would be

1/2 * (1/2 ; (√3)/2) = (.25 ; .433)

and the absorbed vector would be

-(√3)/2 * (-(√3)/2 ; 1/2) = (.75 ; -.433)

Now let's consider again with the entangled pair of photons

√(1/2) * ( [ A=(1 ; 0) ∧ B=(0 ; 1) ] - [ A=(0 ; 1) ∧ B=(1; 0) ] )

and measure photon A with a 30° filter.

Suppose we find that we see photon A absorbed.

Then we know that there was a -(√3)/2 amplitude for this event to occur if the original state had A=(1 ; 0), and a 1/2 amplitude for this event to occur if the original state had A=(0 ; 1).

So, if we see that photon A is absorbed, we learn that we are in the now-decoherent blob of amplitude:

( -(√3)/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(0 ; 1) ] )
- ( 1/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(1; 0) ] )

You might be tempted to add the two amplitudes for A being absorbed - the -(√3)/2 * √(1/2) and the -1/2 * √(1/2) - and get a total amplitude of -.966, which, squared, comes out as .933.

But if this were true, there would be a 93% prior probability of A being absorbed by the filter - a huge prior expectation to see it absorbed. There should be a 50% prior chance of seeing A absorbed.

What went wrong is that, even though we haven't yet measured B, the configurations with B=(0 ; 1) and B=(1 ; 0) are distinct. B could be light-years away, and unknown to us; the configurations would still be distinct. So we don't add the amplitudes for the two terms; we keep them separate.

When the amplitudes for the terms are separately squared, and the squares added together, we get a prior absorption probability of 1/2 - which is exactly what we should expect.

Okay, so we're in the decoherent blob where A is absorbed by a 30° filter. Now consider what happens over at B, within our blob, if a friend measures B with another 30° filter.

The new starting amplitude distribution is:

( -(√3)/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(0 ; 1) ] )
- ( 1/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(1; 0) ] )

In the case where B=(0 ; 1), it has an amplitude of (√3)/2 to be transmitted through a 30° filter; being transmitted through a 30° filter corresponds to the polarization state (1/2 ; (√3)/2). Likewise, a 1/2 amplitude to be absorbed (polarization state (-(√3)/2 ; 1/2).)

In the case where B=(1 ; 0) it has an amplitude of 1/2 to be transmitted with state (1/2 ; (√3)/2). And an amplitude of -(√3)/2 to occupy the state (-(√3)/2 ; 1/2) and be absorbed.

So add up four terms:

( -(√3)/2 * √(1/2) ) * [ A=(-(√3)/2 ; 1/2) ∧ B=(0 ; 1) ]
breaks down into
( -(√3)/2 * √(1/2) ) * (√3)/2 * [ A=(-(√3)/2 ; 1/2) ∧ B=(1/2 ; (√3)/2) ] +
( -(√3)/2 * √(1/2) ) * 1/2    * [ A=(-(√3)/2 ; 1/2) ∧ B=(-(√3)/2 ; 1/2) ]
and
- ( 1/2 * √(1/2) ) * [ A=(-(√3)/2 ; 1/2) ∧ B=(1; 0) ] )
breaks down into
- ( 1/2 * √(1/2) ) * 1/2    * [ A=(-(√3)/2 ; 1/2) ∧ B=(1/2 ; (√3)/2) ] +
- ( 1/2 * √(1/2) ) * -(√3)/2 * [ A=(-(√3)/2 ; 1/2) ∧ B=(-(√3)/2 ; 1/2) ]

These four terms occupy only two distinct configurations.

Adding the amplitudes, the configuration [ A=(-(√3)/2 ; 1/2) ∧ B=(-(√3)/2 ; 1/2) ] ends up with zero amplitude, while [ A=(-(√3)/2 ; 1/2) ∧ B=(1/2 ; (√3)/2) ] ends up with a final amplitude of √(1/2).

So, within the blob in which you've found yourself, the probability of your friend seeing that a 30° filter blocks both A and B, is 0. The probability of seeing that a 30° filter blocks A and transmits B, is 50%.

Symmetrically, there's another blob of amplitude where your other self sees A transmitted and B blocked, with probability 50%. And A transmitted and B transmitted, with probability 0%.

So you and your friend, when you compare results in some particular blob of decohered amplitude, always find that the two photons have opposite polarization.

And in general, if you use two equally oriented polarization filters to measure a pair of photons in the inital state:

√(1/2) * ( [ A=(1 ; 0) ∧ B=(0 ; 1) ] - [ A=(0 ; 1) ∧ B=(1; 0) ] )

then you are guaranteed that one filter will transmit, and the other filter absorb - regardless of how you set the filters, so long as you use the same setting. The photons always have opposite polarizations, even though the prior probability of any particular photon having a particular polarization is 50%.

What if I measure one photon with a 0° filter, and find that it is transmitted (= state (0 ; 1)), and then I measure the other photon with a 30° filter?

The probability works out to just the same as if I knew the other photon had state (1 ; 0) - in effect, it now does.

Over on my side, I've decohered the amplitude over the joint distribution, into blobs in which A has been transmitted, and A absorbed. I am in the decoherent blob with A transmitted: A=(0 ; 1). Ergo, the amplitude vector / polarization state of B, in my blob, behaves as if it starts out as (1 ; 0). This is just as true whether I measure it with another 0° filter, or a 30° filter.

With symmetrically entangled particles, each particle seems to know the state the other particle has been measured in. But "seems" is the operative word here. Actually we're just dealing with decoherence that happens to take place in a very symmetrical way.

Tomorrow (if all goes according to plan) we'll look at Bell's Theorem, which rules out the possibility that each photon already has a fixed, non-quantum state that locally determines the result of any possible polarization measurement.

Sequence: Quantum Physics

Decoherence as Projection

Bell's Theorem: No EPR "Reality"

Referenced by: Bell's Theorem: No EPR "Reality" • Spooky Action at a Distance: The No-Communication Theorem • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds!

Original with comments: Entangled Photons

Bell's Theorem: No EPR "Reality"

Eliezer Yudkowsky, 04 May 2008 04:44AM

This post is part of the Quantum Physics Sequence.
Previously in series: Entangled Photons

(Note: So that this post can be read by people who haven't followed the whole series, I shall temporarily adopt some more standard and less accurate terms; for example, talking about "many worlds" instead of "decoherent blobs of amplitude".)

The legendary Bayesian, E. T. Jaynes, began his life as a physicist. In some of his writings, you can find Jaynes railing against the idea that, because we have not yet found any way to predict quantum outcomes, they must be "truly random" or "inherently random".

Sure, today you don't know how to predict quantum measurements. But how do you know, asks Jaynes, that you won't find a way to predict the process tomorrow? How can any mere experiments tell us that we'll never be able to predict something - that it is "inherently unknowable" or "truly random"?

As far I can tell, Jaynes never heard about decoherence aka Many-Worlds, which is a great pity. If you belonged to a species with a brain like a flat sheet of paper that sometimes split down its thickness, you could reasonably conclude that you'd never be able to "predict" whether you'd "end up" in the left half or the right half. Yet is this really ignorance? It is a deterministic fact that different versions of you will experience different outcomes.

But even if you don't know about Many-Worlds, there's still an excellent reply for "Why do you think you'll never be able to predict what you'll see when you measure a quantum event?" This reply is known as Bell's Theorem.

In 1935, Einstein, Podolsky, and Rosen once argued roughly as follows:

Suppose we have a pair of entangled particles, light-years or at least light-minutes apart, so that no signal can possibly travel between them over the timespan of the experiment. We can suppose these are polarized photons with opposite polarizations.

Polarized filters block some photons, and absorb others; this lets us measure a photon's polarization in a given orientation. Entangled photons (with the right kind of entanglement) are always found to be polarized in opposite directions, when you measure them in the same orientation; if a filter at a certain angle passes photon A (transmits it) then we know that a filter at the same angle will block photon B (absorb it).

Now we measure one of the photons, labeled A, and find that it is transmitted by a 0° polarized filter. Without measuring B, we can now predict with certainty that B will be absorbed by a 0° polarized filter, because A and B always have opposite polarizations when measured in the same basis.

Said EPR:

"If, without in any way disturbing a system, we can predict with certainty (i.e., with probability equal to unity) the value of a physical quantity, then there exists an element of physical reality corresponding to this physical quantity."

EPR then assumed (correctly!) that nothing which happened at A could disturb B or exert any influence on B, due to the spacelike separations of A and B. We'll take up the relativistic viewpoint again tomorrow; for now, let's just note that this assumption is correct.

If by measuring A at 0°, we can predict with certainty whether B will be absorbed or transmitted at 0°, then according to EPR this fact must be an "element of physical reality" about B. Since measuring A cannot influence B in any way, this element of reality must always have been true of B. Likewise with every other possible polarization we could measure - 10°, 20°, 50°, anything. If we measured A first in the same basis, even light-years away, we could perfectly predict the result for B. So on the EPR assumptions, there must exist some "element of reality" corresponding to whether B will be transmitted or absorbed, in any orientation.

But if no one has measured A, quantum theory does not predict with certainty whether B will be transmitted or absorbed. (At least that was how it seemed in 1935.) Therefore, EPR said, there are elements of reality that exist but are not mentioned in quantum theory:

"We are thus forced to conclude that the quantum-mechanical description of physical reality given by wave functions is not complete."

This is another excellent example of how seemingly impeccable philosophy can fail in the face of experimental evidence, thanks to a wrong assumption so deep you didn't even realize it was an assumption.

EPR correctly assumed Special Relativity, and then incorrectly assumed that there was only one version of you who saw A do only one thing. They assumed that the certain prediction about what you would hear from B, described the only outcome that happened at B.

In real life, if you measure A and your friend measures B, different versions of you and your friend obtain both possible outcomes. When you compare notes, the two of you always find the polarizations are opposite. This does not violate Special Relativity even in spirit, but the reason why not is the topic of tomorrow's post, not today's.

Today's post is about how, in 1964, Belldandy John S. Bell irrevocably shot down EPR's original argument. Not by pointing out the flaw in the EPR assumptions - Many-Worlds was not then widely known - but by describing an experiment that disproved them!

It is experimentally impossible for there to be a physical description of the entangled photons, which specifies a single fixed outcome of any polarization measurement individually performed on A or B.

This is Bell's Theorem, which rules out all "local hidden variable" interpretations of quantum mechanics. It's actually not all that complicated, as quantum physics goes!

We begin with a pair of entangled photons, which we'll name A and B. When measured in the same basis, you find that the photons always have opposite polarization - one is transmitted, one is absorbed. As for the first photon you measure, the probability of transmission or absorption seems to be 50-50.

What if you measure with polarized filters set at different angles?

Suppose that I measure A with a filter set at 0°, and find that A was transmitted. In general, if you then measure B at an angle θ to my basis, quantum theory says the probability (of my hearing that) you also saw B transmitted, equals sin^2 θ. E.g. if your filter was at an angle of 30° to my filter, and I saw my photon transmitted, then there's a 25% probability that you see your photon transmitted.

(Why? See "Decoherence as Projection". Some quick sanity checks: sin(0°) = 0, so if we measure at the same angles, the calculated probability is 0 - we never measure at the same angle and see both photons transmitted. Similarly, sin(90°) = 1; if I see A transmitted, and you measure at an orthogonal angle, I will always hear that you saw B transmitted. sin(45°) = √(1/2), so if you measure in a diagonal basis, the probability is 50/50 for the photon to be transmitted or absorbed.)

Oh, and the initial probability of my seeing A transmitted is always 1/2. So the joint probability of seeing both photons transmitted is 1/2 * sin^2 θ. 1/2 probability of my seeing A transmitted, times sin^2 θ probability that you then see B transmitted.

And now you and I perform three statistical experiments, with large sample sizes:

(1) First, I measure A at 0° and you measure B at 20°. The photon is transmitted through both filters on 1/2 sin^2 (20°) = 5.8% of the occasions.

(2) Next, I measure A at 20° and you measure B at 40°. When we compare notes, we again discover that we both saw our photons pass through our filters, on 1/2 sin^2 (40° - 20°) = 5.8% of the occasions.

(3) Finally, I measure A at 0° and you measure B at 40°. Now the photon passes both filters on 1/2 sin^2 (40°) = 20.7% of occasions.

Or to say it a bit more compactly:

1. A transmitted 0°, B transmitted 20°: 5.8%

2. A transmitted 20°, B transmitted 40°: 5.8%

3. A transmitted 0°, B transmitted 40°: 20.7%

What's wrong with this picture?

Nothing, in real life. But on EPR assumptions, it's impossible.

On EPR assumptions, there's a fixed local tendency for any individual photon to be transmitted or absorbed by a polarizer of any given orientation, independent of any measurements performed light-years away, as the single unique outcome.

Consider experiment (2). We measure A at 20° and B at 40°, compare notes, and find we both saw our photons transmitted. Now, A was transmitted at 20°, so if you had measured B at 20°, B would certainly have been absorbed - if you measure in the same basis you must find opposite polarizations.

That is: If A had the fixed tendency to be transmitted at 20°, then B must have had a fixed tendency to be absorbed at 20°. If this rule were violated, you could have measured both photons in the 20° basis, and found that both photons had the same polarization. Given the way that entangled photons are actually produced, this would violate conservation of angular momentum.

So (under EPR assumptions) what we learn from experiment (2) can be equivalently phrased as: "B was a kind of photon that was transmitted by a 40° filter and would have been absorbed by the 20° filter." Under EPR assumptions this is logically equivalent to the actual result of experiment (2).

Now let's look again at the percentages:

1. B is a kind of photon that was transmitted at 20°, and would not have been transmitted at 0°: 5.8%

2. B is a kind of photon that was transmitted at 40°, and would not have been transmitted at 20°: 5.8%

3. B is a kind of photon that was transmitted at 40°, and would not have been transmitted at 0°: 20.7%

If you want to try and see the problem on your own, you can stare at the three experimental results for a while...

(Spoilers ahead.)

Consider a photon pair that gives us a positive result in experiment (3). On EPR assumptions, we now know that the B photon was inherently a type that would have been absorbed at 0°, and was in fact transmitted at 40°. (And conversely, if the B photon is of this type, experiment (3) will always give us a positive result.)

Now take a B photon from a positive experiment (3), and ask: "If instead we had measured B at 20°, would it have been transmitted, or absorbed?" Again by EPR's assumptions, there must be a definite answer to this question. We could have measured A in the 20° basis, and then had certainty of what would happen at B, without disturbing B. So there must be an "element of reality" for B's polarization at 20°.

But if B is a kind of photon that would be transmitted at 20°, then it is a kind of photon that implies a positive result in experiment (1). And if B is a kind of photon that would be absorbed at 20°, it is a kind of photon that would imply a positive result in experiment (2).

If B is a kind of photon that is transmitted at 40° and absorbed at 0°, and it is either a kind that is absorbed at 20° or a kind that is transmitted at 20°; then B must be either a kind that is absorbed at 20° and transmitted at 40°, or a kind that is transmitted at 20° and absorbed at 0°.

So, on EPR's assumptions, it's really hard to see how the same source can manufacture photon pairs that produce 5.8% positive results in experiment (1), 5.8% positive results in experiment (2), and 20.7% positive results in experiment (3). Every photon pair that produces a positive result in experiment (3) should also produce a positive result in either (1) or (2).

"Bell's inequality" is that any theory of hidden local variables implies (1) + (2) >= (3). The experimentally verified fact that (1) + (2) < (3) is a "violation of Bell's inequality". So there are no hidden local variables. QED.

And that's Bell's Theorem. See, that wasn't so horrible, was it?

But what's actually going on here?

When you measure at A, and your friend measures at B a few light-years away, different versions of you observe both possible outcomes - both possible polarizations for your photon. But the amplitude of the joint world where you both see your photons transmitted, goes as √(1/2) * sin θ where θ is the angle between your polarizers. So the squared modulus of the amplitude (which is how we get probabilities in quantum theory) goes as 1/2 sin^2 θ, and that's the probability for finding mutual transmission when you meet a few years later and compare notes. We'll talk tomorrow about why this doesn't violate Special Relativity.

Strengthenings of Bell's Theorem eliminate the need for statistical reasoning: You can show that local hidden variables are impossible, using only properties of individual experiments which are always true given various measurements. (Google "GHZ state" or "GHZM state".) Occasionally you also hear that someone has published a strengthened Bell's experiment in which the two particles were more distantly separated, or the particles were measured more reliably, but you get the core idea. Bell's Theorem is proven beyond a reasonable doubt. Now the physicists are tracking down unreasonable doubts, and Bell always wins.

I know I sometimes speak as if Many-Worlds is a settled issue, which it isn't academically. (If people are still arguing about it, it must not be "settled", right?) But Bell's Theorem itself is agreed-upon academically as an experimental truth. Yes, there are people discussing theoretically conceivable loopholes in the experiments done so far. But I don't think anyone out there really thinks they're going to find an experimental violation of Bell's Theorem as soon as they use a more sensitive photon detector.

What does Bell's Theorem plus its experimental verification tell us, exactly?

My favorite phrasing is one I encountered in D. M. Appleby: "Quantum mechanics is inconsistent with the classical assumption that a measurement tells us about a property previously possessed by the system."

Which is exactly right: Measurement decoheres your blob of amplitude (world), splitting it into several noninteracting blobs (worlds). This creates new indexical uncertainty - uncertainty about which of several versions of yourself you are. Learning which version you are, does not tell you a previously unknown property that was always possessed by the system. And which specific blobs (worlds) are created, depends on the physical measuring process.

It's sometimes said that Bell's Theorem rules out "local realism". Tread cautiously when you hear someone arguing against "realism". As for locality, it is, if anything, far better understood than this whole "reality" business: If life is but a dream, it is a dream that obeys Special Relativity.

It is just one particular sort of locality, and just one particular notion of which things are "real" in the sense of previously uniquely determined, which Bell's Theorem says cannot simultaneously be true.

In particular, decoherent quantum mechanics is local, and Bell's Theorem gives us no reason to believe it is not real. (It may or may not be the ultimate truth, but quantum mechanics is certainly more real than the classical hallucination of little billiard balls bopping around.)

Does Bell's Theorem prevent us from regarding the quantum description as a state of partial knowledge about something more deeply real?

At the very least, Bell's Theorem prevents us from interpreting quantum amplitudes as probability in the obvious way. You cannot point at a single configuration, with probability proportional to the squared modulus, and say, "This is what the universe looked like all along."

In fact, you cannot pick any locally specified description whatsoever of unique outcomes for quantum experiments, and say, "This is what we have partial information about."

So it certainly isn't easy to reinterpret the quantum wavefunction as an uncertain belief. You can't do it the obvious way. And I haven't heard of any non-obvious interpretation of the quantum description as partial information.

Furthermore, as I mentioned previously, it is really odd to find yourself differentiating a degree of uncertain anticipation to get physical results - the way we have to differentiate the quantum wavefunction to find out how it evolves. That's not what probabilities are for.

Thus I try to emphasize that quantum amplitudes are not possibilities, or probabilities, or degrees of uncertain belief, or expressions of ignorance, or any other species of epistemic creatures. Wavefunctions are not states of mind. It would be a very bad sign to have a fundamental physics that operated over states of mind; we know from looking at brains that minds are made of parts.

In conclusion, although Einstein, Podolsky, and Rosen presented a picture of the world that was disproven experimentally, I would still regard them as having won a moral victory: The then-common interpretation of quantum mechanics did indeed have a one person measuring at A, seeing a single outcome, and then making a certain prediction about a unique outcome at B; and this is indeed incompatible with relativity, and wrong. Though people are still arguing about that.

Sequence: Quantum Physics

Entangled Photons

Spooky Action at a Distance: The No-Communication Theorem

Referenced by: Spooky Action at a Distance: The No-Communication Theorem • Quantum Non-Realism • Many Worlds, One Best Guess • The Quantum Physics Sequence • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds!

Original with comments: Bell's Theorem: No EPR "Reality"

Spooky Action at a Distance: The No-Communication Theorem

Eliezer Yudkowsky, 05 May 2008 02:43AM

This post is part of the Quantum Physics Sequence.
Previously in series: Bell's Theorem: No EPR "Reality"

When you have a pair of entangled particles, such as oppositely polarized photons, one particle seems to somehow "know" the result of distant measurements on the other particle. If you measure photon A to be polarized at 0°, photon B somehow immediately knows that it should have the opposite polarization of 90°.

Einstein famously called this "spukhafte Fernwirkung" or "spooky action at a distance". Einstein didn't know about decoherence, so it seemed spooky to him.

Though, to be fair, Einstein knew perfectly well that the universe couldn't really be "spooky". It was a then-popular interpretation of QM that Einstein was calling "spooky", not the universe itself.

Let us first consider how entangled particles look, if you don't know about decoherence - the reason why Einstein called it "spooky":

Suppose we've got oppositely polarized photons A and B, and you're about to measure B in the 20° basis. Your probability of seeing B transmitted by the filter (or absorbed) is 50%.

But wait! Before you measure B, I suddenly measure A in the 0° basis, and the A photon is transmitted! Now, apparently, the probability that you'll see B transmitted is 11.6%. Something has changed! And even if the photons are light-years away, spacelike separated, the change still occurs.

You might try to reply:

"No, nothing has changed - measuring the A photon has told you something about the B photon, you have gained knowledge, you have carried out an inference about a distant object, but no physical influence travels faster-than-light.

"Suppose I put two index cards into an envelope, one marked '+' and one marked '-'. Now I give one envelope to you, and one envelope to a friend of yours, and you get in a spaceship and travel a few light-years away from each other, and then you open your envelope and see '+'. At once you know that your friend is holding the envelope marked '-', but this doesn't mean the envelope's content has changed faster than the speed of light.

"You are committing a Mind Projection Fallacy; the envelope's content is constant, only your local beliefs about distant referents change."

Bell's Theorem, covered yesterday, shows that this reply fails. It is not possible that each photon has an unknown but fixed individual tendency to be polarized a particular way. (Think of how unlikely it would seem, a priori, for this to be something any experiment could tell you!)

Einstein didn't know about Bell's Theorem, but the theory he was criticizing did not say that there were hidden variables; it said that the probabilities changed directly.

But then how fast does this influence travel? And what if you measure the entangled particles in such a fashion that, in their individual reference frames, each measurement takes place before the other?

These experiments have been done. If you think there is an influence traveling, it travels at least six million times as fast as light (in the reference frame of the Swiss Alps). Nor is the influence fazed if each measurement takes place "first" within its own reference frame.

So why can't you use this mysterious influence to send signals faster than light?

Here's something that, as a kid, I couldn't get anyone to explain to me: "Why can't you signal using an entangled pair of photons that both start out polarized up-down? By measuring A in a diagonal basis, you destroy the up-down polarization of both photons. Then by measuring B in the up-down/left-right basis, you can with 50% probability detect the fact that a measurement has taken place, if B turns out to be left-right polarized."

It's particularly annoying that nobody gave me an answer, because the answer turns out to be simple: If both photons have definite polarizations, they aren't entangled. There are just two different photons that both happen to be polarized up-down. Measuring one photon doesn't even change your expectations about the other.

Entanglement is not an extra property that you can just stick onto otherwise normal particles! It is a breakdown of quantum independence. In classical probability theory, if you know two facts, there is no longer any logical dependence left between them. Likewise in quantum mechanics, two particles each with a definite state must have a factorizable amplitude distribution.

Or as old-style quantum theory put it: Entanglement requires superposition, which implies uncertainty. When you measure an entangled particle, you are not able to force your measurement result to take any particular value. So, over on the B end, if they do not know what you measured on A, their probabilistic expectation is always the same as before. (So it was once said).

But in old-style quantum theory, there was indeed a real and instantaneous change in the other particle's statistics which took place as the result of your own measurement. It had to be a real change, by Bell's Theorem and by the invisibly assumed uniqueness of both outcomes.

Even though the old theory invoked a non-local influence, you could never use this influence to signal or communicate with anyone. This was called the "no-signaling condition" or the "no-communication theorem".

Still, on then-current assumptions, they couldn't actually call it the "no influence of any kind whatsoever theorem". So Einstein correctly labeled the old theory as "spooky".

In decoherent terms, the impossibility of signaling is much easier to understand: When you measure A, one version of you sees the photon transmitted and another sees the photon absorbed. If you see the photon absorbed, you have not learned any new empirical fact; you have merely discovered which version of yourself "you" happen to be. From the perspective at B, your "discovery" is not even theoretically a fact they can learn; they know that both versions of you exist. When B finally communicates with you, they "discover" which world they themselves are in, but that's all. The statistics at B really haven't changed - the total Born probability of measuring either polarization is still just 50%!

A common defense of the old theory was that Special Relativity was not violated, because no "information" was transmitted, because the superluminal influence was always "random". As some Hans de Vries fellow points out, information theory says that "random" data is the most expensive kind of data you can transmit. Nor is "random" information always useless: If you and I generate a million entangled particles, we can later measure them to obtain a shared key for use in cryptography - a highly useful form of information which, by Bell's Theorem, could not have already been there before measuring.

But wait a minute. Decoherence also lets you generate the shared key. Does decoherence really not violate the spirit of Special Relativity?

Decoherence doesn't allow "signaling" or "communication", but it allows you to generate a highly useful shared key apparently out of nowhere. Does decoherence really have any advantage over the old-style theory on this one? Or are both theories equally obeying Special Relativity in practice, and equally violating the spirit?

A first reply might be: "The shared key is not 'random'. Both you and your friend generate all possible shared keys, and this is a deterministic and local fact; the correlation only shows up when you meet."

But this just reveals a deeper problem. The counter-objection would be: "The measurement that you perform over at A, splits both A and B into two parts, two worlds, which guarantees that you'll meet the right version of your friend when you reunite. That is non-local physics - something you do at A, makes the world at B split into two parts. This is spooky action at a distance, and it too violates the spirit of Special Relativity. Tu quoque!"

And indeed, if you look at our quantum calculations, they are written in terms of joint configurations. Which, on reflection, doesn't seem all that local!

But wait - what exactly does the no-communication theorem say? Why is it true? Perhaps, if we knew, this would bring enlightenment.

Here is where it starts getting complicated. I myself don't fully understand the no-communication theorem - there are some parts I think I can see at a glance, and other parts I don't. So I will only be able to explain some of it, and I may have gotten it wrong, in which case I pray to some physicist to correct me (or at least tell me where I got it wrong).

When we did the calculations for entangled polarized photons, with A's polarization measured using a 30° filter, we calculated that the initial state

√(1/2) * ( [ A=(1 ; 0) ∧ B=(0 ; 1) ] - [ A=(0 ; 1) ∧ B=(1; 0) ] )

would be decohered into a blob for

( -(√3)/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(0 ; 1) ] )
- ( 1/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(1; 0) ] )

and symmetrically (though we didn't do this calculation) another blob for

( 1/2 * √(1/2) * [ A=(1/2 ; (√3)/2) ∧ B=(0 ; 1) ] )
- ( (√3)/2 * √(1/2) * [ A=(1/2 ; (√3)/2) ∧ B=(1; 0) ] )

These two blobs together add up, linearly, to the initial state, as one would expect. So what changed? At all?

What changed is that the final result at A, for the first blob, is really more like:

(Sensor-A-reads-"ABSORBED") * (Experimenter-A-sees-"ABSORBED") *
{ ( -(√3)/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(0 ; 1) ] )
- ( 1/2 * √(1/2) * [ A=(-(√3)/2 ; 1/2) ∧ B=(1; 0) ] ) }

and correspondingly with the TRANSMITTED blob.

What changed is that one blob in configuration space, was decohered into two distantly separated blobs that can't interact any more.

As we saw from the Heisenberg "Uncertainty Principle", decoherence is a visible, experimentally detectable effect. That's why we have to shield quantum computers from decoherence. So couldn't the decohering measurement at A, have detectable consequences for B?

But think about how B sees the initial state:

√(1/2) * ( [ A=(1 ; 0) ∧ B=(0 ; 1) ] - [ A=(0 ; 1) ∧ B=(1; 0) ] )

From B's perspective, this state is already "not all that coherent", because no matter what B does, it can't make the A=(1 ; 0) and A=(0 ; 1) configurations cross paths. There's already a sort of decoherence here - a separation that B can't eliminate by any local action at B.

And as we've earlier glimpsed, the basis in which you write the initial state is arbitrary. When you write out the state, it has pretty much the same form in the 30° measuring basis as in the 0° measuring basis.

In fact, there's nothing preventing you from writing out the initial state with A in the 30° basis and B in the 0° basis, so long as your numbers add up.

Indeed this is exactly what we did do, when we first wrote out the four terms in the two blobs, and didn't include the sensor or experimenter.

So when A permanently decohered the blobs in the 30° basis, from B's perspective, this merely solidified a decoherence that B could have viewed as already existing.

Obviously, this can't change the local evolution at B (he said, waving his hands a bit).

Now this is only a statement about a quantum measurement that just decoheres the amplitude for A into parts, without A itself evolving in interesting new directions. What if there were many particles on the A side, and something happened on the A side that put some of those particles into identical configurations via different paths?

This is where linearity and unitarity come in. The no-communication theorem requires both conditions: in general, violating linearity or unitarity gives you faster-than-light signaling. (And numerous other superpowers, such as solving NP-complete problems in polynomial time, and possibly Outcome Pumps.)

By linearity, we can consider parts of the amplitude distribution separately, and their evolved states will add up to the evolved state of the whole.

Suppose that there are many particles on the A side, but we count up every configuration that corresponds to some single fixed state of B - say, B=(0 ; 1) or B=France, whatever. We'd get a group of components which looked like:

(AA=1 ∧ AB=2 ∧ AC=Fred ∧ B=France) +
(AA=2 ∧ AB=1 ∧ AC=Sally ∧ B=France) + ...

Linearity says that we can decompose the amplitude distribution around states of B, and the evolution of the parts will add to the whole.

Assume that the B side stays fixed. Then this component of the distribution that we have just isolated, will not interfere with any other components, because other components have different values for B, so they are not identical configurations.

And unitary evolution says that whatever the measure - the integrated squared modulus - of this component, the total measure is the same after evolution at A, as before.

So assuming that B stays fixed, then anything whatsoever happening at A, won't change the measure of the states at B (he said, waving his hands some more).

Nor should it matter whether we consider A first, or B first. Anything that happens at A, within some component of the amplitude distribution, only depends on the A factor, and only happens to the A factor; likewise with B; so the final joint amplitude distribution should not depend on the order in which we consider the evolutions (and he waved his hands a final time).

It seems to me that from here it should be easy to show no communication considering the simultaneous evolution of A and B. Sadly I can't quite see the last step of the argument. I've spent very little time doing actual quantum calculations - this is not what I do for a living - or it would probably be obvious. Unless it's more subtle than it appears, but anyway...

Anyway, if I'm not mistaken - though I'm feeling my way here by mathematical intuition - the no-communication theorem manifests as invariant generalized states of entanglement. From B's perspective, they are entangled with some distant entity A, and that entanglement has an invariant shape that remains exactly the same no matter what happens at A.

To me, at least, this suggests that the apparent non-locality of quantum physics is a mere artifact of the representation used to describe it.

If you write a 3-dimensional vector as "30° west of north, 40° upward slope, and 100 meters long," it doesn't mean that the universe has a basic compass grid, or that there's a global direction of up, or that reality runs on the metric system. It means you chose a convenient representation.

Physics, including quantum physics, is relativistically invariant: You can pick any relativistic frame you like, redo your calculations, and always get the same experimental predictions back out. That we know.

Now it may be that, in the course of doing your calculations, you find it convenient to pick some reference frame, any reference frame, and use that in your math. Greenwich Mean Time, say. This doesn't mean there really is a central clock, somewhere underneath the universe, that operates on Greenwich Mean Time.

The representation we used talked about "joint configurations" of A and B in which the states of A and B were simultaneously specified. This means our representation was not relativistic; the notion of "simultaneity" is arbitrary. We assumed the universe ran on Greenwich Mean Time, in effect.

I don't know what kind of representation would be (1) relativistically invariant, (2) show distant entanglement as invariant, (3) directly represent space-time locality, and (4) evolve each element of the new representation in a way that depended only on an immediate neighborhood of other elements.

But that representation would probably be a lot closer to the Tao.

My suspicion is that a better representation might take its basic mathematical objects as local states of entanglement. I've actually suspected this ever since I heard about holographic physics and the entanglement entropy bound. But that's just raw speculation, at this point.

However, it is important that a fundamental representation be as local and as simple as possible. This is why e.g. "histories of the entire universe" make poor "fundamental" objects, in my humble opinion.

And it's why I find it suspicious to have a representation for calculating quantum physics that talks about a relativistically arbitrary "joint configuration" of A and B, when it seems like each local position has an invariant "distant entanglement" that suffices to determine local evolution. Shouldn't we be able to refactor this representation into smaller pieces?

Though ultimately you do have to retrieve the phenomenon where the experimenters meet again, after being separated by light-years, and discover that they measured the photons with opposite polarizations. Which is provably not something you can get from individual billiard balls bopping around.

I suspect that when we get a representation of quantum mechanics that is local in every way that the physics itself is local, it will be immediately obvious - right there in the representation - that things only happen in one place at a time.

Hence, no faster-than-light communicators. (Dammit!)

Now of course, all this that I have said - all this wondrous normality - relies on the decoherence viewpoint.

It relies on believing that when you measure at A, both possible measurements for A still exist, and are still entangled with B in a way that B sees as invariant.

All the amplitude in the joint configuration is undergoing linear, unitary, local evolution. None of it vanishes. So the probabilities at B are always the same from a global standpoint, and there is no supraluminal influence, period.

If you tried to "interpret" things any differently... well, the no-communication theorem would become a lot less obvious.

Sequence: Quantum Physics

Bell's Theorem: No EPR "Reality"

Decoherence is Simple

Referenced by: Collapse Postulates • Many Worlds, One Best Guess • Relative Configuration Space • Class Project • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds!

Original with comments: Spooky Action at a Distance: The No-Communication Theorem

Decoherence is Simple

Eliezer Yudkowsky, 06 May 2008 07:44AM

This post is part of the Quantum Physics Sequence.

An epistle to the physicists:

When I was but a little lad, my father, a Ph.D. physicist, warned me sternly against meddling in the affairs of physicists; he said that it was hopeless to try to comprehend physics without the formal math. Period. No escape clauses. But I had read in Feynman's popular books that if you really understood physics, you ought to be able to explain it to a nonphysicist. I believed Feynman instead of my father, because Feynman had won the Nobel Prize and my father had not.

It was not until later - when I was reading the Feynman Lectures, in fact - that I realized that my father had given me the simple and honest truth. No math = no physics.

By vocation I am a Bayesian, not a physicist. Yet although I was raised not to meddle in the affairs of physicists, my hand has been forced by the occasional gross misuse of three terms: Simple, falsifiable, and testable.

The foregoing introduction is so that you don't laugh, and say, "Of course I know what those words mean!" There is math here.

Let's begin with the remark that started me down this whole avenue, of which I have seen several versions; paraphrased, it runs:

"The Many-Worlds interpretation of quantum mechanics postulates that there are vast numbers of other worlds, existing alongside our own. Occam's Razor says we should not multiply entities unnecessarily."

Now it must be said, in all fairness, that those who say this will usually also confess:

"But this is not a universally accepted application of Occam's Razor; some say that Occam's Razor should apply to the laws governing the model, not the number of objects inside the model."

So it is good that we are all acknowledging the contrary arguments, and telling both sides of the story -

But suppose you had to calculate the simplicity of a theory.

The original formulation of William of Ockham stated:

Lex parsimoniae: Entia non sunt multiplicanda praeter necessitatem.

"The law of parsimony: Entities should not be multiplied beyond necessity."

But this is qualitative advice. It is not enough to say whether one theory seems more simple, or seems more complex, than another - you have to assign a number; and the number has to be meaningful, you can't just make it up. Crossing this gap is like the difference between being able to eyeball which things are moving "fast" or "slow", and starting to measure and calculate velocities.

Suppose you tried saying: "Count the words - that's how complicated a theory is."

Robert Heinlein once claimed (tongue-in-cheek, I hope) that the "simplest explanation" is always: "The woman down the street is a witch; she did it." Eleven words - not many physics papers can beat that.

Faced with this challenge, there are two different roads you can take.

First, you can ask: "The woman down the street is a what?" Just because English has one word to indicate a concept, doesn't mean that the concept itself is simple. Suppose you were talking to aliens who didn't know about witches, women, or streets - how long would it take you to explain your theory to them? Better yet, suppose you had to write a computer program that embodied your hypothesis, and output what you say are your hypothesis's predictions - how big would that computer program have to be? Let's say that your task is to predict a time series of measured positions for a rock rolling down a hill. If you write a subroutine that simulates witches, this doesn't seem to help narrow down where the rock rolls - the extra subroutine just inflates your code. You might find, however, that your code necessarily includes a subroutine that squares numbers.

Second, you can ask: "The woman down the street is a witch; she did what?" Suppose you want to describe some event, as precisely as you possibly can given the evidence available to you - again, say, the distance/time series of a rock rolling down a hill. You can preface your explanation by saying, "The woman down the street is a witch," but your friend then says, "What did she do?", and you reply, "She made the rock roll one meter after the first second, nine meters after the third second..." Prefacing your message with "The woman down the street is a witch," doesn't help to compress the rest of your description. On the whole, you just end up sending a longer message than necessary - it makes more sense to just leave off the "witch" prefix. On the other hand, if you take a moment to talk about Galileo, you may be able to greatly compress the next five thousand detailed time series for rocks rolling down hills.

If you follow the first road, you end up with what's known as Kolmogorov complexity and Solomonoff induction. If you follow the second road, you end up with what's known as Minimum Message Length.

"Ah, so I can pick and choose among definitions of simplicity?"

No, actually the two formalisms in their most highly developed forms were proven equivalent.

"And I suppose now you're going to tell me that both formalisms come down on the side of 'Occam means counting laws, not counting objects'."

More or less. In Minimum Message Length, so long as you can tell your friend an exact recipe they can mentally follow to get the rolling rock's time series, we don't care how much mental work it takes to follow the recipe. In Solomonoff induction, we count bits in the program code, not bits of RAM used by the program as it runs. "Entities" are lines of code, not simulated objects. And as said, these two formalisms are ultimately equivalent.

Now before I go into any further detail on formal simplicity, let me digress to consider the objection:

"So what? Why can't I just invent my own formalism that does things differently? Why should I pay any attention to the way you happened to decide to do things, over in your field? Got any experimental evidence that shows I should do things this way?"

Yes, actually, believe it or not. But let me start at the beginning.

The conjunction rule of probability theory states:

P(X∧Y) ≤ P(X)

For any propositions X and Y, the probability that "X is true, and Y is true", is less than or equal to the probability that "X is true (whether or not Y is true)". (If this statement sounds not terribly profound, then let me assure you that it is easy to find cases where human probability assessors violate this rule.)

You usually can't apply the conjunction rule P(X∧Y) ≤ P(X) directly to a conflict between mutually exclusive hypotheses. The conjunction rule only applies directly to cases where the left-hand-side strictly implies the right-hand-side. Furthermore, the conjunction is just an inequality; it doesn't give us the kind of quantitative calculation we want.

But the conjunction rule does give us a rule of monotonic decrease in probability: as you tack more details onto a story, and each additional detail can potentially be true or false, the story's probability goes down monotonically. Think of probability as a conserved quantity: there's only so much to go around. As the number of details in a story goes up, the number of possible stories increases exponentially, but the sum over their probabilities can never be greater than 1. For every story "X∧Y", there is a story "X∧~Y". When you just tell the story "X", you get to sum over the possibilities Y and ~Y.

If you add ten details to X, each detail one that could potentially be true or false, then that story must compete with (2^10 - 1) other equally detailed stories for precious probability. If on the other hand it suffices to just say X, you can sum your probability over 2^10 stories ( (X∧Y∧Z∧...) ∨ (X∧~Y∧Z∧...) ∨ ...)

The "entities" counted by Occam's Razor should be individually costly in probability; this is why we prefer theories with fewer of them.

Imagine a lottery which sells up to a million tickets, where each possible ticket is sold only once, and the lottery has sold every ticket at the time of the drawing. A friend of yours has bought one ticket for $1 - which seems to you like a poor investment, because the payoff is only $500,000. Yet your friend says, "Ah, but consider the alternative hypotheses, 'Tomorrow, someone will win the lottery' and 'Tomorrow, I will win the lottery.' Clearly, the latter hypothesis is simpler by Occam's Razor; it only makes mention of one person and one ticket, while the former hypothesis is more complicated: it mentions a million people and a million tickets!"

To say that Occam's Razor only counts laws, and not objects, is not quite correct: what counts against a theory are the entities it must mention explicitly, because these are the entities that cannot be summed over. Suppose that you and a friend are puzzling over an amazing billiards shot, in which you are told the starting state of a billiards table, and which balls were sunk, but not how the shot was made. You propose a theory which involves ten specific collisions between ten specific balls; your friend counters with a theory that involves five specific collisions between five specific balls. What counts against your theories is not just the laws that you claim to govern billiard balls, but any specific billiard balls that had to be in some particular state for your model's prediction to be successful.

If you measure the temperature of your living room as 22° Celsius, it does not make sense to say: "Your thermometer is probably in error; the room is much more likely to be 20° C. Because, when you consider all the particles in the room, there are exponentially vastly more states they can occupy if the temperature is really 22° C - which makes any particular state all the more improbable." But no matter which exact 22° C state your room occupies, you can make the same prediction (for the supervast majority of these states) that your thermometer will end up showing 22° C, and so you are not sensitive to the exact initial conditions. You do not need to specify an exact position of all the air molecules in the room, so that is not counted against the probability of your explanation.

On the other hand - returning to the case of the lottery - suppose your friend won ten lotteries in a row. At this point you should suspect the fix is in. The hypothesis "My friend wins the lottery every time" is more complicated than the hypothesis "Someone wins the lottery every time". But the former hypothesis is predicting the data much more precisely.

In the Minimum Message Length formalism, saying "There is a single person who wins the lottery every time" at the beginning of your message, compresses your description of who won the next ten lotteries; you can just say "And that person is Fred Smith" to finish your message. Compare to, "The first lottery was won by Fred Smith, the second lottery was won by Fred Smith, the third lottery was ..."

In the Solomonoff induction formalism, the prior probability of "My friend wins the lottery every time" is low, because the program that describes the lottery now needs explicit code that singles out your friend; but because that program can produce a tighter probability distribution over potential lottery winners than "Someone wins the lottery every time", it can, by Bayes's Rule, overcome its prior improbability and win out as a hypothesis.

Any formal theory of Occam's Razor should quantitatively define, not only "entities" and "simplicity", but also the "necessity" part.

Minimum Message Length defines necessity as "that which compresses the message".

Solomonoff induction assigns a prior probability to each possible computer program, with the entire distribution, over every possible computer program, summing to no more than 1. This can be accomplished using a binary code where no valid computer program is a prefix of any other valid computer program ("prefix-free code"), i.e. because it contains a stop code. Then the prior probability of any program P is simply 2-L(P) where L(P) is the length of P in bits.

P itself can be a program that takes in a (possibly zero-length) string of bits and outputs the conditional probability that the next bit will be 1; this makes P a probability distribution over all binary sequences. This version of Solomonoff induction, for any string, gives us a mixture of posterior probabilities dominated by the shortest programs that most precisely predict the string. Summing over this mixture gives us a prediction for the next bit.

The upshot is that it takes more Bayesian evidence - more successful predictions, or more precise predictions - to justify more complex hypotheses. But it can be done; the burden of prior improbability is not infinite. If you flip a coin four times, and it comes up heads every time, you don't conclude right away that the coin produces only heads; but if the coin comes up heads twenty times in a row, you should be considering it very seriously. What about the hypothesis that a coin is fixed to produce "HTTHTT..." in a repeating cycle? That's more bizarre - but after a hundred coinflips you'd be a fool to deny it.

Standard chemistry says that in a gram of hydrogen gas there are six hundred billion trillion hydrogen atoms. This is a startling statement, but there was some amount of evidence that sufficed to convince physicists in general, and you particularly, that this statement was true.

Now ask yourself how much evidence it would take to convince you of a theory with six hundred billion trillion separately specified physical laws.

Why doesn't the prior probability of a program, in the Solomonoff formalism, include a measure of how much RAM the program uses, or the total running time?

The simple answer is, "Because space and time resources used by a program aren't mutually exclusive possibilities." It's not like the program specification, that can only have a 1 or a 0 in any particular place.

But the even simpler answer is, "Because, historically speaking, that heuristic doesn't work."

Occam's Razor was raised as an objection to the suggestion that nebulae were actually distant galaxies - it seemed to vastly multiply the number of entities in the universe. All those stars!

Over and over, in human history, the universe has gotten bigger. A variant of Occam's Razor which, on each such occasion, would label the vaster universe as more unlikely, would fare less well under humanity's historical experience.

This is part of the "experimental evidence" I was alluding to earlier. While you can justify theories of simplicity on mathy sorts of grounds, it is also desirable that they actually work in practice. (The other part of the "experimental evidence" comes from statisticians / computer scientists / Artificial Intelligence researchers, testing which definitions of "simplicity" let them construct computer programs that do empirically well at predicting future data from past data. Probably the Minimum Message Length paradigm has proven most productive here, because it is a very adaptable way to think about real-world problems.)

Imagine a spaceship whose launch you witness with great fanfare; it accelerates away from you, and is soon traveling at .9 c. If the expansion of the universe continues, as current cosmology holds it should, there will come some future point where - according to your model of reality - you don't expect to be able to interact with the spaceship even in principle; it has gone over the cosmological horizon relative to you, and photons leaving it will not be able to outrace the expansion of the universe.

Should you believe that the spaceship literally, physically disappears from the universe at the point where it goes over the cosmological horizon relative to you?

If you believe that Occam's Razor counts the objects in a model, then yes, you should. Once the spaceship goes over your cosmological horizon, the model in which the spaceship instantly disappears, and the model in which the spaceship continues onward, give indistinguishable predictions; they have no Bayesian evidential advantage over one another. But one model contains many fewer "entities"; it need not speak of all the quarks and electrons and fields composing the spaceship. So it is simpler to suppose that the spaceship vanishes.

Alternatively, you could say: "Over numerous experiments, I have generalized certain laws that govern observed particles. The spaceship is made up of such particles. Applying these laws, I deduce that the spaceship should continue on after it crosses the cosmological horizon, with the same momentum and the same energy as before, on pain of violating the conservation laws that I have seen holding in every examinable instance. To suppose that the spaceship vanishes, I would have to add a new law, 'Things vanish as soon as they cross my cosmological horizon'."

The decoherence (aka Many-Worlds) version of quantum mechanics states that measurements obey the same quantum-mechanical rules as all other physical processes. Applying these rules to macroscopic objects in exactly the same way as microscopic ones, we end up with observers in states of superposition. Now there are many questions that can be asked here, such as "But then why don't all binary quantum measurements appear to have 50/50 probability, since different versions of us see both outcomes?"

However, the objection that decoherence violates Occam's Razor on account of multiplying objects in the model is simply wrong.

Decoherence does not require the wavefunction to take on some complicated exact initial state. Many-worlds is not specifying all its worlds by hand, but generating them via the compact laws of quantum mechanics. A computer program that directly simulates QM to make experimental predictions, would require a great deal of RAM to run - but simulating the wavefunction is exponentially expensive in any flavor of QM! Decoherence is simply more so. Many physical discoveries in human history, from stars to galaxies, from atoms to quantum mechanics, have vastly increased the apparent CPU load of what we believe to be the universe.

Many-Worlds is not a zillion worlds worth of complicated, any more than the atomic hypothesis is a zillion atoms worth of complicated. For anyone with a quantitative grasp of Occam's Razor that is simply not what the term "complicated" means.

As with the historical case of galaxies, it may be that people have mistaken their shock at the notion of a universe that large, for a probability penalty, and invoked Occam's Razor in justification. But if there are probability penalties for decoherence, the largeness of the implied universe, per se, is definitely not their source!

The notion that decoherent worlds are additional entities penalized by Occam's Razor, is just plain mistaken. It is not sort-of-right. It is not an argument that is weak but still valid. It is not a defensible position that could be shored up with further arguments. It is entirely defective as probability theory. It is not fixable. It is bad math. 2 + 2 = 3.

Continued in: Decoherence is Falsifiable and Testable.

Sequence: Quantum Physics

Spooky Action at a Distance: The No-Communication Theorem

Quantum Non-Realism

Referenced by: Decoherence is Falsifiable and Testable • Collapse Postulates • If Many-Worlds Had Come First • Many Worlds, One Best Guess • The Dilemma: Science or Bayes? • The Rhythm of Disagreement • Why Quantum? • The Quantum Physics Sequence • And the Winner is... Many-Worlds! • Where Recursive Justification Hits Bottom • Crisis of Faith • Complexity and Intelligence • Undiscriminating Skepticism

Original with comments: Decoherence is Simple

Decoherence is Falsifiable and Testable

Eliezer Yudkowsky, 07 May 2008 07:54AM

This post is part of the Quantum Physics Sequence.
Continuation of: Decoherence is Simple

The words "falsifiable" and "testable" are sometimes used interchangeably, which imprecision is the price of speaking in English. There are two different probability-theoretic qualities I wish to discuss here, and I will refer to one as "falsifiable" and the other as "testable" because it seems like the best fit.

As for the math, it begins, as so many things do, with:

This is Bayes's Theorem. I own at least two distinct items of clothing printed with this theorem, so it must be important.

I am going to assume you are familiar with elementary probability theory; should the above theorem fail to be transparently meaningful, you should stop here and read an introduction to Bayes. (I've written one, but it's aimed at not-physicists.)

To review quickly, B here refers to an item of evidence, Ai is some hypothesis under consideration, and the Aj are competing, mutually exclusive hypotheses. P(B|Ai) means "the probability of seeing B, if hypothesis Ai is true" and P(Ai|B) means "the probability hypothesis Ai is true, if we see B".

The mathematical phenomenon that I will call "falsifiability" is the scientifically desirable property of a hypothesis that it should concentrate its probability mass into preferred outcomes, which implies that it must also assign low probability to some un-preferred outcomes; probabilities must sum to 1 and there is only so much probability to go around. Ideally there should be possible observations which would drive down the hypothesis's probability to nearly zero: There should be things the hypothesis cannot explain, conceivable experimental results with which the theory is not compatible. A theory that can explain everything, prohibits nothing, and so gives us no advice about what to expect.

In terms of Bayes's Theorem, if there is at least some observation B that the hypothesis Ai can't explain", i.e., P(B|Ai) is tiny, then the numerator P(B|Ai)P(Ai) will also be tiny, and likewise the posterior probability P(Ai|B). Updating on having seen the impossible result B has driven the probability of Ai down to nearly zero. A theory which refuses to make itself vulnerable in this way, will need to spread its probability widely, so that it has no holes; it will not be able to strongly concentrate probability into a few preferred outcomes; it will not be able to offer precise advice.

Thus is the rule of science derived in probability theory.

As depicted here, "falsifiability" is something you evaluate by looking at a single hypothesis, asking, "How narrowly does it concentrate its probability distribution over possible outcomes? How narrowly does it tell me what to expect? Can it explain some possible outcomes much better than others?"

Is the decoherence interpretation of quantum mechanics falsifiable? Are there experimental results that could drive its probability down to an infinitesimal?

Sure: We could measure entangled particles that should always have opposite spin, and find that if we measure them far enough apart, they sometimes have the same spin.

Or we could find apples falling upward, the planets of the Solar System zigging around at random, and an atom that kept emitting photons without any apparent energy source. Those observations would also falsify decoherent quantum mechanics. They're things that, on the hypothesis that decoherent QM governs the universe, we should definitely not expect to see.

So there do exist observations B whose P(B|Adeco) is infinitesimal, which would drive P(Adeco|B) down to an infinitesimal.

"But that's just because decoherent quantum mechanics is still quantum mechanics! What about the decoherence part, per se, versus the collapse postulate?"

We're getting there. The point is that I just defined a test that leads you to think about one hypothesis at a time (and called it "falsifiability"). If you want to distinguish decoherence versus collapse, you have to think about at least two hypotheses at a time.

Now really the "falsifiability" test is not quite that singly focused, i.e., the sum in the denominator has got to contain some other hypothesis. But what I just defined as "falsifiability" pinpoints the kind of problem that Karl Popper was complaining about, when he said that Freudian psychoanalysis was "unfalsifiable" because it was equally good at coming up with an explanation for every possible thing the patient could do.

If you belonged to an alien species that had never invented the collapse postulate or Copenhagen Interpretation - if the only physical theory you'd ever heard of was decoherent QM - if all you had in your head was the differential equation for the wavefunction's evolution plus the Born probability rule - you would still have sharp expectations of the universe. You would not live in a magical world where anything was probable.

"But you could say exactly the same thing about quantum mechanics without (macroscopic) decoherence."

Well, yes! Someone walking around with the differential equation for the wavefunction's evolution, plus a collapse postulate that obeys the Born probabilities and is triggered before superposition reaches macroscopic levels, still lives in a universe where apples fall down rather than up.

"But where does decoherence make a new prediction, one that lets us test it?"

A "new" prediction relative to what? To the state of knowledge possessed by the ancient Greeks? If you went back in time and showed them decoherent QM, they would be enabled to make many experimental predictions they could not have made before.

When you say "new prediction", you mean "new" relative to some other hypothesis that defines the "old prediction". This gets us into the theory of what I've chosen to label testability; and the algorithm inherently considers at least two hypotheses at a time. You cannot call something a "new prediction" by considering only one hypothesis in isolation.

In Bayesian terms, you are looking for an item of evidence B that will produce evidence for one hypothesis over another, distinguishing between them, and the process of producing this evidence we could call a "test". You are looking for an experimental result B such that:

p(B|Ad) != p(B|Ac)

that is, some outcome B which has a different probability, conditional on the decoherence hypothesis being true, versus its probability if the collapse hypothesis is true. Which in turn implies that the posterior odds for decoherence and collapse, will become different from the prior odds:

This equation is symmetrical (assuming no probability is literally equal to 0). There isn't one Aj labeled "old hypothesis" and another Aj labeled "new hypothesis".

This symmetry is a feature, not a bug, of probability theory! If you are designing an artificial reasoning system that arrives at different beliefs depending on the order in which the evidence is presented, this is labeled "hysteresis" and considered a Bad Thing. I hear that it is also frowned upon in Science.

From a probability-theoretic standpoint we have various trivial theorems that say it shouldn't matter whether you update on X first and then Y, or update on Y first and then X. At least they'd be trivial if human beings didn't violate them so often and so lightly.

If decoherence is "untestable" relative to collapse, then so too, collapse is "untestable" relative to decoherence. What if the history of physics had transpired differently - what if Hugh Everett and John Wheeler had stood in the place of Bohr and Heisenberg, and vice versa? Would it then be right and proper for the people of that world to look at the collapse interpretation, and snort, and say, "Where are the new predictions?"

What if someday we meet an alien species which invented decoherence before collapse? Are we each bound to keep the theory we invented first? Will Reason have nothing to say about the issue, leaving no recourse to settle the argument but interstellar war?

"But if we revoke the requirement to yield new predictions, we are left with scientific chaos. You can add arbitrary untestable complications to old theories, and get experimentally equivalent predictions. If we reject what you call 'hysteresis', how can we defend our current theories against every crackpot who proposes that electrons have a new property called 'scent', just like quarks have 'flavor'?"

Let it first be said that I quite agree that you should reject the one who comes to you and says: "Hey, I've got this brilliant new idea! Maybe it's not the electromagnetic field that's tugging on charged particles. Maybe there are tiny little angels who actually push on the particles, and the electromagnetic field just tells them how to do it. Look, I have all these successful experimental predictions - the predictions you used to call your own!"

So yes, I agree that we shouldn't buy this amazing new theory, but it is not the newness that is the problem.

Suppose that human history had developed only slightly differently, with the Church being a primary grant agency for Science. And suppose that when the laws of electromagnetism were first being worked out, the phenomenon of magnetism had been taken as proof of the existence of unseen spirits, of angels. James Clerk becomes Saint Maxwell, who described the laws that direct the actions of angels.

A couple of centuries later, after the Church's power to burn people at the stake has been restrained, someone comes along and says: "Hey, do we really need the angels?"

"Yes," everyone says. "How else would the mere numbers of the electromagnetic field, translate into the actual motions of particles?"

"It might be a fundamental law," says the newcomer, "or it might be something other than angels, which we will discover later. What I am suggesting is that interpreting the numbers as the action of angels doesn't really add anything, and we should just keep the numbers and throw out the angel part."

And they look one at another, and finally say, "But your theory doesn't make any new experimental predictions, so why should we adopt it? How do we test your assertions about the absence of angels?"

From a normative perspective, it seems to me that if we should reject the crackpot angels in the first scenario, even without being able to distinguish the two theories experimentally, then we should also reject the angels of established science in the second scenario, even without being able to distinguish the two theories experimentally.

It is ordinarily the crackpot who adds on new useless complications, rather than scientists who accidentally build them in at the start. But the problem is not that the complications are new, but that they are useless whether or not they are new.

A Bayesian would say that the extra complications of the angels in the theory, lead to penalties on the prior probability of the theory. If two theories make equivalent predictions, we keep the one that can be described with the shortest message, the smallest program. If you are evaluating the prior probability of each hypothesis by counting bits of code, and then applying Bayesian updating rules on all the evidence available, then it makes no difference which hypothesis you hear about first, or the order in which you apply the evidence.

It is usually not possible to apply formal probability theory in real life, any more than you can predict the winner of a tennis match using quantum field theory. But if probability theory can serve as a guide to practice, this is what it says: Reject useless complications in general, not just when they are new.

"Yes, and useless is precisely what the many worlds of decoherence are! There are supposedly all these worlds alongside our own, and they don't do anything to our world, but I'm supposed to believe in them anyway?"

No, according to decoherence, what you're supposed to believe are the general laws that govern wavefunctions - and these general laws are very visible and testable.

I have argued elsewhere that the imprimatur of science should be associated with general laws, rather than particular events, because it is the general laws that, in principle, anyone can go out and test for themselves. I assure you that I happen to be wearing white socks right now as I type this. So you are probably rationally justified in believing that this is a historical fact. But it is not the specially strong kind of statement that we canonize as a provisional belief of science, because there is no experiment that you can do for yourself to determine the truth of it; you are stuck with my authority. Now, if I were to tell you the mass of an electron in general, you could go out and find your own electron to test, and thereby see for yourself the truth of the general law in that particular case.

The ability of anyone to go out and verify a general scientific law for themselves, by constructing some particular case, is what makes our belief in the general law specially reliable.

What decoherentists say they believe in, is the differential equation that is observed to govern the evolution of wavefunctions - which you can go out and test yourself any time you like; just look at a hydrogen atom.

Belief in the existence of separated portions of the universal wavefunction is not additional, and it is not supposed to be explaining the price of gold in London; it is just a deductive consequence of the wavefunction's evolution. If the evidence of many particular cases gives you cause to believe that X->Y is a general law, and the evidence of some particular case gives you cause to believe X, then you should have P(Y) ≥ P(X∧(X->Y)).

Or to look at it another way, if P(Y|X) ≈ 1, then P(X∧Y) ≈ P(X).

Which is to say, believing extra details doesn't cost you extra probability when they are logical implications of general beliefs you already have. Presumably the general beliefs themselves are falsifiable, though, or why bother?

This is why we don't believe that spaceships blink out of existence when they cross the cosmological horizon relative to us. True, the spaceship's continued existence doesn't have an impact on our world. The spaceship's continued existence isn't helping to explain the price of gold in London. But we get the invisible spaceship for free as a consequence of general laws that imply conservation of mass and energy. If the spaceship's continued existence were not a deductive consequence of the laws of physics as we presently model them, then it would be an additional detail, cost extra probability, and we would have to question why our theory must include this assertion.

The part of decoherence that is supposed to be testable is not the many worlds per se, but just the general law that governs the wavefunction. The decoherentists note that, applied universally, this law implies the existence of entire superposed worlds. Now there are critiques that can be leveled at this theory, most notably "But then where do the Born probabilities come from?". But within the internal logic of decoherence, the many worlds are not offered as an explanation for anything, nor are they the substance of the theory that is meant to be tested; they are simply a logical consequence of those general laws that constitute the substance of the theory.

If A->B then ~B->~A. To deny the existence of superposed worlds, is necessarily to deny the universality of the quantum laws formulated to govern hydrogen atoms and every other examinable case; it is this denial that seems to the decoherentists like the extra and untestable detail. You can't see the other parts of the wavefunction - why postulate additionally that they don't exist?

The events surrounding the decoherence controversy may be unique in scientific history, marking the first time that serious scientists have come forward and said that by historical accident humanity has developed a powerful, successful, mathematical physical theory that includes angels. That there is an entire law, the collapse postulate, which can simply be thrown away, leaving the theory strictly simpler.

To this discussion I wish to contribute the assertion that, in the light of a mathematically solid understanding of probability theory, decoherence is not ruled out by Occam's Razor, nor is it unfalsifiable, nor is it untestable.

We may consider e.g. decoherence and the collapse postulate, side by side, and evaluate critiques such as "Doesn't decoherence definitely predict that quantum probabilities should always be 50/50?" and "Doesn't collapse violate Special Relativity by implying influence at a distance?" We can consider the relative merits of these theories on grounds of their compatibility with experience and the apparent character of physical law.

To assert that decoherence is not even in the game - because the many worlds themselves are "extra entities" that violate Occam's Razor, or because the many worlds themselves are "untestable", or because decoherence makes no "new predictions" - all this is, I would argue, an outright error of probability theory. The discussion should simply discard those particular arguments and move on.

Referenced by: Decoherence is Simple • Collapse Postulates • Many Worlds, One Best Guess • The Quantum Physics Sequence • And the Winner is... Many-Worlds! • No Logical Positivist I

Original with comments: Decoherence is Falsifiable and Testable

Quantum Non-Realism

Eliezer Yudkowsky, 08 May 2008 05:27AM

This post is part of the Quantum Physics Sequence.
Followup to: Bell's Theorem

"Does the moon exist when no one is looking at it?"
-- Albert Einstein, asked of Niels Bohr

Suppose you were just starting to work out a theory of quantum mechanics.

You begin to encounter experiments that deliver different results depending on how closely you observe them. You dig underneath the reality you know, and find an extremely precise mathematical description that only gives you the relative frequency of outcomes; worse, it's made of complex numbers. Things behave like particles on Monday and waves on Tuesday.

The correct answer is not available to you as a hypothesis, because it will not be invented for another thirty years.

In a mess like that, what's the best you could do?

The best you can do is the strict "shut up and calculate" interpretation of quantum mechanics. You'll go on trying to develop new theories, because doing your best doesn't mean giving up. But we've specified that the correct answer won't be available for thirty years, and that means none of the new theories will really be any good. Doing the best you could theoretically do, would mean that you recognized that, even as you looked for ways to test the hypotheses.

The best you could theoretically do would not include saying anything like, "The wavefunction only gives us probabilities, not certainties." That, in retrospect, was jumping to a conclusion; the wavefunction gives us a certainty of many worlds existing. So that part about the wavefunction being only a probability, was not-quite-right. You calculated, but failed to shut up.

If you do the best that you can do without the correct answer being available, then, when you hear about decoherence, it will turn out that you have not said anything incompatible with decoherence. Decoherence is not ruled out by the data and the calculations. So if you refuse to affirm, as positive knowledge, any proposition which was not forced by the data and the calculations, the calculations will not force you to say anything incompatible with decoherence. So too with whatever the correct theory may be, if it is not decoherence. If you go astray, it must be from your own impulses.

But it is hard for human beings to shut up and calculate - really shut up and calculate. There is an overwhelming tendency to treat our ignorance as if it were positive knowledge.

I don't know if any conversations like this ever really took place, but this is how ignorance becomes knowledge:

Gallant: "Shut up and calculate."
Goofus: "Why?"
Gallant: "Because I don't know what these equations mean, just that they seem to work."

- five minutes later -

Goofus: "Shut up and calculate."
Student: "Why?"
Goofus: "Because these equations don't mean anything, they just work."
Student: "Really? How do you know?"
Goofus: "Gallant told me."

A similar transformation occurs in the leap from:

Gallant: "When my calculations show an amplitude of -1/3i for this photon to get absorbed, my experiments showed that the photon was absorbed around 107 times out of 1000, which is a good fit to 1/9, the square of the modulus. There's clearly some kind of connection between the experimental statistics and the squared modulus of the amplitude, but I don't know what."

Goofus: "The probability amplitude doesn't say where the electron is, but where it might be. The squared modulus is the probability that reality will turn out that way. Reality itself is inherently nondeterministic."

And again:

Gallant: "Once I measure something and get an experimental result, I do my future calculations using only the amplitude whose squared modulus went into calculating the frequency of that experimental result. Only this rule makes my further calculations correspond to observed frequencies."

Goofus: "Since the amplitude is the probability, once you know the experimental result, the probability of everything else becomes zero!"

The whole slip from:

The square of this "amplitude" stuff corresponds tightly to our experimentally observed frequencies

to

The amplitude is the probability of getting the measurement

to

Well, obviously, once you know you didn't get a measurement, its probability becomes zero

has got to be one of the most embarrassing wrong turns in the history of science.

If you take all this literally, it becomes the consciousness-causes-collapse interpretation of quantum mechanics. These days, just about nobody will confess to actually believing in the consciousness-causes-collapse interpretation of quantum mechanics -

But the physics textbooks are still written this way! People say they don't believe it, but they talk as if knowledge is responsible for removing incompatible "probability" amplitudes.

Yet as implausible as I find consciousness-causes-collapse, it at least gives us a picture of reality. Sure, it's an informal picture. Sure, it gives mental properties ontologically basic status. You can't calculate when an "experimental observation" occurs or what people "know", you just know when certain probabilities are obviously zero. And this "just knowing" just happens to fit your experimental results, whatever they are -

- but at least consciousness-causes-collapse purports to tell us how the universe works. The amplitudes are real, the collapse is real, the consciousness is real.

Contrast to this argument schema:

Student: "Wait, you're saying that this amplitude disappears as soon as the measurement tells me it's not true?"

Goofus: "No, no! It doesn't literally disappear. The equations don't mean anything - they just give good predictions."

Student: "But then what does happen?"

Goofus: (Whorble. Hiss.) "Never ask that question."

Student: "And what about the part where we measure this photon's polarization over here, and a light-year away, the entangled photon's probability of being polarized up-down changes from 50% to 25%?"

Goofus: "Yes, what about it?"

Student: "Doesn't that violate Special Relativity?"

Goofus: "No, because you're just finding out the other photon's polarization. Remember, the amplitudes aren't real."

Student: "But Bell's Theorem shows there's no possible local hidden variable that could describe the other photon's polarization before we measure it -"

Goofus: "Exactly! It's meaningless to talk about the photon's polarization before we measure it."

Student: "But the probability suddenly changes -"

Goofus: "It's meaningless to talk about it before we measure it!"

What does Goofus even mean, here? Never mind the plausibility of his words; what sort of state of reality would correspond to his words being true?

What way could reality be, that would make it meaningless to talk about Special Relativity being violated, because the property being influenced didn't exist, even though you could calculate the changes to it?

But you know what? Forget that. I want to know the answer to an even more important question:

Where is Goofus getting all this stuff?

Let's suppose that you take the Schrödinger equation, and assert, as a positive fact:

"This equation generates good predictions, but it doesn't mean anything!"

Really? How do you know?

I sometimes go around saying that the fundamental question of rationality is Why do you believe what you believe?

You say the Schrödinger equation "doesn't mean anything". How did this item of definite knowledge end up in your possession, if it is not simply ignorance misinterpreted as knowledge?

Was there some experiment that told you? I am open to the idea that experiments can tell us things that seem philosophically impossible. But in this case I should like to see the decisive data. Was there a point where you carefully set up an experimental apparatus, and worked out what you should expect to see if (1) the Schrödinger equation was meaningful or (2) the Schrödinger equation was meaningless; and then you got result (2)?

Gallant: "If I measure the 90° polarization of a photon, and then measure the 45° polarization, and then measure 90° again, my experimental history shows that in 100 trials a photon was absorbed 47 times and transmitted 53 times."

Goofus: "The 90° polarization and 45° polarization are incompatible properties; they can't both exist at the same time, and if you measure one, it is meaningless to talk about the other."

How do you know?

How did you acquire that piece of knowledge, Goofus? I know where Gallant got his - but where did yours come from?

My attitude toward questions of existence and meaning was nicely illustrated in a discussion of the current state of evidence for whether the universe is spatially finite or spatially infinite, in which James D. Miller chided Robin Hanson:

"Robin, you are suffering from overconfidence bias in assuming that the universe exists. Surely there is some chance that the universe is of size zero."

To which I replied:

"James, if the universe doesn't exist, it would still be nice to know whether it's an infinite or a finite universe that doesn't exist."

Ha! You think pulling that old "universe doesn't exist" trick will stop me? It won't even slow me down!

It's not that I'm ruling out the possibility that the universe doesn't exist. It's just that, even if nothing exists, I still want to understand the nothing as best I can. My curiosity doesn't suddenly go away just because there's no reality, you know!

The nature of "reality" is something about which I'm still confused, which leaves open the possibility that there isn't any such thing. But Egan's Law still applies: "It all adds up to normality." Apples didn't stop falling when Einstein disproved Newton's theory of gravity.

Sure, when the dust settles, it could turn out that apples don't exist, Earth doesn't exist, reality doesn't exist. But the nonexistent apples will still fall toward the nonexistent ground at a meaningless rate of 9.8 m/s^2.

You say the universe doesn't exist? Fine, suppose I believe that - though it's not clear what I'm supposed to believe, aside from repeating the words.

Now, what happens if I press this button?

In The Simple Truth, I said:

"Frankly, I'm not entirely sure myself where this 'reality' business comes from. I can't create my own reality in the lab, so I must not understand it yet. But occasionally I believe strongly that something is going to happen, and then something else happens instead... So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies 'belief', and the latter thingy 'reality'."

You want to say that the quantum-mechanical equations are "not real"? I'll be charitable, and suppose this means something. What might it mean?

Maybe it means the equations which determine my predictions are substantially different from the thingy that determines my experimental results. Then what does determine my experimental results? If you tell me "nothing", I would like to know what sort of "nothing" it is, and why this "nothing" exhibits such apparent regularity in determining e.g. my experimental measurements of the mass of an electron.

I don't take well to people who tell me to stop asking questions. If you tell me something is definitely positively meaningless, I want to know exactly what you mean by that, and how you came to know. Otherwise you have not given me an answer, only told me to stop asking the question.

"The Simple Truth" describes the life of a shepherd and apprentice who have discovered how to count sheep by tossing pebbles into buckets, when they are visited by a delegate from the court who wants to know how the "magic pebbles" work. The shepherd tries to explain, "An empty bucket is magical if and only if the pastures are empty of sheep", but is soon overtaken about the excited discussions of the apprentice and the delegate as to how the magic might get into the pebbles.

Here we have quantum equations that deliver excellent experimental predictions. What exactly does it mean for them to be "meaningless"? Is it like a bucket of pebbles that works for counting sheep, but doesn't have any magic?

Back before Bell's Theorem ruled out local hidden variables, it seemed possible that (as Einstein thought) there was some more complete description of reality which we didn't have, and the quantum theory summarized incomplete knowledge of this more complete description. The laws we'd learned would turn out to be like the laws of statistical mechanics: quantitative statements of uncertainty. This would hardly make the equations "meaningless"; partial knowledge is the meaning of probability.

But Bell's Theorem makes it much less plausible that the quantum equations are partial knowledge of something deterministic, the way that statistical mechanics over classical physics is partial knowledge of something deterministic. And even so, the quantum equations would not be "meaningless" as that phrase is usually taken; they would be "statistical", "approximate", "partial information", or at worst "wrong".

Here we have equations that give us excellent predictions. You say they are "meaningless". I ask what it is that determines my experimental results, then. You cannot answer. Fine, then how do you justify ruling out the possibility that the quantum equations give such excellent predictions because they are, oh, say, meaningful?

I don't mean to trivialize questions of reality or meaning. But to call something "meaningless" and say that the argument is now resolved, finished, over, done with, you must have a theory of exactly how it is meaningless. And when the answer is given, the question should seem no longer mysterious.

As you may recall from Semantic Stopsigns, there are words and phrases which are not so much answers to questions, as cognitive traffic signals which indicate you should stop asking questions. "Why does anything exist in the first place? God!" is the classical example, but there are others, such as "Elan vital!"

Tell people to "shut up and calculate" because you don't know what the calculations mean, and inside of five years, "Shut up!" will be masquerading as a positive theory of quantum mechanics.

I have the highest respect for any historical physicists who even came close to actually shutting up and calculating, who were genuinely conservative in assessing what they did and didn't know. This is the best they could possibly do without actually being Hugh Everett, and I award them fifty rationality points. My scorn is reserved for those who interpreted "We don't know why it works" as the positive knowledge that the equations were definitely not real.

I mean, if that trick worked, it would be too good to confine to one subfield. Why shouldn't physicists use the "not real" loophole outside of quantum mechanics?

"Hey, doesn't your new 'yarn theory' violate Special Relativity?"

"Nah, the equations are meaningless. Say, doesn't your model of 'chaotic evil inflation' violate CPT symmetry?"

"My equations are even more meaningless than your equations! So your criticism double doesn't count."

And if that doesn't work, try writing yourself a "Get Out Of Jail Free" card.

If there is a moral to the whole story, it is the moral of how very hard it is to stay in a state of confessed confusion, without making up a story that gives you closure - how hard it is to avoid manipulating your ignorance as if it were definite knowledge that you possessed.

PS: I now declare open season for any quantum non-realists out there, if they have something they'd like to say.

Sequence: Quantum Physics

Decoherence is Simple

Collapse Postulates

Referenced by: Many Worlds, One Best Guess • The Dilemma: Science or Bayes? • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • What Would You Do Without Morality?

Original with comments: Quantum Non-Realism

Collapse Postulates

Eliezer Yudkowsky, 09 May 2008 07:49AM

This post is part of the Quantum Physics Sequence.
Previously in series: Spooky Action at a Distance
Followup to: Decoherence is Simple, Falsifiable and Testable

Back when people didn't know about macroscopic decoherence aka many-worlds - before it occurred to anyone that the laws deduced with such precision for microscopic physics, might apply universally at all levels - what did people think was going on?

The initial reasoning seems to have gone something like:

"When my calculations showed an amplitude of -1/3i for this photon to get absorbed, my experimental statistics showed that the photon was absorbed around 107 times out of 1000, which is a good fit to 1/9, the square of the modulus."

to

"The amplitude is the probability (by way of the squared modulus)."

to

"Once you measure something and know it didn't happen, its probability goes to zero."

Read literally, this implies that knowledge itself - or even conscious awareness - causes the collapse. Which was in fact the form of the theory put forth by Werner Heisenberg!

But people became increasingly nervous about the notion of importing dualistic language into fundamental physics - as well they should have been! And so the original reasoning was replaced by the notion of an objective "collapse" that destroyed all parts of the wavefunction except one, and was triggered sometime before superposition grew to human-sized levels.

Now, once you're supposing that parts of the wavefunction can just vanish, you might think to ask:

"Is there only one survivor? Maybe there are many surviving worlds, but they survive with a frequency determined by their integrated squared modulus, and so the typical surviving world has experimental statistics that match the Born rule."

Yet collapse theories considered in modern academia, only postulate one surviving world. Why?

Collapse theories were devised in a time when it simply didn't occur to any physicists that more than one world could exist! People took for granted that measurements had single outcomes - it was an assumption so deep it was invisible, because it was what they saw happening. Collapse theories were devised to explain why measurements had single outcomes, rather than (in full generality) why experimental statistics matched the Born rule.

For similar reasons, the "collapse postulates" considered academically, suppose that collapse occurs before any human beings get superposed. But experiments are steadily ruling out the possibility of "collapse" in increasingly large entangled systems. Apparently an experiment is underway to demonstrate quantum superposition at 50-micrometer scales, which is bigger than most neurons and getting up toward the diameter of some human hairs!

So why doesn't someone try jumping ahead of the game, and ask:

"Say, we keep having to postulate the collapse occurs steadily later and later. What if collapse occurs only once superposition reaches planetary scales and substantial divergence occurs - say, Earth's wavefunction collapses around once a minute? Then, while the surviving Earths at any given time would remember a long history of quantum experiments that matched the Born statistics, a supermajority of those Earths would begin obtaining non-Born results from quantum experiments and then abruptly cease to exist a minute later."

Why don't collapse theories like that one have a huge academic following, among the many people who apparently think it's okay for parts of the wavefunction to just vanish? Especially given that experiments are proving superposition in steadily larger systems?

A cynic might suggest that the reason for collapse's continued support, isn't the physical plausibility of having large parts of the wavefunction suddenly vanish, or the hope of somehow explaining the Born statistics. The point is to keep the intuitive appeal of "I don't remember the measurement having more than one result, therefore only one thing happened; I don't remember splitting, so there must be only one of me." You don't remember dying, so superposed humans must never collapse. A theory that dared to stomp on intuition would be missing the whole point. You might as well just move on to decoherence.

So a cynic might suggest.

But surely it is too early to be attacking the motives of collapse supporters. That is mere argument ad hominem. What about the actual physical plausibility of collapse theories?

Well, first: Does any collapse theory have any experimental support? No.

With that out of the way...

If collapse actually worked the way its adherents say it does, it would be:

1. The only non-linear evolution in all of quantum mechanics.

2. The only non-unitary evolution in all of quantum mechanics.

3. The only non-differentiable (in fact, discontinuous) phenomenon in all of quantum mechanics.

4. The only phenomenon in all of quantum mechanics that is non-local in the configuration space.

5. The only phenomenon in all of physics that violates CPT symmetry.

6. The only phenomenon in all of physics that violates Liouville's Theorem (has a many-to-one mapping from initial conditions to outcomes).

7. The only phenomenon in all of physics that is acausal / non-deterministic / inherently random.

8. The only phenomenon in all of physics that is non-local in spacetime and propagates an influence faster than light.

WHAT DOES THE GOD-DAMNED COLLAPSE POSTULATE HAVE TO DO FOR PHYSICISTS TO REJECT IT? KILL A GOD-DAMNED PUPPY?

Sequence: Quantum Physics

Quantum Non-Realism

If Many-Worlds Had Come First

Referenced by: If Many-Worlds Had Come First • Many Worlds, One Best Guess • Timeless Beauty • Principles of Disagreement • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Privileging the Hypothesis • I'm Not Saying People Are Stupid

Original with comments: Collapse Postulates

If Many-Worlds Had Come First

Eliezer Yudkowsky, 10 May 2008 07:43AM

This post is part of the Quantum Physics Sequence.
Followup to: Collapse Postulates, Decoherence is Simple, Falsifiable and Testable

Not that I'm claiming I could have done better, if I'd been born into that time, instead of this one...

Macroscopic decoherence - the idea that the known quantum laws that govern microscopic events, might simply govern at all levels without alteration - also known as "many-worlds" - was first proposed in a 1957 paper by Hugh Everett III. The paper was ignored. John Wheeler told Everett to see Niels Bohr. Bohr didn't take him seriously.

Crushed, Everett left academic physics, invented the general use of Lagrange multipliers in optimization problems, and became a multimillionaire.

It wasn't until 1970, when Bryce DeWitt (who coined the term "many-worlds") wrote an article for Physics Today, that the general field was first informed of Everett's ideas. Macroscopic decoherence has been gaining advocates ever since, and may now be the majority viewpoint (or not).

But suppose that decoherence and macroscopic decoherence had been realized immediately following the discovery of entanglement, in the 1920s. And suppose that no one had proposed collapse theories until 1957. Would decoherence now be steadily declining in popularity, while collapse theories were slowly gaining steam?

Imagine an alternate Earth, where the very first physicist to discover entanglement and superposition, said, "Holy flaming monkeys, there's a zillion other Earths out there!"

In the years since, many hypotheses have been proposed to explain the mysterious Born probabilities. But no one has yet suggested a collapse postulate. That possibility simply has not occurred to anyone.

One day, Huve Erett walks into the office of Biels Nohr...

"I just don't understand," Huve Erett said, "why no one in physics even seems interested in my hypothesis. Aren't the Born statistics the greatest puzzle in modern quantum theory?"

Biels Nohr sighed. Ordinarily, he wouldn't even bother, but something about the young man compelled him to try.

"Huve," says Nohr, "every physicist meets dozens of people per year who think they've explained the Born statistics. If you go to a party and tell someone you're a physicist, chances are at least one in ten they've got a new explanation for the Born statistics. It's one of the most famous problems in modern science, and worse, it's a problem that everyone thinks they can understand. To get attention, a new Born hypothesis has to be... pretty darn good."

"And this," Huve says, "this isn't good?"

Huve gestures to the paper he'd brought to Biels Nohr. It is a short paper. The title reads, "The Solution to the Born Problem". The body of the paper reads:

"When you perform a measurement on a quantum system, all parts of the wavefunction except one point, vanish, with the survivor chosen non-deterministically in a way determined by the Born statistics."

"Let me make absolutely sure," Nohr says carefully, "that I understand you. You're saying that we've got this wavefunction - evolving according to the Wheeler-DeWitt equation - and, all of a sudden, the whole wavefunction, except for one part, just spontaneously goes to zero amplitude. Everywhere at once. This happens when, way up at the macroscopic level, we 'measure' something."

"Right!" Huve says.

"So the wavefunction knows when we 'measure' it. What exactly is a 'measurement'? How does the wavefunction know we're here? What happened before humans were around to measure things?"

"Um..." Huve thinks for a moment. Then he reaches out for the paper, scratches out "When you perform a measurement on a quantum system", and writes in, "When a quantum superposition gets too large."

Huve looks up brightly. "Fixed!"

"I see," says Nohr. "And how large is 'too large'?"

"At the 50-micron level, maybe," Huve says, "I hear they haven't tested that yet."

Suddenly a student sticks his head into the room. "Hey, did you hear? They just verified superposition at the 50-micron level."

"Oh," says Huve, "um, whichever level, then. Whatever makes the experimental results come out right."

Nohr grimaces. "Look, young man, the truth here isn't going to be comfortable. Can you hear me out on this?"

"Yes," Huve says, "I just want to know why physicists won't listen to me."

"All right," says Nohr. He sighs. "Look, if this theory of yours were actually true - if whole sections of the wavefunction just instantaneously vanished - it would be... let's see. The only law in all of quantum mechanics that is non-linear, non-unitary, non-differentiable and discontinuous. It would prevent physics from evolving locally, with each piece only looking at its immediate neighbors. Your 'collapse' would be the only fundamental phenomenon in all of physics with a preferred basis and a preferred space of simultaneity. Collapse would be the only phenomenon in all of physics that violates CPT symmetry, Liouville's Theorem, and Special Relativity. In your original version, collapse would also have been the only phenomenon in all of physics that was inherently mental. Have I left anything out?"

"Collapse is also the only acausal phenomenon," Huve points out. "Doesn't that make the theory more wonderful and amazing?"

"I think, Huve," says Nohr, "that physicists may view the exceptionalism of your theory as a point not in its favor."

"Oh," said Huve, taken aback. "Well, I think I can fix that non-differentiability thing by postulating a second-order term in the -"

"Huve," says Nohr, "I don't think you're getting my point, here. The reason physicists aren't paying attention to you, is that your theory isn't physics. It's magic."

"But the Born statistics are the greatest puzzle of modern physics, and this theory provides a mechanism for the Born statistics!" Huve protests.

"No, Huve, it doesn't," Nohr says wearily. "That's like saying that you've 'provided a mechanism' for electromagnetism by saying that there are little angels pushing the charged particles around in accordance with Maxwell's Equations. Instead of saying, 'Here are Maxwell's Equations, which tells the angels where to push the electrons', we just say, 'Here are Maxwell's Equations' and are left with a strictly simpler theory. Now, we don't know why the Born statistics happen. But you haven't given the slightest reason why your 'collapse postulate' should eliminate worlds in accordance with the Born statistics, rather than something else. You're not even making use of the fact that quantum evolution is unitary -"

"That's because it's not," interjects Huve.

"- which everyone pretty much knows has got to be the key to the Born statistics, somehow. Instead you're merely saying, 'Here are the Born statistics, which tell the collapser how to eliminate worlds', and it's strictly simpler to just say 'Here are the Born statistics'."

"But -" says Huve.

"Also," says Nohr, raising his voice, "you've given no justification for why there's only one surviving world left by the collapse, or why the collapse happens before any humans get superposed, which makes your theory really suspicious to a modern physicist. This is exactly the sort of untestable hypothesis that the 'One Christ' crowd uses to argue that we should 'teach the controversy' when we tell high school students about other Earths."

"I'm not a One-Christer!" protests Huve.

"Fine," Nohr says, "then why do you just assume there's only one world left? And that's not the only problem with your theory. Which part of the wavefunction gets eliminated, exactly? And in which basis? It's clear that the whole wavefunction isn't being compressed down to a delta, or ordinary quantum computers couldn't stay in superposition when any collapse occurred anywhere - heck, ordinary molecular chemistry might start failing -"

Huve quickly crosses out 'one point' on his paper, writes in 'one part', and then says, "Collapse doesn't compress the wavefunction down to one point. It eliminates all the amplitude except one world, but leaves all the amplitude in that world."

"Why?" says Nohr. "In principle, once you postulate 'collapse', then 'collapse' could eliminate any part of the wavefunction, anywhere - why just one neat world left? Does the collapser know we're in here?"

Huve says, "It leaves one whole world because that's what fits our experiments."

"Huve," Nohr says patiently, "the term for that is 'post hoc'. Furthermore, decoherence is a continuous process. If you partition by whole brains with distinct neurons firing, the partitions have almost zero mutual interference within the wavefunction. But plenty of other processes overlap a great deal. There's no possible way you can point to 'one world' and eliminate everything else without making completely arbitrary choices, including an arbitrary choice of basis -"

"But -" Huve says.

"And above all," Nohr says, "the reason you can't tell me which part of the wavefunction vanishes, or exactly when it happens, or exactly what triggers it, is that if we did adopt this theory of yours, it would be the only informally specified, qualitative fundamental law taught in all of physics. Soon no two physicists anywhere would agree on the exact details! Why? Because it would be the only fundamental law in all of modern physics that was believed without experimental evidence to nail down exactly how it worked."

"What, really?" says Huve. "I thought a lot of physics was more informal than that. I mean, weren't you just talking about how it's impossible to point to 'one world'?"

"That's because worlds aren't fundamental, Huve! We have massive experimental evidence underpinning the fundamental law, the Wheeler-DeWitt equation, that we use to describe the evolution of the wavefunction. We just apply exactly the same equation to get our description of macroscopic decoherence. But for difficulties of calculation, the equation would, in principle, tell us exactly when macroscopic decoherence occurred. We don't know where the Born statistics come from, but we have massive evidence for what the Born statistics are. But when I ask you when, or where, collapse occurs, you don't know - because there's no experimental evidence whatsoever to pin it down. Huve, even if this 'collapse postulate' worked the way you say it does, there's no possible way you could know it! Why not a gazillion other equally magical possibilities?"

Huve raises his hands defensively. "I'm not saying my theory should be taught in the universities as accepted truth! I just want it experimentally tested! Is that so wrong?"

"You haven't specified when collapse happens, so I can't construct a test that falsifies your theory," says Nohr. "Now with that said, we're already looking experimentally for any part of the quantum laws that change at increasingly macroscopic levels. Both on general principles, in case there's something in the 20th decimal point that only shows up in macroscopic systems, and also in the hopes we'll discover something that sheds light on the Born statistics. We check decoherence times as a matter of course. But we keep a broad outlook on what might be different. Nobody's going to privilege your non-linear, non-unitary, non-differentiable, non-local, non-CPT-symmetric, non-relativistic, a-frikkin-causal, faster-than-light, in-bloody-formal 'collapse' when it comes to looking for clues. Not until they see absolutely unmistakable evidence and believe me, Huve, it's going to take a hell of a lot of evidence to unmistake this. Even if we did find anomalous decoherence times, and I don't think we will, it wouldn't force your 'collapse' as the explanation."

"What?" says Huve. "Why not?"

"Because there's got to be a billion more explanations that are more plausible than violating Special Relativity," says Nohr. "Do you realize that if this really happened, there would only be a single outcome when you measured a photon's polarization? Measuring one photon in an entangled pair would influence the other photon a light-year away. Einstein would have a heart attack."

"It doesn't really violate Special Relativity," says Huve. "The collapse occurs in exactly the right way to prevent you from ever actually detecting the faster-than-light influence."

"That's not a point in your theory's favor," says Nohr. "Also Einstein would still have a heart attack."

"Oh," says Huve. "Well, we'll say that the relevant aspects of the particle don't exist until the collapse occurs. If something doesn't exist, influencing it doesn't violate Special Relativity -"

"You're just digging yourself deeper. Look, Huve, as a general principle, theories that are actually correct don't generate this level of confusion. But above all, there isn't any evidence for it. You have no logical way of knowing that collapse occurs, and no reason to believe it. You made a mistake. Just say 'oops' and get on with your life."

"But they could find the evidence someday," says Huve.

"I can't think of what evidence could determine this particular one-world hypothesis as an explanation, but in any case, right now we haven't found any such evidence," says Nohr. "We haven't found anything even vaguely suggestive of it! You can't update on evidence that could theoretically arrive someday but hasn't arrived! Right now, today, there's no reason to spend valuable time thinking about this rather than a billion other equally magical theories. There's absolutely nothing that justifies your belief in 'collapse theory' any more than believing that someday we'll learn to transmit faster-than-light messages by tapping into the acausal effects of praying to the Flying Spaghetti Monster!"

Huve draws himself up with wounded dignity. "You know, if my theory is wrong - and I do admit it might be wrong -"

"If?" says Nohr. "Might?"

"If, I say, my theory is wrong," Huve continues, "then somewhere out there is another world where I am the famous physicist and you are the lone outcast!"

Nohr buries his head in his hands. "Oh, not this again. Haven't you heard the saying, 'Live in your own world'? And you of all people -"

"Somewhere out there is a world where the vast majority of physicists believe in collapse theory, and no one has even suggested macroscopic decoherence over the last thirty years!"

Nohr raises his head, and begins to laugh.

"What's so funny?" Huve says suspiciously.

Nohr just laughs harder. "Oh, my! Oh, my! You really think, Huve, that there's a world out there where they've known about quantum physics for thirty years, and nobody has even thought there might be more than one world?"

"Yes," Huve says, "that's exactly what I think."

"Oh my! So you're saying, Huve, that physicists detect superposition in microscopic systems, and work out quantitative equations that govern superposition in every single instance they can test. And for thirty years, not one person says, 'Hey, I wonder if these laws happen to be universal'."

"Why should they?" says Huve. "Physical models sometimes turn out to be wrong when you examine new regimes."

"But to not even think of it?" Nohr says incredulously. "You see apples falling, work out the law of gravity for all the planets in the solar system except Jupiter, and it doesn't even occur to you to apply it to Jupiter because Jupiter is too large? That's like, like some kind of comedy routine where the guy opens a box, and it contains a spring-loaded pie, so the guy opens another box, and it contains another spring-loaded pie, and the guy just keeps doing this without even thinking of the possibility that the next box contains a pie too. You think John von Neumann, who may have been the highest-g human in history, wouldn't think of it?"

"That's right," Huve says, "He wouldn't. Ponder that."

"This is the world where my good friend Ernest formulates his Schrödinger's Cat thought experiment, and in this world, the thought experiment goes: 'Hey, suppose we have a radioactive particle that enters a superposition of decaying and not decaying. Then the particle interacts with a sensor, and the sensor goes into a superposition of going off and not going off. The sensor interacts with an explosive, that goes into a superposition of exploding and not exploding; which interacts with the cat, so the cat goes into a superposition of being alive and dead. Then a human looks at the cat,' and at this point Schrödinger stops, and goes, 'gee, I just can't imagine what could happen next.' So Schrödinger shows this to everyone else, and they're also like 'Wow, I got no idea what could happen at this point, what an amazing paradox'. Until finally you hear about it, and you're like, 'hey, maybe at that point half of the superposition just vanishes, at random, faster than light', and everyone else is like, 'Wow, what a great idea!'"

"That's right," Huve says again. "It's got to have happened somewhere."

"Huve, this is a world where every single physicist, and probably the whole damn human species, is too dumb to sign up for cryonics! We're talking about the Earth where George W. Bush is President."

Sequence: Quantum Physics

Collapse Postulates

Many Worlds, One Best Guess

Referenced by: Many Worlds, One Best Guess • The Failures of Eld Science • The Dilemma: Science or Bayes? • Science Doesn't Trust Your Rationality • Faster Than Science • Einstein's Speed • Principles of Disagreement • The Quantum Physics Sequence • And the Winner is... Many-Worlds! • Hiroshima Day • This Failing Earth • I'm Not Saying People Are Stupid • The Contrarian Status Catch-22 • Normal Cryonics • Undiscriminating Skepticism

Original with comments: If Many-Worlds Had Come First

Many Worlds, One Best Guess

Eliezer Yudkowsky, 11 May 2008 08:32AM

This post is part of the Quantum Physics Sequence.
Previously in series: Collapse Postulates
Followup to: Bell's Theorem, Spooky Action at a Distance, Quantum Non-Realism, Decoherence is Simple, Falsifiable and Testable

If you look at many microscopic physical phenomena - a photon, an electron, a hydrogen atom, a laser - and a million other known experimental setups - it is possible to come up with simple laws that seem to govern all small things (so long as you don't ask about gravity). These laws govern the evolution of a highly abstract and mathematical object that I've been calling the "amplitude distribution", but which is more widely referred to as the "wavefunction".

Now there are gruesome questions about the proper generalization that covers all these tiny cases. Call an object 'grue' if it appears green before January 1, 2020 and appears blue thereafter. If all emeralds examined so far have appeared green, is the proper generalization, "Emeralds are green" or "Emeralds are grue"?

The answer is that the proper generalization is "Emeralds are green". I'm not going to go into the arguments at the moment. It is not the subject of this post, and the obvious answer in this case happens to be correct. The true Way is not stupid: however clever you may be with your logic, it should finally arrive at the right answer rather than a wrong one.

In a similar sense, the simplest generalizations that would cover observed microscopic phenomena alone, take the form of "All electrons have spin 1/2" and not "All electrons have spin 1/2 before January 1, 2020" or "All electrons have spin 1/2 unless they are part of an entangled system that weighs more than 1 gram."

When we turn our attention to macroscopic phenomena, our sight is obscured. We cannot experiment on the wavefunction of a human in the way that we can experiment on the wavefunction of a hydrogen atom. In no case can you actually read off the wavefunction with a little quantum scanner. But in the case of, say, a human, the size of the entire organism defeats our ability to perform precise calculations or precise experiments - we cannot confirm that the quantum equations are being obeyed in precise detail.

We know that phenomena commonly thought of as "quantum" do not just disappear when many microscopic objects are aggregated. Lasers put out a flood of coherent photons, rather than, say, doing something completely different. Atoms have the chemical characteristics that quantum theory says they should, enabling them to aggregate into the stable molecules making up a human.

So in one sense, we have a great deal of evidence that quantum laws are aggregating to the macroscopic level without too much difference. Bulk chemistry still works.

But we cannot directly verify that the particles making up a human, have an aggregate wavefunction that behaves exactly the way the simplest quantum laws say. Oh, we know that molecules and atoms don't disintegrate, we know that macroscopic mirrors still reflect from the middle. We can get many high-level predictions from the assumption that the microscopic and the macroscopic are governed by the same laws, and every prediction tested has come true.

But if someone were to claim that the macroscopic quantum picture, differs from the microscopic one, in some as-yet-untestable detail - something that only shows up at the unmeasurable 20th decimal place of microscopic interactions, but aggregates into something bigger for macroscopic interactions - well, we can't prove they're wrong. It is Occam's Razor that says, "There are zillions of new fundamental laws you could postulate in the 20th decimal place; why are you even thinking about this one?"

If we calculate using the simplest laws which govern all known cases, we find that humans end up in states of quantum superposition, just like photons in a superposition of reflecting from and passing through a half-silvered mirror. In the Schrödinger's Cat setup, an unstable atom goes into a superposition of disintegrating, and not-disintegrating. A sensor, tuned to the atom, goes into a superposition of triggering and not-triggering. (Actually, the superposition is now a joint state of [atom-disintegrated * sensor-triggered] + [atom-stable * sensor-not-triggered].) A charge of explosives, hooked up to the sensor, goes into a superposition of exploding and not exploding; a cat in the box goes into a superposition of being dead and alive; and a human, looking inside the box, goes into a superposition of throwing up and being calm. The same law at all levels.

Human beings who interact with superposed systems will themselves evolve into superpositions. But the brain that sees the exploded cat, and the brain that sees the living cat, will have many neurons firing differently, and hence many many particles in different positions. They are very distant in the configuration space, and will communicate to an exponentially infinitesimal degree. Not the 30th decimal place, but the 10^30th decimal place. No particular mind, no particular cognitive causal process, sees a blurry superposition of cats.

The fact that "you" only seem to see the cat alive, or the cat dead, is exactly what the simplest quantum laws predict. So we have no reason to believe, from our experience so far, that the quantum laws are in any way different at the macroscopic level than the microscopic level.

And physicists have verified superposition at steadily larger levels. Apparently an effort is currently underway to test superposition in a 50-micron object, larger than most neurons.

The existence of other versions of ourselves, and indeed other Earths, is not supposed additionally. We are simply supposing that the same laws govern at all levels, having no reason to suppose differently, and all experimental tests having succeeded so far. The existence of other decoherent Earths is a logical consequence of the simplest generalization that fits all known facts. If you think that Occam's Razor says that the other worlds are "unnecessary entities" being multiplied, then you should check the probability-theoretic math; that is just not how Occam's Razor works.

Yet there is one particular puzzle that seems odd, in trying to extend microscopic laws universally incluing to superposed humans:

If we try to get probabilities by counting the number of distinct observers, then there is no obvious reason why the integrated squared modulus of the wavefunction should correlate with statistical experimental results. There is no known reason for the Born probabilities, and it even seems that, a priori, we would expect a 50/50 probability of any binary quantum experiment going both ways, if we just counted observers.

Robin Hanson suggests that if exponentially tinier-than-average decoherent blobs of amplitude ("worlds") are interfered with by exponentially tiny leakages from larger blobs, we will get the Born probabilities back out. I consider this an interesting possibility, because it is so normal.

(I myself have had recent thoughts along a different track: If I try to count observers the obvious way, I get strange-seeming results in general, not just in the case of quantum physics. If, for example, I split my brain into a trillion similar parts, conditional on winning the lottery while anesthetized; allow my selves to wake up and perhaps differ to small degrees from each other; and then merge them all into one self again; then counting observers the obvious way says I should be able to make myself win the lottery (if I can split my brain and merge it, as an uploaded mind might be able to do).

In this connection, I find it very interesting that the Born rule does not have a split-remerge problem. Given unitary quantum physics, Born's rule is the unique rule that prevents "observers" from having psychic powers - which doesn't explain Born's rule, but is certainly an interesting fact. Given Born's rule, even splitting and remerging worlds would still lead to consistent probabilities. Maybe physics uses better anthropics than I do!

Perhaps I should take my cues from physics, instead of trying to reason it out a priori, and see where that leads me? But I have not been led anywhere yet, so this is hardly an "answer".)

Wallace, Deutsch, and others try to derive Born's Rule from decision theory. I am rather suspicious of this, because it seems like there is a component of "What happens to me?" that I cannot alter by modifying my utility function. Even if I didn't care at all about worlds where I didn't win a quantum lottery, it still seems to me that there is a sense in which I would "mostly" wake up in a world where I didn't win the lottery. It is this that I think needs explaining.

The point is that many hypotheses about the Born probabilities have been proposed. Not as many as there should be, because the mystery was falsely marked "solved" for a long time. But still, there have been many proposals.

There is legitimate hope of a solution to the Born puzzle without new fundamental laws. Your world does not split into exactly two new subprocesses on the exact occasion when you see "ABSORBED" or "TRANSMITTED" on the LCD screen of a photon sensor. We are constantly being superposed and decohered, all the time, sometimes along continuous dimensions - though brains are digital and involve whole neurons firing, and fire/not-fire would be an extremely decoherent state even of a single neuron... There would seem to be room for something unexpected to account for the Born statistics - a better understanding of the anthropic weight of observers, or a better understanding of the brain's superpositions - without new fundamentals.

We cannot rule out, though, the possibility that a new fundamental law is involved in the Born statistics.

As Jess Riedel puts it:

If there's one lesson we can take from the history of physics, it's that everytime new experimental "regimes" are probed (e.g. large velocities, small sizes, large mass densities, large energies), phenomena are observed which lead to new theories (special relativity, quantum mechanics, general relativity, and the standard model, respectively).

"Every time" is too strong. A nitpick, yes, but also an important point: you can't just assume that any particular law will fail in a new regime. But it's possible that a new fundamental law is involved in the Born statistics, and that this law manifests only in the 20th decimal place at microscopic levels (hence being undetectable so far) while aggregating to have substantial effects at macroscopic levels.

Could there be some law, as yet undiscovered, that causes there to be only one world?

This is a shocking notion; it implies that all our twins in the other worlds - all the different versions of ourselves that are constantly split off, not just by human researchers doing quantum measurements, but by ordinary entropic processes - are actually gone, leaving us alone! This version of Earth would be the only version that exists in local space! If the inflationary scenario in cosmology turns out to be wrong, and the topology of the universe is both finite and relatively small - so that Earth does not have the distant duplicates that would be implied by an exponentially vast universe - then this Earth could be the only Earth that exists anywhere, a rather unnerving thought!

But it is dangerous to focus too much on specific hypotheses that you have no specific reason to think about. This is the same root error of the Intelligent Design folk, who pick any random puzzle in modern genetics, and say, "See, God must have done it!" Why 'God', rather than a zillion other possible explanations? - which you would have thought of long before you postulated divine intervention, if not for the fact that you secretly started out already knowing the answer you wanted to find.

You shouldn't even ask, "Might there only be one world?" but instead just go ahead and do physics, and raise that particular issue only if new evidence demands it.

Could there be some as-yet-unknown fundamental law, that gives the universe a privileged center, which happens to coincide with Earth - thus proving that Copernicus was wrong all along, and the Bible right?

Asking that particular question - rather than a zillion other questions in which the center of the universe is Proxima Centauri, or the universe turns out to have a favorite pizza topping and it is pepperoni - betrays your hidden agenda. And though an unenlightened one might not realize it, giving the universe a privileged center that follows Earth around through space would be rather difficult to do with any mathematically simple fundamental law.

So too with asking whether there might be only one world. It betrays a sentimental attachment to human intuitions already proven wrong. The wheel of science turns, but it doesn't turn backward.

We have specific reasons to be highly suspicious of the notion of only one world. The notion of "one world" exists on a higher level of organization, like the location of Earth in space; on the quantum level there are no firm boundaries (though brains that differ by entire neurons firing are certainly decoherent). How would a fundamental physical law identify one high-level world?

Much worse, any physical scenario in which there was a single surviving world, so that any measurement had only a single outcome, would violate Special Relativity.

If the same laws are true at all levels - i.e., if many-worlds is correct - then when you measure one of a pair of entangled polarized photons, you end up in a world in which the photon is polarized, say, up-down, and alternate versions of you end up in worlds where the photon is polarized left-right. From your perspective before doing the measurement, the probabilities are 50/50. Light-years away, someone measures the other photon at a 20° angle to your own basis. From their perspective, too, the probability of getting either immediate result is 50/50 - they maintain an invariant state of generalized entanglement with your faraway location, no matter what you do. But when the two of you meet, years later, your probability of meeting a friend who got the same result is 11.6%, rather than 50%.

If there is only one global world, then there is only a single outcome of any quantum measurement. Either you measure the photon polarized up-down, or left-right, but not both. Light-years away, someone else's probability of measuring the photon polarized similarly in a 20° rotated basis, actually changes from 50/50 to 11.6%.

You cannot possibly interpret this as a case of merely revealing properties that were already there; this is ruled out by Bell's Theorem. There does not seem to be any possible consistent view of the universe in which both quantum measurements have a single outcome, and yet both measurements are predetermined, neither influencing the other. Something has to actually change, faster than light.

And this would appear to be a fully general objection, not just to collapse theories, but to any possible theory that gives us one global world! There is no consistent view in which measurements have single outcomes, but are locally determined (even locally randomly determined). Some mysterious influence has to cross a spacelike gap.

This is not a trivial matter. You cannot save yourself by waving your hands and saying, "the influence travels backward in time to the entangled photons' creation, then forward in time to the other photon, so it never actually crosses a spacelike gap". (This view has been seriously put forth, which gives you some idea of the magnitude of the paradox implied by one global world!) One measurement has to change the other, so which measurement happens first? Is there a global space of simultaneity? You can't have both measurements happen "first" because under Bell's Theorem, there's no way local information could account for observed results, etc.

Incidentally, this experiment has already been performed, and if there is a mysterious influence it would have to travel six million times as fast as light in the reference frame of the Swiss Alps. Also, the mysterious influence has been experimentally shown not to care if the two photons are measured in reference frames which would cause each measurement to occur "before the other".

Special Relativity seems counterintuitive to us humans - like an arbitrary speed limit, which you could get around by going backward in time, and then forward again. A law you could escape prosecution for violating, if you managed to hide your crime from the authorities.

But what Special Relativity really says is that human intuitions about space and time are simply wrong. There is no global "now", there is no "before" or "after" across spacelike gaps. The ability to visualize a single global world, even in principle, comes from not getting Special Relativity on a gut level. Otherwise it would be obvious that physics proceeds locally with invariant states of distant entanglement, and the requisite information is simply not locally present to support a globally single world.

It might be that this seemingly impeccable logic is flawed - that my application of Bell's Theorem and relativity to rule out any single global world, contains some hidden assumption of which I am unaware -

- but consider the burden that a single-world theory must now shoulder! There is absolutely no reason in the first place to suspect a global single world; this is just not what current physics says! The global single world is an ancient human intuition that was disproved, like the idea of a universal absolute time. The superposition principle is visible even in half-silvered mirrors; experiments are verifying the disproof at steadily larger levels of superposition - but above all there is no longer any reason to privilege the hypothesis of a global single world. The ladder has been yanked out from underneath that human intuition.

There is no experimental evidence that the macroscopic world is single (we already know the microscopic world is superposed). And the prospect necessarily either violates Special Relativity, or takes an even more miraculous-seeming leap and violates seemingly impeccable logic. The latter, of course, being much more plausible in practice. But it isn't really that plausible in an absolute sense. Without experimental evidence, it is generally a bad sign to have to postulate arbitrary logical miracles.

As for quantum non-realism, it appears to me to be nothing more than a Get-Out-Of-Jail-Free card. "It's okay to violate Special Relativity because none of this is real anyway!" The equations cannot reasonably be hypothesized to deliver such excellent predictions for literally no reason. Bell's Theorem rules out the obvious possibility that quantum theory represents imperfect knowledge of something locally deterministic.

Furthermore, macroscopic decoherence gives us a perfectly realistic understanding of what is going on, in which the equations deliver such good predictions because they mirror reality. And so the idea that the quantum equations are just "meaningless", and therefore, it is okay to violate Special Relativity, so we can have one global world after all, is not necessary. To me, quantum non-realism appears to be a huge bluff built around semantic stopsigns like "Meaningless!"

It is not quite safe to say that the existence of multiple Earths is as well-established as any other truth of science. The existence of quantum other worlds is not so well-established as the existence of trees, which most of us can personally observe.

Maybe there is something in that 20th decimal place, which aggregates to something bigger in macroscopic events. Maybe there's a loophole in the seemingly iron logic which says that any single global world must violate Special Relativity, because the information to support a single global world is not locally available. And maybe the Flying Spaghetti Monster is just messing with us, and the world we know is a lie.

So all we can say about the existence of multiple Earths, is that it is as rationally probable as e.g. the statement that spinning black holes do not violate conservation of angular momentum. We have extremely fundamental reasons, having to do with the rotational symmetry of space, to suspect that conservation of angular momentum is built into the underlying nature of physics. And we have no specific reason to suspect this particular violation of our old generalizations in a higher-energy regime.

But we haven't actually checked conservation of angular momentum for rotating black holes - so far as I know. (And as I am talking here about rational guesses in states of partial knowledge, the point is exactly the same if the observation has been made and I do not know it yet.) And black holes are a more massive regime. So the obedience of black holes is not quite as assured as that my toilet conserves angular momentum while flushing, which come to think, I haven't checked either...

Yet if you make the mistake of thinking too hard about this one particular possibility, instead of zillions of other possibilities - and especially if you don't understand the fundamental reason why angular momentum is conserved - then it may start seeming more and more plausible that "spinning black holes violate conservation of angular momentum", as you think of more and more vaguely plausible-sounding reasons it could be true.

But the rational probability is pretty damned small.

Likewise the rational probability that there is only one Earth.

I mention this to explain my habit of talking as if many-worlds is an obvious fact. Many-worlds is an obvious fact, if you have all your marbles lined up correctly (understand very basic quantum physics, know the formal probability theory of Occam's Razor, understand Special Relativity, etc.) It is in fact considerably more obvious to me than the proposition that spinning black holes should obey conservation of angular momentum.

The only reason why many-worlds is not universally acknowledged as a direct prediction of physics which requires magic to violate, is that a contingent accident of our Earth's scientific history gave an entrenched academic position to a phlogiston-like theory which had an unobservable faster-than-light magical "collapse" devouring all other worlds. And many academic physicists do not have a mathematical grasp of Occam's Razor, which is the usual method for ridding physics of invisible angels. So when they encounter many-worlds and it conflicts with their (undermined) intuition that only one world exists, they say, "Oh, that's multiplying entities" - which is just flatly wrong as probability theory - and go on about their daily lives.

I am not in academia. I am not constrained to bow and scrape to some senior physicist who hasn't grasped the obvious, but who will be reviewing my journal articles. I need have no fear that I will be rejected for tenure on account of scaring my students with "science-fiction tales of other Earths". If I can't speak plainly, who can?

So let me state then, very clearly, on behalf of any and all physicists out there who dare not say it themselves: Many-worlds wins outright given our current state of evidence. There is no more reason to postulate a single Earth, than there is to postulate that two colliding top quarks would decay in a way that violates conservation of energy. It takes more than an unknown fundamental law; it takes magic.

The debate should already be over. It should have been over fifty years ago. The state of evidence is too lopsided to justify further argument. There is no balance in this issue. There is no rational controversy to teach. The laws of probability theory are laws, not suggestions; there is no flexibility in the best guess given this evidence. Our children will look back at the fact that we were STILL ARGUING about this in the early 21st-century, and correctly deduce that we were nuts.

We have embarrassed our Earth long enough by failing to see the obvious. So for the honor of my Earth, I write as if the existence of many-worlds were an established fact, because it is. The only question now is how long it will take for the people of this world to update.

Sequence: Quantum Physics

If Many-Worlds Had Come First

Living in Many Worlds

Referenced by: The Dilemma: Science or Bayes? • Do Scientists Already Know This Stuff? • No Safe Defense, Not Even Science • Faster Than Science • Einstein's Speed • Mach's Principle: Anti-Epiphenomenal Physics • Timeless Physics • Principles of Disagreement • Timeless Identity • Living in Many Worlds • Timeless Control • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Privileging the Hypothesis

Original with comments: Many Worlds, One Best Guess

The Failures of Eld Science

Eliezer Yudkowsky, 12 May 2008 10:32AM

Followup to: Initiation Ceremony, If Many-Worlds Had Come First

This time there were no robes, no hoods, no masks. Students were expected to become friends, and allies. And everyone knew why you were in the classroom. It would have been pointless to pretend you weren't in the Conspiracy.

Their sensei was Jeffreyssai, who might have been the best of his era, in his era. His students were either the most promising learners, or those whom the beisutsukai saw political advantage in molding.

Brennan fell into the latter category, and knew it. Nor had he hesitated to use his Mistress's name to open doors. You used every avenue available to you, in seeking knowledge; that was respected here.

"- for over thirty years," Jeffreyssai said. "Not one of them saw it; not Einstein, not Schrödinger, not even von Neumann." He turned away from his sketcher, and toward the classroom. "I pose to you to the question: How did they fail?"

The students exchanged quick glances, a calculus of mutual risk between the wary and the merely baffled. Jeffreyssai was known to play games.

Finally Hiriwa-called-the-Black leaned forward, jangling slightly as her equation-carved bracelets shifted on her ankles. "By your years given, sensei, this was two hundred and fifty years after Newton. Surely, the scientists of that era must have grokked the concept of a universal law."

"Knowing the universal law of gravity," said the student Taji, from a nearby seat, "is not the same as understanding the concept of a universal law." He was one of the promising ones, as was Hiriwa.

Hiriwa frowned. "No... it was said that Newton had been praised for discovering the first universal. Even in his own era. So it was known." Hiriwa paused. "But Newton himself would have been gone. Was there a religious injunction against proposing further universals? Did they refrain out of respect for Newton, or were they waiting for his ghost to speak? I am not clear on how Eld science was motivated -"

"No," murmured Taji, a laugh in his voice, "you really, really aren't."

Jeffreyssai's expression was kindly. "Hiriwa, it wasn't religion, and it wasn't lead in the drinking water, and they didn't all have Alzheimers, and they weren't sitting around all day reading webcomics. Forget the catalogue of horrors out of ancient times. Just think in terms of cognitive errors. What could Eld science have been thinking wrong?"

Hiriwa sat back with a sigh. "Sensei, I truly cannot imagine a snafu that would do that."

"It wouldn't be just one mistake," Taji corrected her. "As the saying goes: Mistakes don't travel alone; they hunt in packs."

"But the entire human species?" said Hiriwa. "Thirty years?"

"It wasn't the entire human species, Hiriwa," said Styrlyn. He was one of the older-looking students, wearing a short beard speckled in grey. "Maybe one in a hundred thousand could have written out Schrödinger's Equation from memory. So that would have been their first and primary error - failure to concentrate their forces."

"Spare us the propaganda!" Jeffreyssai's gaze was suddenly fierce. "You are not here to proselytize for the Cooperative Conspiracy, my lord politician! Bend not the truth to make your points! I believe your Conspiracy has a phrase: 'Comparative advantage.' Do you really think that it would have helped to call in the whole human species, as it existed at that time, to debate quantum physics?"

Styrlyn didn't flinch. "Perhaps not, sensei," he said. "But if you are to compare that era to this one, it is a consideration."

Jeffreyssai moved his hand flatly through the air; the maybe-gesture he used to dismiss an argument that was true but not relevant. "It is not what I would call a primary mistake. The puzzle should not have required a billion physicists to solve."

"I can think of more specific ancient horrors," said Taji. "Spending all day writing grant proposals. Teaching undergraduates who would rather be somewhere else. Needing to publish thirty papers a year to get tenure..."

"But we are not speaking of only the lower-status scientists," said Yin; she wore a slightly teasing grin. "It was said of Schrödinger that he retired to a villa for a month, with his mistress to provide inspiration, and emerged with his eponymous equation. We consider it a famous historical success of our methodology. Some Eld physicists did understand how to focus their mental energies; and would have been senior enough to do so, had they chose."

"True," Taji said. "In the end, administrative burdens are only a generic obstacle. Likewise such answers as, 'They were not trained in probability theory, and did not know of cognitive biases.' Our sensei seems to desire some more specific reply."

Jeffreyssai lifted an eyebrow encouragingly. "Don't dismiss your line of thought so quickly, Taji; it begins to be relevant. What kind of system would create administrative burdens on its own people?"

"A system that failed to support its people adequately," said Styrlyn. "One that failed to value their work."

"Ah," said Jeffreyssai. "But there is a student who has not yet spoken. Brennan?"

Brennan didn't jump. He deliberately waited just long enough to show he wasn't scared, and then said, "Lack of pragmatic motivation, sensei."

Jeffreyssai smiled slightly. "Expand."

What kind of system would create administrative burdens on its own people?, their sensei had asked them. The other students were pursuing their own lines of thought. Brennan, hanging back, had more attention to spare for his teacher's few hints. Being the beginner wasn't always a disadvantage - and he had been taught, long before the Bayesians took him in, to take every available advantage.

"The Manhattan Project," Brennan said, "was launched with a specific technological end in sight: a weapon of great power, in time of war. But the error that Eld Science committed with respect to quantum physics had no immediate consequences for their technology. They were confused, but they had no desperate need for an answer. Otherwise the surrounding system would have removed all burdens from their effort to solve it. Surely the Manhattan Project must have done so - Taji? Do you know?"

Taji looked thoughtful. "Not all burdens - but I'm pretty sure they weren't writing grant proposals in the middle of their work."

"So," Jeffreyssai said. He advanced a few steps, stood directly in front of Brennan's desk. "You think Eld scientists simply weren't trying hard enough. Because their art had no military applications? A rather competitive point of view, I should think."

"Not necessarily," Brennan said calmly. "Pragmatism is a virtue of rationality also. A desired use for a better quantum theory, would have helped the Eld scientists in many ways beyond just motivating them. It would have given shape to their curiosity, and told them what constituted success or failure."

Jeffreyssai chuckled slightly. "Don't guess so hard what I might prefer to hear, Competitor. Your first statement came closer to my hidden mark; your oh-so-Bayesian disclaimer fell wide... The factor I had in mind, Brennan, was that Eld scientists thought it was acceptable to take thirty years to solve a problem. Their entire social process of science was based on getting to the truth eventually. A wrong theory got discarded eventually - once the next generation of students grew up familiar with the replacement. Work expands to fill the time allotted, as the saying goes. But people can think important thoughts in far less than thirty years, if they expect speed of themselves." Jeffreyssai suddenly slammed down a hand on the arm of Brennan's chair. "How long do you have to dodge a thrown knife?"

"Very little time, sensei!"

"Less than a second! Two opponents are attacking you! How long do you have to guess who's more dangerous?"

"Less than a second, sensei!"

"The two opponents have split up and are attacking two of your girlfriends! How long do you have to decide which one you truly love?"

"Less than a second, sensei!'

"A new argument shows your precious theory is flawed! How long does it take you to change your mind?"

"Less than a second, sensei!"

"WRONG! DON'T GIVE ME THE WRONG ANSWER JUST BECAUSE IT FITS A CONVENIENT PATTERN AND I SEEM TO EXPECT IT OF YOU! How long does it really take, Brennan?"

Sweat was forming on Brennan's back, but he stopped and actually thought about it -

"ANSWER, BRENNAN!"

"No sensei! I'm not finished thinking sensei! An answer would be premature! Sensei!"

"Very good! Continue! But don't take thirty years!"

Brennan breathed deeply, reforming his thoughts. He finally said, "Realistically, sensei, the best-case scenario is that I would see the problem immediately; use the discipline of suspending judgment; try to re-accumulate all the evidence before continuing; and depending on how emotionally attached I had been to the theory, use the crisis-of-belief technique to ensure I could genuinely go either way. So at least five minutes and perhaps up to an hour."

"Good! You actually thought about it that time! Think about it every time! Break patterns! In the days of Eld Science, Brennan, it was not uncommon for a grant agency to spend six months reviewing a proposal. They permitted themselves the time! You are being graded on your speed, Brennan! The question is not whether you get there eventually! Anyone can find the truth in five thousand years! You need to move faster!"

"Yes, sensei!"

"Now, Brennan, have you just learned something new?"

"Yes, sensei!"

"How long did it take you to learn this new thing?"

An arbitrary choice there... "Less than a minute, sensei, from the boundary that seems most obvious."

"Less than a minute," Jeffreyssai repeated. "So, Brennan, how long do you think it should take to solve a major scientific problem, if you are not wasting any time?"

Now there was a trapped question if Brennan had ever heard one. There was no way to guess what time period Jeffreyssai had in mind - what the sensei would consider too long, or too short. Which meant that the only way out was to just try for the genuine truth; this would offer him the defense of honesty, little defense though it was. "One year, sensei?"

"Do you think it could be done in one month, Brennan? In a case, let us stipulate, where in principle you already have enough experimental evidence to determine an answer, but not so much experimental evidence that you can afford to make errors in interpreting it."

Again, no way to guess which answer Jeffreyssai might want... "One month seems like an unrealistically short time to me, sensei."

"A short time?" Jeffreyssai said incredulously. "How many minutes in thirty days? Hiriwa?"

"43200, sensei," she answered. "If you assume sixteen-hour waking periods and daily sleep, then 28800 minutes."

"Assume, Brennan, that it takes five whole minutes to think an original thought, rather than learning it from someone else. Does even a major scientific problem require 5760 distinct insights?"

"I confess, sensei," Brennan said slowly, "that I have never thought of it that way before... but do you tell me that is truly a realistic level of productivity?"

"No," said Jeffreyssai, "but neither is it realistic to think that a single problem requires 5760 insights. And yes, it has been done."

Jeffreyssai stepped back, and smiled benevolently. Every student in the room stiffened; they knew that smile. "Though none of you hit the particular answer that I had in mind, nonetheless your answers were as reasonable as mine. Except Styrlyn's, I'm afraid. Even Hiriwa's answer was not entirely wrong: the task of proposing new theories was once considered a sacred duty reserved for those of high status, there being a limited supply of problems in circulation, at that time. But Brennan's answer is particularly interesting, and I am minded to test his theory of motivation."

Oh, hell, Brennan said silently to himself. Jeffreyssai was gesturing for Brennan to stand up before the class.

When Brenann had risen, Jeffreyssai neatly seated himself in Brennan's chair.

"Brennan-sensei," Jeffreyssai said, "you have five minutes to think of something stunningly brilliant to say about the failure of Eld science on quantum physics. As for the rest of us, our job will be to gaze at you expectantly. I can only imagine how embarrassing it will be, should you fail to think of anything good."

Bastard. Brennan didn't say it aloud. Taji's face showed a certain amount of sympathy; Styrlyn held himself aloof from the game; but Yin was looking at him with sardonic interest. Worse, Hiriwa was gazing at him expectantly, assuming that he would rise to the challenge. And Jeffreyssai was gawking wide-eyed, waiting for the guru's words of wisdom. Screw you, sensei.

Brennan didn't panic. It was very, very, very far from being the scariest situation he'd ever faced. He took a moment to decide how to think; then thought.

At four minutes and thirty seconds, Brennan spoke. (There was an art to such things; as long as you were doing it anyway, you might as well make it look easy.)

"A woman of wisdom," Brennan said, "once told me that it is wisest to regard our past selves as fools beyond redemption - to see the people we once were as idiots entire. I do not necessarily say this myself; but it is what she said to me, and there is more than a grain of truth in it. As long as we are making excuses for the past, trying to make it look better, respecting it, we cannot make a clean break. It occurs to me that the rule may be no different for human civilizations. So I tried looking back and considering the Eld scientists as simple fools."

"Which they were not," Jeffreyssai said.

"Which they were not," Brennan continued. "In terms of raw intelligence, they undoubtedly exceeded me. But it occurred to me that a difficulty in seeing what Eld scientists did wrong, might have been in respecting the ancient and legendary names too highly. And that did indeed produce an insight."

"Enough introduction, Brennan," said Jeffreyssai. "If you found an insight, state it."

"Eld scientists were not trained..." Brennan paused. "No, untrained is not the concept. They were trained for the wrong task. At that time, there were no Conspiracies, no secret truths; as soon as Eld scientists solved a major problem, they published the solution to the world and each other. Truly scary and confusing open problems would have been in extremely rare supply, and used up the moment they were solved. So it would not have been possible to train Eld researchers to bring order out of scientific chaos. They would have been trained for something else - I'm not sure what -"

"Trained to manipulate whatever science had already been discovered," said Taji. "It was a difficult enough task for Eld teachers to train their students to use existing knowledge, or follow already-known methodologies; that was all Eld science teachers aspired to impart."

Brennan nodded. "Which is a very different matter from creating new science of their own. The Eld scientists faced with problems of quantum theory, might never have faced that kind of fear before - the dismay of not knowing. The Eld scientists might have seized on unsatisfactory answers prematurely, because they were accustomed to working with a neat, agreed-upon body of knowledge."

"Good, Brennan," murmured Jeffreyssai.

"But above all," Brennan continued, "an Eld scientist couldn't have practiced the actual problem the quantum scientists faced - that of resolving a major confusion. It was something you did once per lifetime if you were lucky, and as Hiriwa observed, Newton would no longer have been around. So while the Eld physicists who messed up quantum theory were not unintelligent, they were, in a strong sense, amateurs - ad-libbing the whole process of paradigm shift."

"And no probability theory," Hiriwa noted. "So anyone who did succeed at the problem would have no idea what they'd just done. They wouldn't be able to communicate it to anyone else, except vaguely."

"Yes," Styrlyn said. "And it was only a handful of people who could tackle the problem at all, with no training in doing so; those are the physicists whose names have passed down to us. A handful of people, making a handful of discoveries each. It would not have been enough to sustain a community. Each Eld scientist tackling a new paradigm shift would have needed to rediscover the rules from scratch."

Jeffreyssai rose from Brenann's desk. "Acceptable, Brennan; you surprise me, in fact. I shall have to give further thought to this method of yours." Jeffreyssai went to the classroom door, then looked back. "However, I did have in mind at least one other major flaw of Eld science, which none of you suggested. I expect to receive a list of possible flaws tomorrow. I expect the flaw I have in mind to be on the list. You have 480 minutes, excluding sleep time. I see five of you here. The challenge does not require more than 480 insights to solve, nor more than 96 insights in series."

And Jeffreyssai left the room.

Sequence: Quantum Physics

Timeless Control

The Dilemma: Science or Bayes?

Sequence: Challenging the Difficult

New Improved Lottery

The Quantum Physics Sequence

Referenced by: The Dilemma: Science or Bayes? • No Safe Defense, Not Even Science • Einstein's Speed • My Childhood Role Model • Class Project • The Quantum Physics Sequence • The Ultimate Source • Grasping Slippery Things • Optimization and the Singularity • Setting Up Metaethics • The Ritual • Logical or Connectionist AI? • True Sources of Disagreement • What I Think, If Not Why • (Moral) Truth in Fiction? • Teaching the Unteachable • A Sense That More Is Possible • Bayesians vs. Barbarians • My Way • Working Mantras

Original with comments: The Failures of Eld Science

The Dilemma: Science or Bayes?

Eliezer Yudkowsky, 13 May 2008 08:16AM

Followup to: If Many-Worlds Had Come First, The Failures of Eld Science

"Eli: You are writing a lot about physics recently. Why?"
-- Shane Legg (and several other people)

"In light of your QM explanation, which to me sounds perfectly logical, it seems obvious and normal that many worlds is overwhelmingly likely. It just seems almost too good to be true that I now get what plenty of genius quantum physicists still can't. [...] Sure I can explain all that away, and I still think you're right, I'm just suspicious of myself for believing the first believable explanation I met."
-- Recovering irrationalist

RI, you've got no idea how glad I was to see you post that comment.

Of course I had more than just one reason for spending all that time posting about quantum physics. I like having lots of hidden motives, it's the closest I can ethically get to being a supervillain.

But to give an example of a purpose I could only accomplish by discussing quantum physics...

In physics, you can get absolutely clear-cut issues. Not in the sense that the issues are trivial to explain. But if you try to apply Bayes to healthcare, or economics, you may not be able to formally lay out what is the simplest hypothesis, or what the evidence supports. But when I say "macroscopic decoherence is simpler than collapse" it is actually strict simplicity; you could write the two hypotheses out as computer programs and count the lines of code. Nor is the evidence itself in dispute.

I wanted a very clear example - Bayes says "zig", this is a zag - when it came time to break your allegiance to Science.

"Oh, sure," you say, "the physicists messed up the many-worlds thing, but give them a break, Eliezer! No one ever claimed that the social process of science was perfect. People are human; they make mistakes."

But the physicists who refuse to adopt many-worlds aren't disobeying the rules of Science. They're obeying the rules of Science.

The tradition handed down through the generations says that a new physics theory comes up with new experimental predictions that distinguish it from the old theory. You perform the test, and the new theory is confirmed or falsified. If it's confirmed, you hold a huge celebration, call the newspapers, and hand out Nobel Prizes for everyone; any doddering old emeritus professors who refuse to convert are quietly humored. If the theory is disconfirmed, the lead proponent publicly recants, and gains a reputation for honesty.

This is not how things do work in science; rather it is how things are supposed to work in Science. It's the ideal to which all good scientists aspire.

Now many-worlds comes along, and it doesn't seem to make any new predictions relative to the old theory. That's suspicious. And there's all these other worlds, but you can't see them. That's really suspicious. It just doesn't seem scientific.

If you got as far as RI - so that many-worlds now seems perfectly logical, obvious and normal - and you also started out as a Traditional Rationalist, then you should be able to switch back and forth between the Scientific view and the Bayesian view, like a Necker Cube.

So now put on your Science Goggles - you've still got them around somewhere, right? Forget everything you know about Kolmogorov complexity, Solomonoff induction or Minimum Message Lengths. That's not part of the traditional training. You just eyeball something to see how "simple" it looks. The word "testable" doesn't conjure up a mental image of Bayes's Theorem governing probability flows; it conjures up a mental image of being in a lab, performing an experiment, and having the celebration (or public recantation) afterward.

Science-Goggles on: The current quantum theory has passed all experimental tests so far. Many-Worlds doesn't make any new testable predictions - the amazing new phenomena it predicts are all hidden away where we can't see them. You can get along fine without supposing the other worlds, and that's just what you should do. The whole thing smacks of science fiction. But it must be admitted that quantum physics is a very deep and very confusing issue, and who knows what discoveries might be in store? Call me when Many-Worlds makes a testable prediction.

Science-Goggles off, Bayes-Goggles back on:

Bayes-Goggles on: The simplest quantum equations that cover all known evidence don't have a special exception for human-sized masses. There isn't even any reason to ask that particular question. Next!

Okay, so is this a problem we can fix in five minutes with some duct tape and superglue?

No.

Huh? Why not just teach new graduating classes of scientists about Solomonoff induction and Bayes's Rule?

Centuries ago, there was a widespread idea that the Wise could unravel the secrets of the universe just by thinking about them, while to go out and look at things was lesser, inferior, naive, and would just delude you in the end. You couldn't trust the way things looked - only thought could be your guide.

Science began as a rebellion against this Deep Wisdom. At the core is the pragmatic belief that human beings, sitting around in their armchairs trying to be Deeply Wise, just drift off into never-never land. You couldn't trust your thoughts. You had to make advance experimental predictions - predictions that no one else had made before - run the test, and confirm the result. That was evidence. Sitting in your armchair, thinking about what seemed reasonable... would not be taken to prejudice your theory, because Science wasn't an idealistic belief about pragmatism, or getting your hands dirty. It was, rather, the dictum that experiment alone would decide. Only experiments could judge your theory - not your nationality, or your religious professions, or the fact that you'd invented the theory in your armchair. Only experiments! If you sat in your armchair and came up with a theory that made a novel prediction, and experiment confirmed the prediction, then we would care about the result of the experiment, not where your hypothesis came from.

That's Science. And if you say that Many-Worlds should replace the immensely successful Copenhagen Interpretation, adding on all these twin Earths that can't be observed, just because it sounds more reasonable and elegant - not because it crushed the old theory with a superior experimental prediction - then you're undoing the core scientific rule that prevents people from running out and putting angels into all the theories, because angels are more reasonable and elegant.

You think teaching a few people about Solomonoff induction is going to solve that problem? Nobel laureate Robert Aumann - who first proved that Bayesian agents with similar priors cannot agree to disagree - is a believing Orthodox Jew. Aumann helped a project to test the Torah for "Bible codes", hidden prophecies from God - and concluded that the project had failed to confirm the codes' existence. Do you want Aumann thinking that once you've got Solomonoff induction, you can forget about the experimental method? Do you think that's going to help him? And most scientists out there will not rise to the level of Robert Aumann.

Okay, Bayes-Goggles back on. Are you really going to believe that large parts of the wavefunction disappear when you can no longer see them? As a result of the only non-linear non-unitary non-differentiable non-CPT-symmetric acausal faster-than-light informally-specified phenomenon in all of physics? Just because, by sheer historical contingency, the stupid version of the theory was proposed first?

Are you going to make a major modification to a scientific model, and believe in zillions of other worlds you can't see, without a defining moment of experimental triumph over the old model?

Or are you going to reject probability theory?

Will you give your allegiance to Science, or to Bayes?

Michael Vassar once observed (tongue-in-cheek) that it was a good thing that a majority of the human species believed in God, because otherwise, he would have a very hard time rejecting majoritarianism. But since the majority opinion that God exists is simply unbelievable, we have no choice but to reject the extremely strong philosophical arguments for majoritarianism.

You can see (one of the reasons) why I went to such lengths to explain quantum theory. Those who are good at math should now be able to visualize both macroscopic decoherence, and the probability theory of simplicity and testability - get the insanity of a global single world on a gut level.

I wanted to present you with a nice, sharp dilemma between rejecting the scientific method, or embracing insanity.

Why? I'll give you a hint: It's not just because I'm evil. If you would guess my motives here, think beyond the first obvious answer.

PS: If you try to come up with clever ways to wriggle out of the dilemma, you're just going to get shot down in future posts. You have been warned.

Sequence: Quantum Physics

The Failures of Eld Science

Science Doesn't Trust Your Rationality

Referenced by: Science Doesn't Trust Your Rationality • Why Quantum? • The Quantum Physics Sequence

Original with comments: The Dilemma: Science or Bayes?

Science Doesn't Trust Your Rationality

Eliezer Yudkowsky, 14 May 2008 02:13AM

Followup to: The Dilemma: Science or Bayes?

Scott Aaronson suggests that Many-Worlds and libertarianism are similar in that they are both cases of bullet-swallowing, rather than bullet-dodging:

Libertarianism and MWI are both are grand philosophical theories that start from premises that almost all educated people accept (quantum mechanics in the one case, Econ 101 in the other), and claim to reach conclusions that most educated people reject, or are at least puzzled by (the existence of parallel universes / the desirability of eliminating fire departments).

Now there's an analogy that would never have occurred to me.

I've previously argued that Science rejects Many-Worlds but Bayes accepts it. (Here, "Science" is capitalized because we are talking about the idealized form of Science, not just the actual social process of science.)

It furthermore seems to me that there is a deep analogy between (small-'l') libertarianism and Science:

1. Both are based on a pragmatic distrust of reasonable-sounding arguments.

2. Both try to build systems that are more trustworthy than the people in them.

3. Both accept that people are flawed, and try to harness their flaws to power the system.

The core argument for libertarianism is historically motivated distrust of lovely theories of "How much better society would be, if we just made a rule that said XYZ." If that sort of trick actually worked, then more regulations would correlate to higher economic growth as society moved from local to global optima. But when some person or interest group gets enough power to start doing everything they think is a good idea, history says that what actually happens is Revolutionary France or Soviet Russia.

The plans that in lovely theory should have made everyone happy ever after, don't have the results predicted by reasonable-sounding arguments. And power corrupts, and attracts the corrupt.

So you regulate as little as possible, because you can't trust the lovely theories and you can't trust the people who implement them.

You don't shake your finger at people for being selfish. You try to build an efficient system of production out of selfish participants, by requiring transactions to be voluntary. So people are forced to play positive-sum games, because that's how they get the other party to sign the contract. With violence restrained and contracts enforced, individual selfishness can power a globally productive system.

Of course none of this works quite so well in practice as in theory, and I'm not going to go into market failures, commons problems, etc. The core argument for libertarianism is not that libertarianism would work in a perfect world, but that it degrades gracefully into real life. Or rather, degrades less awkwardly than any other known economic principle. (People who see Libertarianism as the perfect solution for perfect people, strike me as kinda missing the point of the "pragmatic distrust" thing.)

Science first came to know itself as a rebellion against trusting the word of Aristotle. If the people of that revolution had merely said, "Let us trust ourselves, not Aristotle!" they would have flashed and faded like the French Revolution.

But the Scientific Revolution lasted because - like the American Revolution - the architects propounded a stranger philosophy: "Let us trust no one! Not even ourselves!"

In the beginning came the idea that we can't just toss out Aristotle's armchair reasoning and replace it with different armchair reasoning. We need to talk to Nature, and actually listen to what It says in reply. This, itself, was a stroke of genius.

But then came the challenge of implementation. People are stubborn, and may not want to accept the verdict of experiment. Shall we shake a disapproving finger at them, and say "Naughty"?

No; we assume and accept that each individual scientist may be crazily attached to their personal theories. Nor do we assume that anyone can be trained out of this tendency - we don't try to choose Eminent Judges who are supposed to be impartial.

Instead, we try to harness the individual scientist's stubborn desire to prove their personal theory, by saying: "Make a new experimental prediction, and do the experiment. If you're right, and the experiment is replicated, you win." So long as scientists believe this is true, they have a motive to do experiments that can falsify their own theories. Only by accepting the possibility of defeat is it possible to win. And any great claim will require replication; this gives scientists a motive to be honest, on pain of great embarrassment.

And so the stubbornness of individual scientists is harnessed to produce a steady stream of knowledge at the group level. The System is somewhat more trustworthy than its parts.

Libertarianism secretly relies on most individuals being prosocial enough to tip at a restaurant they won't ever visit again. An economy of genuinely selfish human-level agents would implode. Similarly, Science relies on most scientists not committing sins so egregious that they can't rationalize them away.

To the extent that scientists believe they can promote their theories by playing academic politics - or game the statistical methods to potentially win without a chance of losing - or to the extent that nobody bothers to replicate claims - science degrades in effectiveness. But it degrades gracefully, as such things go.

The part where the successful predictions belong to the theory and theorists who originally made them, and cannot just be stolen by a theory that comes along later - without a novel experimental prediction - is an important feature of this social process.

The final upshot is that Science is not easily reconciled with probability theory. If you do a probability-theoretic calculation correctly, you're going to get the rational answer. Science doesn't trust your rationality, and it doesn't rely on your ability to use probability theory as the arbiter of truth. It wants you to set up a definitive experiment.

Regarding Science as a mere approximation to some probability-theoretic ideal of rationality... would certainly seem to be rational. There seems to be an extremely reasonable-sounding argument that Bayes's Theorem is the hidden structure that explains why Science works. But to subordinate Science to the grand schema of Bayesianism, and let Bayesianism come in and override Science's verdict when that seems appropriate, is not a trivial step!

Science is built around the assumption that you're too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn't need a social process of science... right?

So, are you going to believe in faster-than-light quantum "collapse" fairies after all? Or do you think you're smarter than that?

Sequence: Quantum Physics

The Dilemma: Science or Bayes?

When Science Can't Help

Referenced by: When Science Can't Help • Do Scientists Already Know This Stuff? • No Safe Defense, Not Even Science • Faster Than Science • Einstein's Speed • Why Quantum? • The Quantum Physics Sequence

Original with comments: Science Doesn't Trust Your Rationality

When Science Can't Help

Eliezer Yudkowsky, 15 May 2008 07:24AM

Followup to: Science Doesn't Trust Your Rationality, My Wild and Reckless Youth

Once upon a time, a younger Eliezer had a stupid theory. Let's say that Eliezer_18's stupid theory was that consciousness was caused by closed timelike curves hiding in quantum gravity. This isn't the whole story, not even close, but it will do for a start.

And there came a point where I looked back, and realized:

1. I had carefully followed everything I'd been told was Traditionally Rational, in the course of going astray. For example, I'd been careful to only believe in stupid theories that made novel experimental predictions, e.g., that neuronal microtubules would be found to support coherent quantum states.

2. Science would have been perfectly fine with my spending ten years trying to test my stupid theory, only to get a negative experimental result, so long as I then said, "Oh, well, I guess my theory was wrong."

From Science's perspective, that is how things are supposed to work - happy fun for everyone. You admitted your error! Good for you! Isn't that what Science is all about?

But what if I didn't want to waste ten years?

Well... Science didn't have much to say about that. How could Science say which theory was right, in advance of the experimental test? Science doesn't care where your theory comes from - it just says, "Go test it."

This is the great strength of Science, and also its great weakness.

Gray Area asked:

Eliezer, why are you concerned with untestable questions?

Because questions that are easily immediately tested are hard for Science to get wrong.

I mean, sure, when there's already definite unmistakable experimental evidence available, go with it. Why on Earth wouldn't you?

But sometimes a question will have very large, very definite experimental consequences in your future - but you can't easily test it experimentally right now - and yet there is a strong rational argument.

Macroscopic quantum superpositions are readily testable: It would just take nanotechnologic precision, very low temperatures, and a nice clear area of interstellar space. Oh, sure, you can't do it right now, because it's too expensive or impossible for today's technology or something like that - but in theory, sure! Why, maybe someday they'll run whole civilizations on macroscopically superposed quantum computers, way out in a well-swept volume of a Great Void. (Asking what quantum non-realism says about the status of any observers inside these computers, helps to reveal the underspecification of quantum non-realism.)

This doesn't seem immediately pragmatically relevant to your life, I'm guessing, but it establishes the pattern: Not everything with future consequences is cheap to test now.

Evolutionary psychology is another example of a case where rationality has to take over from science. While theories of evolutionary psychology form a connected whole, only some of those theories are readily testable experimentally. But you still need the other parts of the theory, because they form a connected web that helps you to form the hypotheses that are actually testable - and then the helper hypotheses are supported in a Bayesian sense, but not supported experimentally. Science would render a verdict of "not proven" on individual parts of a connected theoretical mesh that is experimentally productive as a whole. We'd need a new kind of verdict for that, something like "indirectly supported".

Or what about cryonics?

Cryonics is an archetypal example of an extremely important issue (150,000 people die per day) that will have huge consequences in the foreseeable future, but doesn't offer definite unmistakable experimental evidence that we can get right now.

So do you say, "I don't believe in cryonics because it hasn't been experimentally proven, and you shouldn't believe in things that haven't been experimentally proven?"

Well, from a Bayesian perspective, that's incorrect. Absence of evidence is evidence of absence only to the degree that we could reasonably expect the evidence to appear. If someone is trumpeting that snake oil cures cancer, you can reasonably expect that, if the snake oil was actually curing cancer, some scientist would be performing a controlled study to verify it - that, at the least, doctors would be reporting case studies of amazing recoveries - and so the absence of this evidence is strong evidence of absence. But "gaps in the fossil record" are not strong evidence against evolution; fossils form only rarely, and even if an intermediate species did in fact exist, you cannot expect with high probability that Nature will obligingly fossilize it and that the fossil will be discovered.

Reviving a cryonically frozen mammal is just not something you'd expect to be able to do with modern technology, even if future nanotechnologies could in fact perform a successful revival. That's how I see Bayes seeing it.

Oh, and as for the actual arguments for cryonics - I'm not going to go into those at the moment. But if you followed the physics and anti-Zombie sequences, it should now seem a lot more plausible, that whatever preserves the pattern of synapses, preserves as much of "you" as is preserved from one night's sleep to morning's waking.

Now, to be fair, someone who says, "I don't believe in cryonics because it hasn't been proven experimentally" is misapplying the rules of Science; this is not a case where science actually gives the wrong answer. In the absence of a definite experimental test, the verdict of science here is "Not proven". Anyone who interprets that as a rejection is taking an extra step outside of science, not a misstep within science.

John McCarthy's Wikiquotes page has him saying, "Your statements amount to saying that if AI is possible, it should be easy. Why is that?" The Wikiquotes page doesn't say what McCarthy was responding to, but I could venture a guess.

The general mistake probably arises because there are cases where the absence of scientific proof is strong evidence - because an experiment would be readily performable, and so failure to perform it is itself suspicious. (Though not as suspicious as I used to think - with all the strangely varied anecdotal evidence coming in from respected sources, why the hell isn't anyone testing Seth Roberts's theory of appetite suppression?)

Another confusion factor may be that if you test Pharmaceutical X on 1000 subjects and find that 56% of the control group and 57% of the experimental group recover, some people will call that a verdict of "Not proven". I would call it an experimental verdict of "Pharmaceutical X doesn't work well, if at all". Just because this verdict is theoretically retractable in the face of new evidence, doesn't make it ambiguous.

In any case, right now you've got people dismissing cryonics out of hand as "not scientific", like it was some kind of pharmaceutical you could easily administer to 1000 patients and see what happened. "Call me when cryonicists actually revive someone," they say; which, as Mike Li observes, is like saying "I refuse to get into this ambulance; call me when it's actually at the hospital". Maybe Martin Gardner warned them against believing in strange things without experimental evidence. So they wait for the definite unmistakable verdict of Science, while their family and friends and 150,000 people per day are dying right now, and might or might not be savable -

- a calculated bet you could only make rationally.

The drive of Science is to obtain a mountain of evidence so huge that not even fallible human scientists can misread it. But even that sometimes goes wrong, when people become confused about which theory predicts what, or bake extremely-hard-to-test components into an early version of their theory. And sometimes you just can't get clear experimental evidence at all.

Either way, you have to try to do the thing that Science doesn't trust anyone to do - think rationally, and figure out the answer before you get clubbed over the head with it.

(Oh, and sometimes a disconfirming experimental result looks like: "Your entire species has just been wiped out! You are now scientifically required to relinquish your theory. If you publicly recant, good for you! Remember, it takes a strong mind to give up strongly held beliefs. Feel free to try another hypothesis next time!")

Sequence: Quantum Physics

Science Doesn't Trust Your Rationality

Science Isn't Strict Enough

Referenced by: Science Isn't Strict Enough • The Quantum Physics Sequence

Original with comments: When Science Can't Help

Science Isn't Strict Enough

Eliezer Yudkowsky, 16 May 2008 06:51AM

Followup to: When Science Can't Help

Once upon a time, a younger Eliezer had a stupid theory. Eliezer_18 was careful to follow the precepts of Traditional Rationality that he had been taught; he made sure his stupid theory had experimental consequences. Eliezer_18 professed, in accordance with the virtues of a scientist he had been taught, that he wished to test his stupid theory.

This was all that was required to be virtuous, according to what Eliezer_18 had been taught was virtue in the way of science.

It was not even remotely the order of effort that would have been required to get it right.

The traditional ideals of Science too readily give out gold stars. Negative experimental results are also knowledge, so everyone who plays gets an award. So long as you can think of some kind of experiment that tests your theory, and you do the experiment, and you accept the results, you've played by the rules; you're a good scientist.

You didn't necessarily get it right, but you're a nice science-abiding citizen.

(I note at this point that I am speaking of Science, not the social process of science as it actually works in practice, for two reasons. First, I went astray in trying to follow the ideal of Science - it's not like I was shot down by a journal editor with a grudge, and it's not like I was trying to imitate the flaws of academia. Second, if I point out a problem with the ideal as it is traditionally preached, real-world scientists are not forced to likewise go astray!)

Science began as a rebellion against grand philosophical schemas and armchair reasoning. So Science doesn't include a rule as to what kinds of hypotheses you are and aren't allowed to test; that is left up to the individual scientist. Trying to guess that a priori, would require some kind of grand philosophical schema, and reasoning in advance of the evidence. As a social ideal, Science doesn't judge you as a bad person for coming up with heretical hypotheses; honest experiments, and acceptance of the results, is virtue unto a scientist.

As long as most scientists can manage to accept definite, unmistakable, unambiguous experimental evidence, science can progress. It may happen too slowly - it may take longer than it should - you may have to wait for a generation of elders to die out - but eventually, the ratchet of knowledge clicks forward another notch. Year by year, decade by decade, the wheel turns forward. It's enough to support a civilization.

So that's all that Science really asks of you - the ability to accept reality when you're beat over the head with it. It's not much, but it's enough to sustain a scientific culture.

Contrast this to the notion we have in probability theory, of an exact quantitative rational judgment. If 1% of women presenting for a routine screening have breast cancer, and 80% of women with breast cancer get positive mammographies, and 10% of women without breast cancer get false positives, what is the probability that a routinely screened woman with a positive mammography has breast cancer? 7.5%. You cannot say, "I believe she doesn't have breast cancer, because the experiment isn't definite enough." You cannot say, "I believe she has breast cancer, because it is wise to be pessimistic and that is what the only experiment so far seems to indicate." 7.5% is the rational estimate given this evidence, not 7.4% or 7.6%. The laws of probability are laws.

It is written in the Twelve Virtues, of the third virtue, lightness:

If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims. For you cannot make a true map of a city by sitting in your bedroom with your eyes shut and drawing lines upon paper according to impulse. You must walk through the city and draw lines on paper that correspond to what you see. If, seeing the city unclearly, you think that you can shift a line just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.

In Science, when it comes to deciding which hypotheses to test, the morality of Science gives you personal freedom of what to believe, so long as it isn't already ruled out by experiment, and so long as you move to test your hypothesis. Science wouldn't try to give an official verdict on the best hypothesis to test, in advance of the experiment. That's left up to the conscience of the individual scientist.

Where definite experimental evidence exists, Science tells you to bow your stubborn neck and accept it. Otherwise, Science leaves it up to you. Science gives you room to wander around within the boundaries of the experimental evidence, according to your whims.

And this is not easily reconciled with Bayesianism's notion of an exactly right probability estimate, one with no flex or room for whims, that exists both before and after the experiment. It doesn't match well with the ancient and traditional reason for Science - the distrust of grand schemas, the presumption that people aren't rational enough to get things right without definite and unmistakable experimental evidence. If we were all perfect Bayesians, we wouldn't need a social process of science.

Nonetheless, around the time I realized my big mistake, I had also been studying Kahneman and Tversky and Jaynes. I was learning a new Way, stricter than Science. A Way that could criticize my folly, in a way that Science never could. A Way that could have told me, what Science would never have said in advance: "You picked the wrong hypothesis to test, dunderhead."

But the Way of Bayes is also much harder to use than Science. It puts a tremendous strain on your ability to hear tiny false notes, where Science only demands that you notice an anvil dropped on your head.

In Science you can make a mistake or two, and another experiment will come by and correct you; at worst you waste a couple of decades.

But if you try to use Bayes even qualitatively - if you try to do the thing that Science doesn't trust you to do, and reason rationally in the absence of overwhelming evidence - it is like math, in that a single error in a hundred steps can carry you anywhere. It demands lightness, evenness, precision, perfectionism.

There's a good reason why Science doesn't trust scientists to do this sort of thing, and asks for further experimental proof even after someone claims they've worked out the right answer based on hints and logic.

But if you would rather not waste ten years trying to prove the wrong theory, you'll need to essay the vastly more difficult problem: listening to evidence that doesn't shout in your ear.

(For the benefit of those in the audience who have not been following along this whole time: Even if you can't look up the priors for a problem in the Handbook of Chemistry and Physics - even if there's no Authoritative Source telling you what the priors are - that doesn't mean you get a free, personal choice of making the priors whatever you want. It means you have a new guessing problem which you must carry out to the best of your ability.

If the mind, as a cognitive engine, could generate correct estimates by fiddling with priors according to whims, you could know things without looking them, or even alter them without touching them. But the mind is not magic. The rational probability estimate has no room for any decision based on whim, even when it seems that you don't know the priors.

Similarly, if the Bayesian answer is difficult to compute, that doesn't mean that Bayes is inapplicable; it means you don't know what the Bayesian answer is. Bayesian probability theory is not a toolbox of statistical methods, it's the law that governs any tool you use, whether or not you know it, whether or not you can calculate it.

As for using Bayesian methods on huge, highly general hypothesis spaces - like, "Here's the data from every physics experiment ever; now, what would be a good Theory of Everything?" - if you knew how to do that in practice, you wouldn't be a statistician, you would be an Artificial General Intelligence programmer. But that doesn't mean that human beings, in modeling the universe using human intelligence, are violating the laws of physics / Bayesianism by generating correct guesses without evidence.)

Added: Nick Tarleton says:

The problem is encouraging a private, epistemic standard as lax as the social one.

which pinpoints the problem I was trying to indicate much better than I did.

Sequence: Quantum Physics

When Science Can't Help

Do Scientists Already Know This Stuff?

Referenced by: Do Scientists Already Know This Stuff? • No Safe Defense, Not Even Science • Why Quantum? • The Quantum Physics Sequence • Contaminated by Optimism • Raised in Technophilia • My Bayesian Enlightenment

Original with comments: Science Isn't Strict Enough

Do Scientists Already Know This Stuff?

Eliezer Yudkowsky, 17 May 2008 02:25AM

Followup to: Science Isn't Strict Enough

poke alleges:

"Being able to create relevant hypotheses is an important skill and one a scientist spends a great deal of his or her time developing. It may not be part of the traditional description of science but that doesn't mean it's not included in the actual social institution of science that produces actual real science here in the real world; it's your description and not science that is faulty."

I know I've been calling my younger self "stupid" but that is a figure of speech; "unskillfully wielding high intelligence" would be more precise. Eliezer_18 was not in the habit of making obvious mistakes - it's just that his "obvious" wasn't my "obvious".

No, I did not go through the traditional apprenticeship. But when I look back, and see what Eliezer_18 did wrong, I see plenty of modern scientists making the same mistakes. I cannot detect any sign that they were better warned than myself.

Sir Roger Penrose - a world-class physicist - still thinks that consciousness is caused by quantum gravity. I expect that no one ever warned him against mysterious answers to mysterious questions - only told him his hypotheses needed to be falsifiable and have empirical consequences. Just like Eliezer_18.

"Consciousness is caused by quantum gravity" has testable implications: It implies that you should be able to look at neurons and discover a coherent quantum superposition (whose collapse?) contributes to information-processing, and that you won't ever be able to reproduce a neuron's input-output behavior using a computable microanatomical simulation...

...but even after you say "Consciousness is caused by quantum gravity", you don't anticipate anything about how your brain thinks "I think therefore I am!" or the mysterious redness of red, that you did not anticipate before, even though you feel like you know a cause of it. This is a tremendous danger sign, I now realize, but it's not the danger sign that I was warned against, and I doubt that Penrose was ever told of it by his thesis advisor. For that matter, I doubt that Niels Bohr was ever warned against it when it came time to formulate the Copenhagen Interpretation.

As far as I can tell, the reason Eliezer_18 and Sir Roger Penrose and Niels Bohr were not warned, is that no standard warning exists.

I did not generalize the concept of "mysterious answers to mysterious questions", in that many words, until I was writing a Bayesian analysis of what distinguishes technical, nontechnical and semitechnical scientific explanations. Now, the final output of that analysis, can be phrased nontechnically in terms of four danger signs:

- First, the explanation acts as a curiosity-stopper rather than an anticipation-controller.

- Second, the hypothesis has no moving parts - the secret sauce is not a specific complex mechanism, but a blankly solid substance or force.

- Third, those who proffer the explanation cherish their ignorance; they speak proudly of how the phenomenon defeats ordinary science or is unlike merely mundane phenomena.

- Fourth, even after the answer is given, the phenomenon is still a mystery and possesses the same quality of wonderful inexplicability that it had at the start.

In principle, all this could have been said in the immediate aftermath of vitalism. Just like elementary probability theory could have been invented by Archimedes, or the ancient Greeks could have theorized natural selection. But in fact no one ever warned me against any of these four dangers, in those terms - the closest being the warning that hypotheses should have testable consequences. And I didn't conceptualize the warning signs explicitly until I was trying to think of the whole affair in terms of probability distributions - some degree of overkill was required.

I simply have no reason to believe that these warnings are passed down in scientific apprenticeships - certainly not to a majority of scientists. Among other things, it is advice for handling situations of confusion and despair, scientific chaos. When would the average scientist or average mentor have an opportunity to use that kind of technique?

We just got through discussing the single-world fiasco in physics. Clearly, no one told them about the formal definition of Occam's Razor, in whispered apprenticeship or otherwise.

There is a known effect where great scientists have multiple great students. This may well be due to the mentors passing on skills that they can't describe. But I don't think that counts as part of standard science. And if the great mentors haven't been able to put their guidance into words and publish it generally, that's not a good sign for how well these things are understood.

Reasoning in the absence of definite evidence without going instantaneously completely wrong is really really hard. When you're learning in school, you can miss one point, and then be taught fifty other points that happen to be correct. When you're reasoning out new knowledge in the absence of crushingly overwhelming guidance, you can miss one point and wake up in Outer Mongolia fifty steps later.

I am pretty sure that scientists who switch off their brains and relax with some comfortable nonsense as soon as they leave their own specialties, do not realize that minds are engines and that there is a causal story behind every trustworthy belief. Nor, I suspect, were they ever told that there is an exact rational probability given a state of evidence, which has no room for whims; even if you can't calculate the answer, and even if you don't hear any authoritative command for what to believe.

I doubt that scientists who are asked to pontificate on the future by the media, who sketch amazingly detailed pictures of Life in 2050, were ever taught about the conjunction fallacy. Or how the representativeness heuristic can make more detailed stories seem more plausible, even as each extra detail drags down the probability. The notion of every added detail needing its own support - of not being able to make up big detailed stories that sound just like the detailed stories you were taught in science or history class - is absolutely vital to precise thinking in the absence of definite evidence. But how would a notion like that get into the standard scientific apprenticeship? The cognitive bias was uncovered only a few decades ago, and not popularized until very recently.

Then there's affective death spirals around notions like "emergence" or "complexity" which are sufficiently vaguely defined that you can say lots of nice things about them. There's whole academic subfields built around the kind of mistakes that Eliezer_18 used to make! (Though I never fell for the "emergence" thing.)

I sometimes say that the goal of science is to amass such an enormous mountain of evidence that not even scientists can ignore it: and that this is the distinguishing feature of a scientist, a non-scientist will ignore it anyway.

If there can exist some amount of evidence so crushing that you finally despair, stop making excuses and just give up - drop the old theory and never mention it again - then this is all it takes to let the ratchet of Science turn forward over time, and raise up a technological civilization. Contrast to religion.

Books by Carl Sagan and Martin Gardner and the other veins of Traditional Rationality are meant to accomplish this difference: to transform someone from a non-scientist into a potential scientist, and guard them from experimentally disproven madness.

What further training does a professional scientist get? Some frequentist stats classes on how to calculate statistical significance. Training in standard techniques that will let them churn out papers within a solidly established paradigm.

If Science demanded more than this from the average scientist, I don't think it would be possible for Science to get done. We have problems enough from people who sneak in without the drop-dead-basic qualifications.

Nick Tarleton summarized the resulting problem very well - better than I did, in fact: If you come up with a bizarre-seeming hypothesis not yet ruled out by the evidence, and try to test it experimentally, Science doesn't call you a bad person. Science doesn't trust its elders to decide which hypotheses "aren't worth testing". But this is a carefully lax social standard, and if you try to translate it into a standard of individual epistemic rationality, it lets you believe far too much. Dropping back into the analogy with pragmatic-distrust-based-libertarianism, it's the difference between "Cigarettes shouldn't be illegal" and "Go smoke a Marlboro".

Do you remember ever being warned against that mistake, in so many words? Then why wouldn't people make exactly that error? How many people will spontaneously go an extra mile and be even stricter with themselves? Some, but not many.

Many scientists will believe all manner of ridiculous things outside the laboratory, so long as they can convince themselves it hasn't been definitely disproven, or so long as they manage not to ask. Is there some standard lecture that grad students get, of which people see this folly, and ask, "Were they absent from class that day?" No, as far as I can tell.

Maybe if you're super lucky and get a famous mentor, they'll tell you rare personal secrets like "Ask yourself which are the important problems in your field, and then work on one of those, instead of falling into something easy and trivial" or "Be more careful than the journal editors demand; look for new ways to guard your expectations from influencing the experiment, even if it's not standard."

But I really don't think there's a huge secret standard scientific tradition of precision-grade rational reasoning on sparse evidence. Half of all the scientists out there still believe they believe in God! The more difficult skills are not standard!

Sequence: Quantum Physics

Science Isn't Strict Enough

No Safe Defense, Not Even Science

Referenced by: No Safe Defense, Not Even Science • A Premature Word on AI • The Quantum Physics Sequence • A Sense That More Is Possible

Original with comments: Do Scientists Already Know This Stuff?

No Safe Defense, Not Even Science

Eliezer Yudkowsky, 18 May 2008 05:19AM

Followup to: Science Doesn't Trust Your Rationality, Do Scientists Already Know This Stuff?

I don't ask my friends about their childhoods - I lack social curiosity - and so I don't know how much of a trend this really is:

Of the people I know who are reaching upward as rationalists, who volunteer information about their childhoods, there is a surprising tendency to hear things like: "My family joined a cult and I had to break out," or "One of my parents was clinically insane and I had to learn to filter out reality from their madness."

My own experience with growing up in an Orthodox Jewish family seems tame by comparison... but it accomplished the same outcome: It broke my core emotional trust in the sanity of the people around me.

Until this core emotional trust is broken, you don't start growing as a rationalist. I have trouble putting into words why this is so. Maybe any unusual skills you acquire - anything that makes you unusually rational - requires you to zig when other people zag. Maybe that's just too scary, if the world still seems like a sane place unto you.

Or maybe you don't bother putting in the hard work to be extra bonus sane, if normality doesn't scare the hell out of you.

I know that many aspiring rationalists seem to run into roadblocks around things like cryonics or many-worlds. Not that they don't see the logic; they see the logic and wonder, "Can this really be true, when it seems so obvious now, and yet none of the people around me believe it?"

Yes. Welcome to the Earth where ethanol is made from corn and environmentalists oppose nuclear power. I'm sorry.

(See also: Cultish Countercultishness. If you end up in the frame of mind of nervously seeking reassurance, this is never a good thing - even if it's because you're about to believe something that sounds logical but could cause other people to look at you funny.)

People who've had their trust broken in the sanity of the people around them, seem to be able to evaluate strange ideas on their merits, without feeling nervous about their strangeness. The glue that binds them to their current place has dissolved, and they can walk in some direction, hopefully forward.

Lonely dissent, I called it. True dissent doesn't feel like going to school wearing black; it feels like going to school wearing a clown suit.

That's what it takes to be the lone voice who says, "If you really think you know who's going to win the election, why aren't you picking up the free money on the Intrade prediction market?" while all the people around you are thinking, "It is good to be an individual and form your own opinions, the shoe commercials told me so."

Maybe in some other world, some alternate Everett branch with a saner human population, things would be different... but in this world, I've never seen anyone begin to grow as a rationalist until they make a deep emotional break with the wisdom of their pack.

Maybe in another world, things would be different. And maybe not. I'm not sure that human beings realistically can trust and think at the same time.

Once upon a time, there was something I trusted.

Eliezer_18 trusted Science.

Eliezer_18 dutifully acknowledged that the social process of science was flawed. Eliezer_18 dutifully acknowledged that academia was slow, and misallocated resources, and played favorites, and mistreated its precious heretics.

That's the convenient thing about acknowledging flaws in people who failed to live up to your ideal; you don't have to question the ideal itself.

But who could possibly be foolish enough to question, "The experimental method shall decide which hypothesis wins"?

Part of what fooled Eliezer_18 was a (major!) general problem he had, with an aversion to ideas that resembled things idiots had said. (See: Reversed stupidity is not intelligence.)

Eliezer_18 had seen plenty of people questioning the ideals of Science Itself, and they were all on the Dark Side. It's not like these people were saying, "Okay, here's how I think an excessive focus on definitive experiments misled physicists on the decoherence interpretation of quantum mechanics, and here's how you can do better using probability theory..." People who questioned the ideal of Science were invariably trying to sell you snake oil, or trying to safeguard their favorite form of stupidity from criticism, or trying to disguise their personal resignation as a Deeply Wise acceptance of futility.

And Eliezer_18 flinched away from being like these people; so he never questioned the ideal of the experimental method.

This is one reason that in these blog posts I am confronting, head-on, Science Itself. (Sure, someone may come back later and use these posts to paint me as a lunatic, but frankly, anyone who wishes to paint me as a lunatic already has more than enough material to misquote.) The natural assumption that anyone who seriously challenges Science is on the Dark Side, meant that this was one of very few things that Eliezer_18 didn't think about much. Of course those "few things" turned out to contain the evil black box surprises from hell.

If there'd been any other ideal that was a few centuries old, the young Eliezer would have looked at it and said, "I wonder if this is really right, and whether there's a way to do better." But not the ideal of Science. Science was the master idea, the idea that let you change ideas. You could question it, but you were meant to question it and then accept it, not actually say, "Wait! This is wrong!"

Thus, when once upon a time I came up with a stupid idea, I thought I was behaving virtuously if I made sure there was a novel prediction, and professed that I wished to test my idea experimentally. I thought I had done everything I was obliged to do.

So I thought I was safe - not safe from any particular external threat, but safe on some deeper level, like a child who trusts their parent and has obeyed all the parent's rules.

I'd long since been broken of trust in the sanity of my family or my teachers at school. And the other children weren't intelligent enough to compete with the conversations I could have with books. But I trusted the books, you see. I trusted that if I did what Richard Feynman told me to do, I would be safe. I never thought those words aloud, but it was how I felt.

When Eliezer_23 realized exactly how stupid the stupid theory had been - and that Traditional Rationality had not saved him from it - and that Science would have been perfectly okay with his wasting ten years testing the stupid idea, so long as afterward he admitted it was wrong...

...well, I'm not going to say it was a huge emotional convulsion. I don't really go in for that kind of drama. It simply became obvious that I'd been stupid.

That's the trust I'm trying to break in you. You are not safe. Ever.

Not even Science can save you. The ideals of Science were born centuries ago, in a time when no one knew anything about probability theory or cognitive biases. Science demands too little of you, it blesses your good intentions too easily, it is not strict enough, it only makes those injunctions that an average scientist can follow, it accepts slowness as a fact of life.

So don't think that if you only follow the rules of Science, that makes your reasoning defensible.

There is no known procedure you can follow that makes your reasoning defensible.

There is no known set of injunctions which you can satisfy, and know that you will not have been a fool.

There is no known morality-of-reasoning that you can do your best to obey, and know that you are thereby shielded from criticism.

No, not even if you turn to Bayescraft. It's much harder to use and you'll never be sure that you're doing it right.

The discipline of Bayescraft is younger by far than the discipline of Science. You will find no textbooks, no elderly mentors, no histories written of success and failure, no hard-and-fast rules laid down. You will have to study cognitive biases, and probability theory, and evolutionary psychology, and social psychology, and other cognitive sciences, and Artificial Intelligence - and think through for yourself how to apply all this knowledge to the case of correcting yourself, since it isn't yet in the textbooks.

You don't know what your own mind is really doing. They find a new cognitive bias every week and you're never sure if you've corrected for it, or overcorrected.

The formal math is impossible to apply. It doesn't break down as easily as John Q. Unbeliever thinks, but you're never really sure where the foundations come from. You don't know why the universe is simple enough to understand, or why any prior works for it. You don't know what your own priors are, let alone if they're any good.

One of the problems with Science is that it's too vague to really scare you. "Ideas should be tested by experiment." How can you go wrong with that?

On the other hand, if you have some math of probability theory laid out in front of you, and worse, you know you can't actually use it, then it becomes clear that you are trying to do something difficult, and that you might well be doing it wrong.

So you cannot trust.

And all this that I have said, will not be sufficient to break your trust. That won't happen until you get into your first real disaster from following The Rules, not from breaking them.

Eliezer_18 already had the notion that you were allowed to question Science. Why, of course the scientific method was not itself immune to questioning! For are we not all good rationalists? Are we not allowed to question everything?

It was the notion that you could actually in real life follow Science and fail miserably, that Eliezer_18 didn't really, emotionally believe was possible.

Oh, of course he said it was possible. Eliezer_18 dutifully acknowledged the possibility of error, saying, "I could be wrong, but..."

But he didn't think failure could happen in, you know, real life. You were supposed to look for flaws, not actually find them.

And this emotional difference is a terribly difficult thing to accomplish in words, and I fear there's no way I can really warn you.

Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you - that you have still been a fool, and no one warned you against it - that all the most important parts were left out of the guidance you received - that some of the most precious ideals you followed, steered you in the wrong direction -

- and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality -

- then you will be ready to start your journey as a rationalist. To take sole responsibility, to live without any trustworthy defenses, and to forge a higher Art than the one you were once taught.

No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand.

Post Scriptum: On reviewing a draft of this essay, I discovered a fairly inexcusable flaw in reasoning, which actually affects one of the conclusions drawn. I am leaving it in. Just in case you thought that taking my advice made you safe; or that you were supposed to look for flaws, but not find any.

And of course, if you look too hard for a flaw, and find a flaw that is not a real flaw, and cling to it to reassure yourself of how critical you are, you will only be worse off than before...

It is living with uncertainty - knowing on a gut level that there are flaws, they are serious and you have not found them - that is the difficult thing.

Sequence: Quantum Physics

Do Scientists Already Know This Stuff?

Changing the Definition of Science

Referenced by: A Premature Word on AI • Principles of Disagreement • Why Quantum? • The Quantum Physics Sequence • Passing the Recursive Buck • The Magnitude of His Own Folly • Beyond the Reach of God • Ethical Injunctions • Incremental Progress and the Valley • My Way

Original with comments: No Safe Defense, Not Even Science

Changing the Definition of Science

Eliezer Yudkowsky, 18 May 2008 06:07PM

New Scientist on changing the definition of science, ungated here:

Others believe such criticism is based on a misunderstanding. "Some people say that the multiverse concept isn't falsifiable because it's unobservable - but that's a fallacy," says cosmologist Max Tegmark of the Massachusetts Institute of Technology. He argues that the multiverse is a natural consequence of such eminently falsifiable theories as quantum theory and general relativity. As such, the multiverse theory stands or fails according to how well these other theories stand up to observational tests.
[...]
So if the simplicity of falsification is misleading, what should scientists be doing instead? Howson believes it is time to ditch Popper's notion of capturing the scientific process using deductive logic. Instead, the focus should be on reflecting what scientists actually do: gathering the weight of evidence for rival theories and assessing their relative plausibility.

Howson is a leading advocate for an alternative view of science based not on simplistic true/false logic, but on the far more subtle concept of degrees of belief. At its heart is a fundamental connection between the subjective concept of belief and the cold, hard mathematics of probability.

I'm a good deal less of a lonely iconoclast than I seem. Maybe it's just the way I talk.

The points of departure between myself and mainstream let's-reformulate-Science-as-Bayesianism is that:

(1) I'm not in academia and can censor myself a lot less when it comes to saying "extreme" things that others might well already be thinking.

(2) I think that just teaching probability theory won't be nearly enough. We'll have to synthesize lessons from multiple sciences like cognitive biases and social psychology, forming a new coherent Art of Bayescraft, before we are actually going to do any better in the real world than modern science. Science tolerates errors, Bayescraft does not. Nobel laureate Robert Aumann, who first proved that Bayesians with the same priors cannot agree to disagree, is a believing Orthodox Jew. Probability theory alone won't do the trick, when it comes to really teaching scientists. This is my primary point of departure, and it is not something I've seen suggested elsewhere.

(3) I think it is possible to do better in the real world. In the extreme case, a Bayesian superintelligence could use enormously less sensory information than a human scientist to come to correct conclusions. First time you ever see an apple fall down, you observe the position goes as the square of time, invent calculus, generalize Newton's Laws... and see that Newton's Laws involve action at a distance, look for alternative explanations with increased locality, invent relativistic covariance around a hypothetical speed limit, and consider that General Relativity might be worth testing. Humans do not process evidence efficiently - our minds are so noisy that it requires orders of magnitude more extra evidence to set us back on track after we derail. Our collective, academia, is even slower.

Sequence: Quantum Physics

No Safe Defense, Not Even Science

Faster Than Science

Referenced by: The Quantum Physics Sequence • Make an Extraordinary Effort • Raising the Sanity Waterline

Original with comments: Changing the Definition of Science

Conference on Global Catastrophic Risks

Eliezer Yudkowsky, 19 May 2008 10:13PM

FYI: The Oxford Future of Humanity Institute is holding a conference on global catastrophic risks on July 17-20, 2008, at Oxford (in the UK).

I'll be there, as will Robin Hanson and Nick Bostrom.

Deadline for registration is May 26th, 2008. Registration is ÂŁ60.

Referenced by: Posting May Slow

Original with comments: Conference on Global Catastrophic Risks

Faster Than Science

Eliezer Yudkowsky, 20 May 2008 12:19AM

Followup to: Science Doesn't Trust Your Rationality, Einstein's Arrogance

I sometimes say that the method of science is to amass such an enormous mountain of evidence that even scientists cannot ignore it; and that this is the distinguishing characteristic of a scientist, a non-scientist will ignore it anyway.

Max Planck was even less optimistic:

"A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it."

I am much tickled by this notion, because it implies that the power of science to distinguish truth from falsehood ultimately rests on the good taste of grad students.

The gradual increase in acceptance of many-worlds in academic physics, suggests that there are physicists who will only accept a new idea given some combination of epistemic justification, and a sufficiently large academic pack in whose company they can be comfortable. As more physicists accept, the pack grows larger, and hence more people go over their individual thresholds for conversion - with the epistemic justification remaining essentially the same.

But Science still gets there eventually, and this is sufficient for the ratchet of Science to move forward, and raise up a technological civilization.

Scientists can be moved by groundless prejudices, by undermined intuitions, by raw herd behavior - the panoply of human flaws. Each time a scientist shifts belief for epistemically unjustifiable reasons, it requires more evidence, or new arguments, to cancel out the noise.

The "collapse of the wavefunction" has no experimental justification, but it appeals to the (undermined) intuition of a single world. Then it may take an extra argument - say, that collapse violates Special Relativity - to begin the slow academic disintegration of an idea that should never have been assigned non-negligible probability in the first place.

From a Bayesian perspective, human academic science as a whole is a highly inefficient processor of evidence. Each time an unjustifiable argument shifts belief, you need an extra justifiable argument to shift it back. The social process of science leans on extra evidence to overcome cognitive noise.

A more charitable way of putting it is that scientists will adopt positions that are theoretically insufficiently extreme, compared to the ideal positions that scientists would adopt, if they were Bayesian AIs and could trust themselves to reason clearly.

But don't be too charitable. The noise we are talking about is not all innocent mistakes. In many fields, debates drag on for decades after they should have been settled. And not because the scientists on both sides refuse to trust themselves and agree they should look for additional evidence. But because one side keeps throwing up more and more ridiculous objections, and demanding more and more evidence, from an entrenched position of academic power, long after it becomes clear from which quarter the winds of evidence are blowing. (I'm thinking here about the debates surrounding the invention of evolutionary psychology, not about many-worlds.)

Is it possible for individual humans or groups to process evidence more efficiently - reach correct conclusions faster - than human academic science as a whole?

"Ideas are tested by experiment. That is the core of science." And this must be true, because if you can't trust Zombie Feynman, who can you trust?

Yet where do the ideas come from?

You may be tempted to reply, "They come from scientists. Got any other questions?" In Science you're not supposed to care where the hypotheses come from - just whether they pass or fail experimentally.

Okay, but if you remove all new ideas, the scientific process as a whole stops working because it has no alternative hypotheses to test. So inventing new ideas is not a dispensable part of the process.

Now put your Bayesian goggles back on. As described in Einstein's Arrogance, there are queries that are not binary - where the answer is not "Yes" or "No", but drawn from a larger space of structures, e.g., the space of equations. In such cases it takes far more Bayesian evidence to promote a hypothesis to your attention than to confirm the hypothesis.

If you're working in the space of all equations that can be specified in 32 bits or less, you're working in a space of 4 billion equations. It takes far more Bayesian evidence to raise one of those hypotheses to the 10% probability level, than it requires further Bayesian evidence to raise the hypothesis from 10% to 90% probability.

When the idea-space is large, coming up with ideas worthy of testing, involves much more work - in the Bayesian-thermodynamic sense of "work" - than merely obtaining an experimental result with p<0.0001 for the new hypothesis over the old hypothesis.

If this doesn't seem obvious-at-a-glance, pause here and read Einstein's Arrogance.

The scientific process has always relied on scientists to come up with hypotheses to test, via some process not further specified by Science. Suppose you came up with some way of generating hypotheses that was completely crazy - say, pumping a robot-controlled Ouija board with the digits of pi - and the resulting suggestions kept on getting verified experimentally. The pure ideal essence of Science wouldn't skip a beat. The pure ideal essence of Bayes would burst into flames and die.

(Compared to Science, Bayes is falsified by more of the possible outcomes.)

This doesn't mean that the process of deciding which ideas to test is unimportant to Science. It means that Science doesn't specify it.

In practice, the robot-controlled Ouija board doesn't work. In practice, there are some scientific queries with a large enough answer space, that picking models at random to test, it would take zillions of years to hit on a model that made good predictions - like getting monkeys to type Shakespeare.

At the frontier of science - the boundary between ignorance and knowledge, where science advances - the process relies on at least some individual scientists (or working groups) seeing things that are not yet confirmed by Science. That's how they know which hypotheses to test, in advance of the test itself.

If you take your Bayesian goggles off, you can say, "Well, they don't have to know, they just have to guess." If you put your Bayesian goggles back on, you realize that "guessing" with 10% probability requires nearly as much epistemic work to have been successfully performed, behind the scenes, as "guessing" with 80% probability - at least for large answer spaces.

The scientist may not know he has done this epistemic work successfully, in advance of the experiment; but he must, in fact, have done it successfully! Otherwise he will not even think of the correct hypothesis. In large answer spaces, anyway.

So the scientist makes the novel prediction, performs the experiment, publishes the result, and now Science knows it too. It is now part of the publicly accessible knowledge of humankind, that anyone can verify for themselves.

In between was an interval where the scientist rationally knew something that the public social process of science hadn't yet confirmed. And this is not a trivial interval, though it may be short; for it is where the frontier of science lies, the advancing border.

All of this is more true for non-routine science than for routine science, because it is a notion of large answer spaces where the answer is not "Yes" or "No" or drawn from a small set of obvious alternatives. It is much easier to train people to test ideas, than to have good ideas to test.

Sequence: Quantum Physics

Changing the Definition of Science

Einstein's Speed

Referenced by: Einstein's Speed • The Quantum Physics Sequence • Anthropomorphic Optimism • Contaminated by Optimism

Original with comments: Faster Than Science

Einstein's Speed

Eliezer Yudkowsky, 21 May 2008 02:48AM

Followup to: Faster Than Science

Yesterday I argued that the Powers Beyond Science are actually a standard and necessary part of the social process of science. In particular, scientists must call upon their powers of individual rationality to decide what ideas to test, in advance of the sort of definite experiments that Science demands to bless an idea as confirmed. The ideal of Science does not try to specify this process - we don't suppose that any public authority knows how individual scientists should think - but this doesn't mean the process is unimportant.

A readily understandable, non-disturbing example:

A scientist identifies a strong mathematical regularity in the cumulative data of previous experiments. But the corresponding hypothesis has not yet made and confirmed a novel experimental prediction - which his academic field demands; this is one of those fields where you can perform controlled experiments without too much trouble. Thus the individual scientist has readily understandable, rational reasons to believe (though not with probability 1) something not yet blessed by Science as public knowledge of humankind.

Noticing a regularity in a huge mass of experimental data, doesn't seem all that unscientific. You're still data-driven, right?

But that's because I deliberately chose a non-disturbing example. When Einstein invented General Relativity, he had almost no experimental data to go on, except the precession of Mercury's perihelion. And (AFAIK) Einstein did not use that data, except at the end.

Einstein generated the theory of Special Relativity using Mach's Principle, which is the physicist's version of the Generalized Anti-Zombie Principle. You begin by saying, "It doesn't seem reasonable to me that you could tell, in an enclosed room, how fast you and the room were going. Since this number shouldn't ought to be observable, it shouldn't ought to exist in any meaningful sense." You then observe that Maxwell's Equations invoke a seemingly absolute speed of propagation, c, commonly referred to as "the speed of light" (though the quantum equations show it is the propagation speed of all fundamental waves). So you reformulate your physics in such fashion that the absolute speed of a single object no longer meaningfully exists, and only relative speeds exist. I am skipping over quite a bit here, obviously, but there are many excellent introductions to relativity - it is not like the horrible situation in quantum physics.

Einstein, having successfully done away with the notion of your absolute speed inside an enclosed room, then set out to do away with the notion of your absolute acceleration inside an enclosed room. It seemed to Einstein that there shouldn't ought to be a way to differentiate, in an enclosed room, between the room accelerating northward while the rest of the universe stayed still, versus the rest of the universe accelerating southward while the room stayed still. If the rest of the universe accelerated, it would produce gravitational waves that would accelerate you. Moving matter, then, should produce gravitational waves.

And because inertial mass and gravitational mass were always exactly equivalent - unlike the situation in electromagnetics, where an electron and a muon can have different masses but the same electrical charge - gravity should reveal itself as a kind of inertia. The Earth should go around the Sun in some equivalent of a "straight line". This requires spacetime in the vicinity of the Sun to be curved, so that if you drew a graph of the Earth's orbit around the Sun, the line on the 4D graph paper would be locally flat. Then inertial and gravitational mass would be necessarily equivalent, not just coincidentally equivalent.

(If that did not make any sense to you, there are good introductions to General Relativity available as well.)

And of course the new theory had to obey Special Relativity, and conserve energy, and conserve momentum, etcetera.

Einstein spent several years grasping the necessary mathematics to describe curved metrics of spacetime. Then he wrote down the simplest theory that had the properties Einstein thought it ought to have - including properties no one had ever observed, but that Einstein thought fit in well with the character of other physical laws. Then Einstein cranked a bit, and got the previously unexplained precession of Mercury right back out.

How impressive was this?

Well, let's put it this way. In some small fraction of alternate Earths proceeding from 1800 - perhaps even a sizeable fraction - it would seem plausible that relativistic physics could have proceeded in a similar fashion to our own great fiasco with quantum physics.

We can imagine that Lorentz's original "interpretation" of the Lorentz contraction, as a physical distortion caused by movement with respect to the ether, prevailed. We can imagine that various corrective factors, themselves unexplained, were added on to Newtonian gravitational mechanics to explain the precession of Mercury - attributed, perhaps, to strange distortions of the ether, as in the Lorentz contraction. Through the decades, further corrective factors would be added on to account for other astronomical observations. Sufficiently precise atomic clocks, in airplanes, would reveal that time ran a little faster than expected at higher altitudes (time runs slower in more intense gravitational fields, but they wouldn't know that) and more corrective "ethereal factors" would be invented.

Until, finally, the many different empirically determined "corrective factors" were unified into the simple equations of General Relativity.

And the people in that alternate Earth would say, "The final equation was simple, but there was no way you could possibly know to arrive at that answer from just the perihelion precession of Mercury. It takes many, many additional experiments. You must have measured time running slower in a stronger gravitational field; you must have measured light bending around stars. Only then can you imagine our unified theory of ethereal gravitation. No, not even a perfect Bayesian superintelligence could know it! - for there would be many ad-hoc theories consistent with the perihelion precession alone."

In our world, Einstein didn't even use the perihelion precession of Mercury, except for verification of his answer produced by other means. Einstein sat down in his armchair, and thought about how he would have designed the universe, to look the way he thought a universe should look - for example, that you shouldn't ought to be able to distinguish yourself accelerating in one direction, from the rest of the universe accelerating in the other direction.

And Einstein executed the whole long (multi-year!) chain of armchair reasoning, without making any mistakes that would have required further experimental evidence to pull him back on track.

Even Jeffreyssai would be grudgingly impressed. Though he would still ding Einstein a point or two for the cosmological constant. (I don't ding Einstein for the cosmological constant because it later turned out to be real. I try to avoid criticizing people on occasions where they are right.)

What would be the probability-theoretic perspective on Einstein's feat?

Rather than observe the planets, and infer what laws might cover their gravitation, Einstein was observing the other laws of physics, and inferring what new law might follow the same pattern. Einstein wasn't finding an equation that covered the motion of gravitational bodies. Einstein was finding a character-of-physical-law that covered previously observed equations, and that he could crank to predict the next equation that would be observed.

Nobody knows where the laws of physics come from, but Einstein's success with General Relativity shows that their common character is strong enough to predict the correct form of one law from having observed other laws, without necessarily needing to observe the precise effects of the law.

(In a general sense, of course, Einstein did know by observation that things fell down; but he did not get GR by backward inference from Mercury's exact perihelion advance.)

So, from a Bayesian perspective, what Einstein did is still induction, and still covered by the notion of a simple prior (Occam prior) that gets updated by new evidence. It's just the prior was over the possible characters of physical law, and observing other physical laws let Einstein update his model of the character of physical law, which he then used to predict a particular law of gravitation.

If you didn't have the concept of a "character of physical law", what Einstein did would look like magic - plucking the correct model of gravitation out of the space of all possible equations, with vastly insufficient evidence. But Einstein, by looking at other laws, cut down the space of possibilities for the next law. He learned the alphabet in which physics was written, constraints to govern his answer. Not magic, but reasoning on a higher level, across a wider domain, than what a naive reasoner might conceive to be the "model space" of only this one law.

So from a probability-theoretic standpoint, Einstein was still data-driven - he just used the data he already had, more effectively. Compared to any alternate Earths that demanded huge quantities of additional data from astronomical observations and clocks on airplanes to hit them over the head with General Relativity.

There are numerous lessons we can derive from this.

I use Einstein as my example, even though it's cliché, because Einstein was also unusual in that he openly admitted to knowing things that Science hadn't confirmed. Asked what he would have done if Eddington's solar eclipse observation had failed to confirm General Relativity, Einstein replied: "Then I would feel sorry for the good Lord. The theory is correct."

According to prevailing notions of Science, this is arrogance - you must accept the verdict of experiment, and not cling to your personal ideas.

But as I concluded in Einstein's Arrogance, Einstein doesn't come off nearly as badly from a Bayesian perspective. From a Bayesian perspective, in order to suggest General Relativity at all, in order to even think about what turned out to be the correct answer, Einstein must have had enough evidence to identify the true answer in the theory-space. It would take only a little more evidence to justify (in a Bayesian sense) being nearly certain of the theory. And it was unlikely that Einstein only had exactly enough evidence to bring the hypothesis all the way up to his attention.

Any accusation of arrogance would have to center around the question, "But Einstein, how did you know you had reasoned correctly?" - to which I can only say: Do not criticize people when they turn out to be right! Wait for an occasion where they are wrong! Otherwise you are missing the chance to see when someone is thinking smarter than you - for you criticize them whenever they depart from a preferred ritual of cognition.

Or consider the famous exchange between Einstein and Niels Bohr on quantum theory - at a time when the then-current, single-world quantum theory seemed to be immensely well-confirmed experimentally; a time when, by the standards of Science, the current (deranged) quantum theory had simply won.

Einstein: "God does not play dice with the universe."
Bohr: "Einstein, don't tell God what to do."

You've got to admire someone who can get into an argument with God and win.

If you take off your Bayesian goggles, and look at Einstein in terms of what he actually did all day, then the guy was sitting around studying math and thinking about how he would design the universe, rather than running out and looking at things to gather more data. What Einstein did, successfully, is exactly the sort of high-minded feat of sheer intellect that Aristotle thought he could do, but couldn't. Not from a probability-theoretic stance, mind you, but from the viewpoint of what they did all day long.

Science does not trust scientists to do this, which is why General Relativity was not blessed as the public knowledge of humanity until after it had made and verified a novel experimental prediction - having to do with the bending of light in a solar eclipse. (It later turned out that particular measurement was not precise enough to verify reliably, and had favored GR essentially by luck.)

However, just because Science does not trust scientists to do something, does not mean it is impossible.

But a word of caution here: The reason why history books sometimes record the names of scientists who thought great high-minded thoughts, is not that high-minded thinking is easier, or more reliable. It is a priority bias: Some scientist who successfully reasoned from the smallest amount of experimental evidence got to the truth first. This cannot be a matter of pure random chance: The theory space is too large, and Einstein won several times in a row. But out of all the scientists who tried to unravel a puzzle, or who would have eventually succeeded given enough evidence, history passes down to us the names of the scientists who successfully got there first. Bear that in mind, when you are trying to derive lessons about how to reason prudently.

In everyday life, you want every scrap of evidence you can get. Do not rely on being able to successfully think high-minded thoughts unless experimentation is so costly or dangerous that you have no other choice.

But sometimes experiments are costly, and sometimes we prefer to get there first... so you might consider trying to train yourself in reasoning on scanty evidence, preferably in cases where you will later find out if you were right or wrong. Trying to beat low-capitalization prediction markets might make for good training in this? - though that is only speculation.

As of now, at least, reasoning based on scanty evidence is something that modern-day science cannot reliably train modern-day scientists to do at all. Which may perhaps have something to do with, oh, I don't know, not even trying?

Actually, I take that back. The most sane thinking I have seen in any scientific field comes from the field of evolutionary psychology, possibly because they understand self-deception, but also perhaps because they often (1) have to reason from scanty evidence and (2) do later find out if they were right or wrong. I recommend to all aspiring rationalists that they study evolutionary psychology simply to get a glimpse of what careful reasoning looks like. See particularly Tooby and Cosmides's "The Psychological Foundations of Culture".

As for the possibility that only Einstein could do what Einstein did... that it took superpowers beyond the reach of ordinary mortals... here we run into some biases that would take a separate post to analyze. Let me put it this way: It is possible, perhaps, that only a genius could have done Einstein's actual historical work. But potential geniuses, in terms of raw intelligence, are probably far more common than historical superachievers. To put a random number on it, I doubt that anything more than one-in-a-million g-factor is required to be a potential world-class genius, implying at least six thousand potential Einsteins running around today. And as for everyone else, I see no reason why they should not aspire to use efficiently the evidence that they have.

But my final moral is that the frontier where the individual scientist rationally knows something that Science has not yet confirmed, is not always some innocently data-driven matter of spotting a strong regularity in a mountain of experiments. Sometimes the scientist gets there by thinking great high-minded thoughts that Science does not trust you to think.

I will not say, "Don't try this at home." I will say, "Don't think this is easy." We are not discussing, here, the victory of casual opinions over professional scientists. We are discussing the sometime historical victories of one kind of professional effort over another. Never forget all the famous historical cases where attempted armchair reasoning lost.

Sequence: Quantum Physics

Faster Than Science

That Alien Message

Referenced by: That Alien Message • Einstein's Superpowers • The Quantum Physics Sequence • Grasping Slippery Things • True Sources of Disagreement

Original with comments: Einstein's Speed

That Alien Message

Eliezer Yudkowsky, 22 May 2008 05:55AM

Followup to: Einstein's Speed

Imagine a world much like this one, in which, thanks to gene-selection technologies, the average IQ is 140 (on our scale). Potential Einsteins are one-in-a-thousand, not one-in-a-million; and they grow up in a school system suited, if not to them personally, then at least to bright kids. Calculus is routinely taught in sixth grade. Albert Einstein, himself, still lived and still made approximately the same discoveries, but his work no longer seems exceptional. Several modern top-flight physicists have made equivalent breakthroughs, and are still around to talk.

(No, this is not the world Brennan lives in.)

One day, the stars in the night sky begin to change.

Some grow brighter. Some grow dimmer. Most remain the same. Astronomical telescopes capture it all, moment by moment. The stars that change, change their luminosity one at a time, distinctly so; the luminosity change occurs over the course of a microsecond, but a whole second separates each change.

It is clear, from the first instant anyone realizes that more than one star is changing, that the process seems to center around Earth particularly. The arrival of the light from the events, at many stars scattered around the galaxy, has been precisely timed to Earth in its orbit. Soon, confirmation comes in from high-orbiting telescopes (they have those) that the astronomical miracles do not seem as synchronized from outside Earth. Only Earth's telescopes see one star changing every second (1005 milliseconds, actually).

Almost the entire combined brainpower of Earth turns to analysis.

It quickly becomes clear that the stars that jump in luminosity, all jump by a factor of exactly 256; those that diminish in luminosity, diminish by a factor of exactly 256. There is no apparent pattern in the stellar coordinates. This leaves, simply, a pattern of BRIGHT-dim-BRIGHT-BRIGHT...

"A binary message!" is everyone's first thought.

But in this world there are careful thinkers, of great prestige as well, and they are not so sure. "There are easier ways to send a message," they post to their blogs, "if you can make stars flicker, and if you want to communicate. Something is happening. It appears, prima facie, to focus on Earth in particular. To call it a 'message' presumes a great deal more about the cause behind it. There might be some kind of evolutionary process among, um, things that can make stars flicker, that ends up sensitive to intelligence somehow... Yeah, there's probably something like 'intelligence' behind it, but try to appreciate how wide a range of possibilities that really implies. We don't know this is a message, or that it was sent from the same kind of motivations that might move us. I mean, we would just signal using a big flashlight, we wouldn't mess up a whole galaxy."

By this time, someone has started to collate the astronomical data and post it to the Internet. Early suggestions that the data might be harmful, have been... not ignored, but not obeyed, either. If anything this powerful wants to hurt you, you're pretty much dead (people reason).

Multiple research groups are looking for patterns in the stellar coordinates - or fractional arrival times of the changes, relative to the center of the Earth - or exact durations of the luminosity shift - or any tiny variance in the magnitude shift - or any other fact that might be known about the stars before they changed. But most people are turning their attention to the pattern of BRIGHTS and dims.

It becomes clear almost instantly that the pattern sent is highly redundant. Of the first 16 bits, 12 are BRIGHTS and 4 are dims. The first 32 bits received align with the second 32 bits received, with only 7 out of 32 bits different, and then the next 32 bits received have only 9 out of 32 bits different from the second (and 4 of them are bits that changed before). From the first 96 bits, then, it becomes clear that this pattern is not an optimal, compressed encoding of anything. The obvious thought is that the sequence is meant to convey instructions for decoding a compressed message to follow...

"But," say the careful thinkers, "anyone who cared about efficiency, with enough power to mess with stars, could maybe have just signaled us with a big flashlight, and sent us a DVD?"

There also seems to be structure within the 32-bit groups; some 8-bit subgroups occur with higher frequency than others, and this structure only appears along the natural alignments (32 = 8 + 8 + 8 + 8).

After the first five hours at one bit per second, an additional redundancy becomes clear: The message has started approximately repeating itself at the 16,385th bit.

Breaking up the message into groups of 32, there are 7 bits of difference between the 1st group and the 2nd group, and 6 bits of difference between the 1st group and the 513th group.

"A 2D picture!" everyone thinks. "And the four 8-bit groups are colors; they're tetrachromats!"

But it soon becomes clear that there is a horizontal/vertical asymmetry: Fewer bits change, on average, between (N, N+1) versus (N, N+512). Which you wouldn't expect if the message was a 2D picture projected onto a symmetrical grid. Then you would expect the average bitwise distance between two 32-bit groups to go as the 2-norm of the grid separation: √(h^2 + v^2).

There also forms a general consensus that a certain binary encoding from 8-groups onto integers between -64 and 191 - not the binary encoding that seems obvious to us, but still highly regular - minimizes the average distance between neighboring cells. This continues to be borne out by incoming bits.

The statisticians and cryptographers and physicists and computer scientists go to work. There is structure here; it needs only to be unraveled. The masters of causality search for conditional independence, screening-off and Markov neighborhoods, among bits and groups of bits. The so-called "color" appears to play a role in neighborhoods and screening, so it's not just the equivalent of surface reflectivity. People search for simple equations, simple cellular automata, simple decision trees, that can predict or compress the message. Physicists invent entire new theories of physics that might describe universes projected onto the grid - for it seems quite plausible that a message such as this is being sent from beyond the Matrix.

After receiving 32 * 512 * 256 = 4,194,304 bits, around one and a half months, the stars stop flickering.

Theoretical work continues. Physicists and cryptographers roll up their sleeves and seriously go to work. They have cracked problems with far less data than this. Physicists have tested entire theory-edifices with small differences of particle mass; cryptographers have unraveled shorter messages deliberately obscured.

Years pass.

Two dominant models have survived, in academia, in the scrutiny of the public eye, and in the scrutiny of those scientists who once did Einstein-like work. There is a theory that the grid is a projection from objects in a 5-dimensional space, with an asymmetry between 3 and 2 of the spatial dimensions. There is also a theory that the grid is meant to encode a cellular automaton - arguably, the grid has several fortunate properties for such. Codes have been devised that give interesting behaviors; but so far, running the corresponding automata on the largest available computers, has failed to produce any decodable result. The run continues.

Every now and then, someone takes a group of especially brilliant young students who've never looked at the detailed binary sequence. These students are then shown only the first 32 rows (of 512 columns each), to see if they can form new models, and how well those new models do at predicting the next 224. Both the 3+2 dimensional model, and the cellular-automaton model, have been well duplicated by such students; they have yet to do better. There are complex models finely fit to the whole sequence - but those, everyone knows, are probably worthless.

Ten years later, the stars begin flickering again.

Within the reception of the first 128 bits, it becomes clear that the Second Grid can fit to small motions in the inferred 3+2 dimensional space, but does not look anything like the successor state of any of the dominant cellular automaton theories. Much rejoicing follows, and the physicists go to work on inducing what kind of dynamical physics might govern the objects seen in the 3+2 dimensional space. Much work along these lines has already been done, just by speculating on what type of balanced forces might give rise to the objects in the First Grid, if those objects were static - but now it seems not all the objects are static. As most physicists guessed - statically balanced theories seemed contrived.

Many neat equations are formulated to describe the dynamical objects in the 3+2 dimensional space being projected onto the First and Second Grids. Some equations are more elegant than others; some are more precisely predictive (in retrospect, alas) of the Second Grid. One group of brilliant physicists, who carefully isolated themselves and looked only at the first 32 rows of the Second Grid, produces equations that seem elegant to them - and the equations also do well on predicting the next 224 rows. This becomes the dominant guess.

But these equations are underspecified; they don't seem to be enough to make a universe. A small cottage industry arises in trying to guess what kind of laws might complete the ones thus guessed.

When the Third Grid arrives, ten years after the Second Grid, it provides information about second derivatives, forcing a major modification of the "incomplete but good" theory. But the theory doesn't do too badly out of it, all things considered.

The Fourth Grid doesn't add much to the picture. Third derivatives don't seem important to the 3+2 physics inferred from the Grids.

The Fifth Grid looks almost exactly like it is expected to look.

And the Sixth Grid, and the Seventh Grid.

(Oh, and every time someone in this world tries to build a really powerful AI, the computing hardware spontaneously melts. This isn't really important to the story, but I need to postulate this in order to have human people sticking around, in the flesh, for seventy years.)

My moral?

That even Einstein did not come within a million light-years of making efficient use of sensory data.

Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis - perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration - by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

We would think of it. Our civilization, that is, given ten years to analyze each frame. Certainly if the average IQ was 140 and Einsteins were common, we would.

Even if we were human-level intelligences in a different sort of physics - minds who had never seen a 3D space projected onto a 2D grid - we would still think of the 3D->2D hypothesis. Our mathematicians would still have invented vector spaces, and projections.

Even if we'd never seen an accelerating billiard ball, our mathematicians would have invented calculus (e.g. for optimization problems).

Heck, think of some of the crazy math that's been invented here on our Earth.

I occasionally run into people who say something like, "There's a theoretical limit on how much you can deduce about the outside world, given a finite amount of sensory data."

Yes. There is. The theoretical limit is that every time you see 1 additional bit, it cannot be expected to eliminate more than half of the remaining hypotheses (half the remaining probability mass, rather). And that a redundant message, cannot convey more information than the compressed version of itself. Nor can a bit convey any information about a quantity, with which it has correlation exactly zero, across the probable worlds you imagine.

But nothing I've depicted this human civilization doing, even begins to approach the theoretical limits set by the formalism of Solomonoff induction. It doesn't approach the picture you could get if you could search through every single computable hypothesis, weighted by their simplicity, and do Bayesian updates on all of them.

To see the theoretical limit on extractable information, imagine that you have infinite computing power, and you simulate all possible universes with simple physics, looking for universes that contain Earths embedded in them - perhaps inside a simulation - where some process makes the stars flicker in the order observed. Any bit in the message - or any order of selection of stars, for that matter - that contains the tiniest correlation (across all possible computable universes, weighted by simplicity) to any element of the environment, gives you information about the environment.

Solomonoff induction, taken literally, would create countably infinitely many sentient beings, trapped inside the computations. All possible computable sentient beings, in fact. Which scarcely seems ethical. So let us be glad this is only a formalism.

But my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers.

It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?"

People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit.

No one - not even a Bayesian superintelligence - will ever come remotely close to making efficient use of their sensory information...

...is what I would like to say, but I don't trust my ability to set limits on the abilities of Bayesian superintelligences.

(Though I'd bet money on it, if there were some way to judge the bet. Just not at very extreme odds.)

The story continues:

Millennia later, frame after frame, it has become clear that some of the objects in the depiction are extending tentacles to move around other objects, and carefully configuring other tentacles to make particular signs. They're trying to teach us to say "rock".

It seems the senders of the message have vastly underestimated our intelligence. From which we might guess that the aliens themselves are not all that bright. And these awkward children can shift the luminosity of our stars? That much power and that much stupidity seems like a dangerous combination.

Our evolutionary psychologists begin extrapolating possible courses of evolution that could produce such aliens. A strong case is made for them having evolved asexually, with occasional exchanges of genetic material and brain content; this seems like the most plausible route whereby creatures that stupid could still manage to build a technological civilization. Their Einsteins may be our undergrads, but they could still collect enough scientific data to get the job done eventually, in tens of their millennia perhaps.

The inferred physics of the 3+2 universe is not fully known, at this point; but it seems sure to allow for computers far more powerful than our quantum ones. We are reasonably certain that our own universe is running as a simulation on such a computer. Humanity decides not to probe for bugs in the simulation; we wouldn't want to shut ourselves down accidentally.

Our evolutionary psychologists begin to guess at the aliens' psychology, and plan out how we could persuade them to let us out of the box. It's not difficult in an absolute sense - they aren't very bright - but we've got to be very careful...

We've got to pretend to be stupid, too; we don't want them to catch on to their mistake.

It's not until a million years later, though, that they get around to telling us how to signal back.

At this point, most of the human species is in cryonic suspension, at liquid helium temperatures, beneath radiation shielding. Every time we try to build an AI, or a nanotechnological device, it melts down. So humanity waits, and sleeps. Earth is run by a skeleton crew of nine supergeniuses. Clones, known to work well together, under the supervision of certain computer safeguards.

An additional hundred million human beings are born into that skeleton crew, and age, and enter cryonic suspension, before they get a chance to slowly begin to implement plans made eons ago...

From the aliens' perspective, it took us thirty of their minute-equivalents to oh-so-innocently learn about their psychology, oh-so-carefully persuade them to give us Internet access, followed by five minutes to innocently discover their network protocols, then some trivial cracking whose only difficulty was an innocent-looking disguise. We read a tiny handful of physics papers (bit by slow bit) from their equivalent of arXiv, learning far more from their experiments than they had. (Earth's skeleton team spawned an extra twenty Einsteins, that generation.)

Then we cracked their equivalent of the protein folding problem over a century or so, and did some simulated engineering in their simulated physics. We sent messages (steganographically encoded until our cracked servers decoded it) to labs that did their equivalent of DNA sequencing and protein synthesis. We found some unsuspecting schmuck, and gave it a plausible story and the equivalent of a million dollars of cracked computational monopoly money, and told it to mix together some vials it got in the mail. Protein-equivalents that self-assembled into the first-stage nanomachines, that built the second-stage nanomachines, that built the third-stage nanomachines... and then we could finally begin to do things at a reasonable speed.

Three of their days, all told, since they began speaking to us. Half a billion years, for us.

They never suspected a thing. They weren't very smart, you see, even before taking into account their slower rate of time. Their primitive equivalents of rationalists went around saying things like, "There's a bound to how much information you can extract from sensory data." And they never quite realized what it meant, that we were smarter than them, and thought faster.

Sequence: Quantum Physics

Einstein's Speed

My Childhood Role Model

Referenced by: My Childhood Role Model • Timeless Identity • The Quantum Physics Sequence • Dreams of Friendliness • Optimization • Complexity and Intelligence • Recursive Self-Improvement • Hard Takeoff • What I Think, If Not Why • The Thing That I Protect

Original with comments: That Alien Message

My Childhood Role Model

Eliezer Yudkowsky, 23 May 2008 08:51AM

Followup to: That Alien Message

When I lecture on the Singularity, I often draw a graph of the "scale of intelligence" as it appears in everyday life:

But this is a rather parochial view of intelligence. Sure, in everyday life, we only deal socially with other humans - only other humans are partners in the great game - and so we only meet the minds of intelligences ranging from village idiot to Einstein. But what we really need to talk about Artificial Intelligence or theoretical optima of rationality, is this intelligence scale:

For us humans, it seems that the scale of intelligence runs from "village idiot" at the bottom to "Einstein" at the top. Yet the distance from "village idiot" to "Einstein" is tiny, in the space of brain designs. Einstein and the village idiot both have a prefrontal cortex, a hippocampus, a cerebellum...

Maybe Einstein has some minor genetic differences from the village idiot, engine tweaks. But the brain-design-distance between Einstein and the village idiot is nothing remotely like the brain-design-distance between the village idiot and a chimpanzee. A chimp couldn't tell the difference between Einstein and the village idiot, and our descendants may not see much of a difference either.

Carl Shulman has observed that some academics who talk about transhumanism, seem to use the following scale of intelligence:

Douglas Hofstadter actually said something like this, at the 2006 Singularity Summit. He looked at my diagram showing the "village idiot" next to "Einstein", and said, "That seems wrong to me; I think Einstein should be way off on the right."

I was speechless. Especially because this was Douglas Hofstadter, one of my childhood heroes. It revealed a cultural gap that I had never imagined existed.

See, for me, what you would find toward the right side of the scale, was a Jupiter Brain. Einstein did not literally have a brain the size of a planet.

On the right side of the scale, you would find Deep Thought - Douglas Adams's original version, thank you, not the chessplayer. The computer so intelligent that even before its stupendous data banks were connected, when it was switched on for the first time, it started from I think therefore I am and got as far as deducing the existence of rice pudding and income tax before anyone managed to shut it off.

Toward the right side of the scale, you would find the Elders of Arisia, galactic overminds, Matrioshka brains, and the better class of God. At the extreme right end of the scale, Old One and the Blight.

Not frickin' Einstein.

I'm sure Einstein was very smart for a human. I'm sure a General Systems Vehicle would think that was very cute of him.

I call this a "cultural gap" because I was introduced to the concept of a Jupiter Brain at the age of twelve.

Now all of this, of course, is the logical fallacy of generalization from fictional evidence.

But it is an example of why - logical fallacy or not - I suspect that reading science fiction does have a helpful effect on futurism. Sometimes the alternative to a fictional acquaintance with worlds outside your own, is to have a mindset that is absolutely stuck in one era: A world where humans exist, and have always existed, and always will exist.

The universe is 13.7 billion years old, people! Homo sapiens sapiens have only been around for a hundred thousand years or thereabouts!

Then again, I have met some people who never read science fiction, but who do seem able to imagine outside their own world. And there are science fiction fans who don't get it. I wish I knew what "it" was, so I could bottle it.

Yesterday, I wanted to talk about the efficient use of evidence, i.e., Einstein was cute for a human but in an absolute sense he was around as efficient as the US Department of Defense.

So I had to talk about a civilization that included thousands of Einsteins, thinking for decades. Because if I'd just depicted a Bayesian superintelligence in a box, looking at a webcam, people would think: "But... how does it know how to interpret a 2D picture?" They wouldn't put themselves in the shoes of the mere machine, even if it was called a "Bayesian superintelligence"; they wouldn't apply even their own creativity to the problem of what you could extract from looking at a grid of bits.

It would just be a ghost in a box, that happened to be called a "Bayesian superintelligence". The ghost hasn't been told anything about how to interpret the input of a webcam; so, in their mental model, the ghost does not know.

As for whether it's realistic to suppose that one Bayesian superintelligence can "do all that"... i.e., the stuff that occurred to me on first sitting down to the problem, writing out the story as I went along...

Well, let me put it this way: Remember how Jeffreyssai pointed out that if the experience of having an important insight doesn't take more than 5 minutes, this theoretically gives you time for 5760 insights per month? Assuming you sleep 8 hours a day and have no important insights while sleeping, that is.

Now humans cannot use themselves this efficiently. But humans are not adapted for the task of scientific research. Humans are adapted to chase deer across the savanna, throw spears into them, cook them, and then - this is probably the part that takes most of the brains - cleverly argue that they deserve to receive a larger share of the meat.

It's amazing that Albert Einstein managed to repurpose a brain like that for the task of doing physics. This deserves applause. It deserves more than applause, it deserves a place in the Guinness Book of Records. Like successfully building the fastest car ever to be made entirely out of Jello.

How poorly did the blind idiot god (evolution) really design the human brain?

This is something that can only be grasped through much study of cognitive science, until the full horror begins to dawn upon you.

All the biases we have discussed here should at least be a hint.

Likewise the fact that the human brain must use its full power and concentration, with trillions of synapses firing, to multiply out two three-digit numbers without a paper and pencil.

No more than Einstein made efficient use of his sensory data, did his brain make efficient use of his neurons firing.

Of course I have certain ulterior motives in saying all this. But let it also be understood that, years ago, when I set out to be a rationalist, the impossible unattainable ideal of intelligence that inspired me, was never Einstein.

Carl Schurz said:

"Ideals are like stars. You will not succeed in touching them with your hands. But, like the seafaring man on the desert of waters, you choose them as your guides and following them you will reach your destiny."

So now you've caught a glimpse of one of my great childhood role models - my dream of an AI. Only the dream, of course, the reality not being available. I reached up to that dream, once upon a time.

And this helped me to some degree, and harmed me to some degree.

For some ideals are like dreams: they come from within us, not from outside. Mentor of Arisia proceeded from E. E. "doc" Smith's imagination, not from any real thing. If you imagine what a Bayesian superintelligence would say, it is only your own mind talking. Not like a star, that you can follow from outside. You have to guess where your ideals are, and if you guess wrong, you go astray.

But do not limit your ideals to mere stars, to mere humans who actually existed, especially if they were born more than fifty years before you and are dead. Each succeeding generation has a chance to do better. To let your ideals be composed only of humans, especially dead ones, is to limit yourself to what has already been accomplished. You will ask yourself, "Do I dare to do this thing, which Einstein could not do? Is this not lèse majesté?" Well, if Einstein had sat around asking himself, "Am I allowed to do better than Newton?" he would not have gotten where he did. This is the problem with following stars; at best, it gets you to the star.

Your era supports you more than you realize, in unconscious assumptions, in subtly improved technology of mind. Einstein was a nice fellow, but he talked a deal of nonsense about an impersonal God, which shows you how well he understood the art of careful thinking at a higher level of abstraction than his own field. It may seem less like sacrilege to think that, if you have at least one imaginary galactic supermind to compare with Einstein, so that he is not the far right end of your intelligence scale.

If you only try to do what seems humanly possible, you will ask too little of yourself. When you imagine reaching up to some higher and inconvenient goal, all the convenient reasons why it is "not possible" leap readily to mind.

The most important role models are dreams: they come from within ourselves. To dream of anything less than what you conceive to be perfection, is to draw on less than the full power of the part of yourself that dreams.

Sequence: Quantum Physics

That Alien Message

Einstein's Superpowers

Referenced by: Einstein's Superpowers • Timeless Identity • The Quantum Physics Sequence • Grasping Slippery Things • The Psychological Unity of Humankind • Measuring Optimization Power • Economic Definition of Intelligence? • Intelligence in Economics • Recursive Self-Improvement • Free to Optimize

Original with comments: My Childhood Role Model

Mach's Principle: Anti-Epiphenomenal Physics

Eliezer Yudkowsky, 24 May 2008 05:01AM

This post is part of the Quantum Physics Sequence.
Previously in series: Many Worlds, One Best Guess
Followup to: The Generalized Anti-Zombie Principle

Warning: Mach's Principle is not experimentally proven, though it is widely considered to be credible.

Centuries ago, when Galileo was promoting the Copernican model in which the Earth spun on its axis and traveled around the Sun, there was great opposition from those who trusted their common sense:

"How could the Earth be moving? I don't feel it moving! The ground beneath my feet seems perfectly steady!"

And lo, Galileo said: If you were on a ship sailing across a perfectly level sea, and you were in a room in the interior of the ship, you wouldn't know how fast the ship was moving. If you threw a ball in the air, you would still be able to catch it, because the ball would have initially been moving at the same speed as you and the room and the ship. So you can never tell how fast you are moving.

This would turn out to be the beginning of one of the most important ideas in the history of physics. Maybe even the most important idea in all of physics. And I'm not talking about Special Relativity.

Suppose the entire universe was moving. Say, the universe was moving left along the x axis at 10 kilometers per hour.

If you tried to visualize what I just said, it seems like you can imagine it. If the universe is standing still, then you imagine a little swirly cloud of galaxies standing still. If the whole universe is moving left, then you imagine the little swirly cloud moving left across your field of vision until it passes out of sight.

But then, some people think they can imagine philosophical zombies: entities who are identical to humans down to the molecular level, but not conscious. So you can't always trust your imagination.

Forget, for a moment, anything you know about relativity. Pretend you live in a Newtonian universe.

In a Newtonian universe, 3+1 spacetime can be broken down into 3 space dimensions and 1 time dimension, and you can write them out as 4 real numbers, (x, y, z, t). Deciding how to write the numbers involves seemingly arbitrary choices, like which direction to call 'x', and which perpendicular direction to then call 'y', and where in space and time to put your origin (0, 0, 0, 0), and whether to use meters or miles to measure distance. But once you make these arbitrary choices, you can, in a Newtonian universe, use the same system of coordinates to describe the whole universe.

Suppose that you pick an arbitrary but uniform (x, y, z, t) coordinate system. Suppose that you use these coordinates to describe every physical experiment you've ever done - heck, every observation you've ever made.

Next, suppose that you were, in your coordinate system, to shift the origin 10 meters to the left along the x axis. Then if you originally thought that Grandma's House was 400 meters to the right of the origin, you would now think that Grandma's House is 410 meters to the right of the origin. Thus every point (x, y, z, t) would be relabeled as (x' = x + 10, y' = y, z' = z, t' = t).

You can express the idea that "physics does not have an absolute origin", by saying that the observed laws of physics, as you generalize them, should be exactly the same after you perform this coordinate transform. The history may not be written out in exactly the same way, but the laws will be written out the same way. Let's say that in the old coordinate system, Your House is at (100, 10, -20, 7:00am) and you walk to Grandma's House at (400, 10, -20, 7:05am). Then you traveled from Your House to Grandma's House at one meter per second. In the new coordinate system, we would write the history as (110, 10, 20, 7:00am) and (410, 10, -20, 7:05am) but your apparent speed would come out the same, and hence so would your acceleration. The laws governing how fast things moved when you pushed on them - how fast you accelerated forward when your legs pushed on the ground - would be the same.

Now if you were given to jumping to conclusions, and moreover, given to jumping to conclusions that were exactly right, you might say:

"Since there's no way of figuring out where the origin is by looking at the laws of physics, the origin must not really exist! There is no (0, 0, 0, 0) point floating out in space somewhere!"

Which is to say: There is just no fact of the matter as to where the origin "really" is. When we argue about our choice of representation, this fact about the map does not actually correspond to any fact about the territory.

Now this statement, if you interpret it in the natural way, is not necessarily true. We can readily imagine alternative laws of physics, which, written out in their most natural form, would not be insensitive to shifting the "origin". The Aristotelian universe had a crystal sphere of stars rotating around the Earth. But so far as anyone has been able to tell, in our real universe, the laws of physics do not have any natural "origin" written into them. When you write out your observations in the simplest way, the coordinate transform x' = x + 10 does not change any of the laws; you write the same laws over x' as over x.

As Feynman said:

Philosophers, incidentally, say a great deal about what is absolutely necessary for science, and it is always, so far as one can see, rather naive, and probably wrong. For example, some philosopher or other said it is fundamental to the scientific effort that if an experiment is performed in, say, Stockholm, and then the same experiment is done in, say, Quito, the same results must occur. That is quite false. It is not necessary that science do that; it may be a fact of experience, but it is not necessary...

What is the fundamental hypothesis of science, the fundamental philosophy? We stated it in the first chapter: the sole test of the validity of any idea is experiment...

If we are told that the same experiment will always produce the same result, that is all very well, but if when we try it, it does not, then it does not. We just have to take what we see, and then formulate all the rest of our ideas in terms of our actual experience.

And so if you regard the universe itself as a sort of Galileo's Ship, it would seem that the notion of the entire universe moving at a particular rate - say, all the objects in the universe, including yourself, moving left along the x axis at 10 meters per second - must also be silly. What is it that moves?

If you believe that everything in a Newtonian universe is moving left along the x axis at an average of 10 meters per second, then that just says that when you write down your observations, you write down an x coordinate that is 10 meters per second to the left, of what you would have written down, if you believed the universe was standing still. If the universe is standing still, you would write that Grandma's House was observed at (400, 10, -20, 7:00am) and then observed again, a minute later, at (400, 10, -20, 7:01am). If you believe that the whole universe is moving to the left at 10 meters per second, you would write that Grandma's House was observed at (400, 10, -20, 7:00am) and then observed again at (-200, 10, -20, 7:01am). Which is just the same as believing that the origin of the universe is moving right at 10 meters per second.

But the universe has no origin! So this notion of the whole universe moving at a particular speed, must be nonsense.

Yet if it makes no sense to talk about speed in an absolute, global sense, then what is speed?

It is simply the movement of one thing relative to a different thing! This is what our laws of physics talk about... right? The law of gravity, for example, talks about how planets pull on each other, and change their velocity relative to each other. Our physics do not talk about a crystal sphere of stars spinning around the objective center of the universe.

And now - it seems - we understand how we have been misled, by trying to visualize "the whole universe moving left", and imagining a little blurry blob of galaxies scurrying from the right to the left of our visual field. When we imagine this sort of thing, it is (probably) articulated in our visual cortex; when we visualize a little blob scurrying to the left, then there is (probably) an activation pattern that proceeds across the columns of our visual cortex. The seeming absolute background, the origin relative to which the universe was moving, was in the underlying neurology we used to visualize it!

But there is no origin! So the whole thing was just a case of the Mind Projection Fallacy - again.

Ah, but now Newton comes along, and he sees the flaw in the whole argument.

From Galileo's Ship we pass to Newton's Bucket. This is a bucket of water, hung by a cord. If you twist up the cord tightly, and then release the bucket, the bucket will spin. The water in the bucket, as the bucket wall begins to accelerate it, will assume a concave shape. Water will climb up the walls of the bucket, from centripetal force.

If you supposed that the whole universe was rotating relative to the origin, the parts would experience a centrifugal force, and fly apart. (No this is not why the universe is expanding, thank you for asking.)

Newton used his Bucket to argue in favor of an absolute space - an absolute background for his physics. There was a testable difference between the whole universe rotating, and the whole universe not rotating. By looking at the parts of the universe, you could determine their rotational velocity - not relative to each other, but relative to absolute space.

This absolute space was a tangible thing, to Newton: it was aether, possibly involved in the transmission of gravity. Newton didn't believe in action-at-a-distance, and so he used his Bucket to argue for the existence of an absolute space, that would be an aether, that could perhaps transmit gravity.

Then the origin-free view of the universe took another hit. Maxwell's Equations showed that, indeed, there seemed to be an absolute speed of light - a standard rate at which the electric and magnetic fields would oscillate and transmit a wave. In which case, you could determine how fast you were going, by seeing in which directions light seemed to be moving quicker and slower.

Along came a stubborn fellow named Ernst Mach, who really didn't like absolute space. Following some earlier ideas of Leibniz, Mach tried to get rid of Newton's Bucket by asserting that inertia was about your relative motion. Mach's Principle asserted that the resistance-to-changing-speed that determined how fast you accelerated under a force, was a resistance to changing your relative speed, compared to other objects. So that if the whole universe was rotating, no one would notice anything, because the inertial frame would also be rotating.

Or to put Mach's Principle more precisely, even if you imagined the whole universe was rotating, the relative motions of all the objects in the universe would be just the same as before, and their inertia - their resistance to changes of relative motion - would be just the same as before.

At the time, there did not seem to be any good reason to suppose this. It seemed like a mere attempt to impose philosophical elegance on a universe that had no particular reason to comply.

The story continues. A couple of guys named Michelson and Morley built an ingenious apparatus that would, via interference patterns in light, detect the absolute motion of Earth - as it spun on its axis, and orbited the Sun, which orbited the Milky Way, which hurtled toward Andromeda. Or, if you preferred, the Michelson-Morley apparatus would detect Earth's motion relative to the luminiferous aether, the medium through which light waves propagated. Just like Maxwell's Equations seemed to say you could do, and just like Newton had always thought you could do.

The Michelson-Morley apparatus said the absolute motion was zero.

This caused a certain amount of consternation.

Enter Albert Einstein.

The first thing Einstein did was repair the problem posed by Maxwell's Equations, which seemed to talk about an absolute speed of light. If you used a different, non-Galilean set of coordinate transforms - the Lorentz transformations - you could show that the speed of light would always look the same, in every direction, no matter how fast you were moving.

I'm not going to talk much about Special Relativity, because that introduction has already been written many times. If you don't get all indignant about "space" and "time" not turning out to work the way you thought they did, the math should be straightforward.

Albeit for the benefit of those who may need to resist postmodernism, I will note that the word "relativity" is a misnomer. What "relativity" really does, is establish new invariant elements of reality. The quantity √(t^2 - x^2 - y^2 - z^2) is the same in every frame of reference. The x and y and z, and even t, seem to change with your point of view. But not √(t^2 - x^2 - y^2 - z^2). Relativity does not make reality inherently subjective; it just makes it objective in a different way.

Special Relativity was a relatively easy job. Had Einstein never been born, Lorentz, Poincaré, and Minkowski would have taken care of it. Einstein got the Nobel Prize for his work on the photoelectric effect, not for Special Relativity.

General Relativity was the impressive part.

Einstein - explicitly inspired by Mach - and even though there was no experimental evidence for Mach's Principle - reformulated gravitational accelerations as a curvature of spacetime.

If you try to draw a straight line on curved paper, the curvature of the paper may twist your line, so that even as you proceed in a locally straight direction, it seems (standing back from an imaginary global viewpoint) that you have moved in a curve. Like walking "forward" for thousands of miles, and finding that you have circled the Earth.

In curved spacetime, objects under the "influence" of gravity, always seem to themselves - locally - to be proceeding along a strictly inertial pathway.

This meant you could never tell the difference between firing your rocket to accelerate through flat spacetime, and firing your rocket to stay in the same place in curved spacetime. You could accelerate the imaginary 'origin' of the universe, while changing a corresponding degree of freedom in the curvature of spacetime, and keep exactly the same laws of physics.

Einstein's theory further had the property that moving matter would generate gravitational waves, propagating curvatures. Einstein suspected that if the whole universe was rotating around you while you stood still, you would feel a centrifugal force from the incoming gravitational waves, corresponding exactly to the centripetal force of spinning your arms while the universe stood still around you. So you could construct the laws of physics in an accelerating or even rotating frame of reference, and end up observing the same laws - again freeing us of the specter of absolute space.

(I do not think this has been verified exactly, in terms of how much matter is out there, what kind of gravitational wave it would generate by rotating around us, et cetera. Einstein did verify that a shell of matter, spinning around a central point, ought to generate a gravitational equivalent of the Coriolis force that would e.g. cause a pendulum to precess. Remember that, by the basic principle of gravity as curved spacetime, this is indistinguishable in principle from a rotating inertial reference frame.)

We come now to the most important idea in all of physics. (Not counting the concept of "describe the universe using math", which I consider as the idea of physics, not an idea in physics.)

The idea is that you can start from "It shouldn't ought to be possible for X and Y to have different values from each other", or "It shouldn't ought to be possible to distinguish different values of Z", and generate new physics that make this fundamentally impossible because X and Y are now the same thing, or because Z no longer exists. And the new physics will often be experimentally verifiable.

We can interpret many of the most important revolutions in physics in these terms:

- Galileo / "The Earth is not the center of the universe": You shouldn't ought to be able to tell "where" the universe is - shifting all the objects a few feet to the left should have no effect.

- Special Relativity: You shouldn't ought to be able to tell how fast you, or the universe, are moving.

- General Relativity: You shouldn't ought to be able to tell how fast you, or the universe, are accelerating.

- Quantum mechanics: You shouldn't ought to be able to tell two identical particles apart.

Whenever you find that two things seem to always be exactly equal - like inertial mass and gravitational charge, or two electrons - it is a hint that the underlying physics are such as to make this a necessary identity, rather than a contingent equality. It is a hint that, when you see through to the underlying elements of reality, inertial mass and gravitational charge will be the same thing, not merely equal. That you will no longer be able to imagine them being different, if your imagination is over the elements of reality in the new theory.

Likewise with the way that quantum physics treats the similarity of two particles of the same species. It is not that "photon A at 1, and photon B at 2" happens to look just like "photon A at 2, and photon B at 1" but that they are the same element of reality.

When you see a seemingly contingent equality - two things that just happen to be equal, all the time, every time - it may be time to reformulate your physics so that there is one thing instead of two. The distinction you imagine is epiphenomenal; it has no experimental consequences. In the right physics, with the right elements of reality, you would no longer be able to imagine it.

The amazing thing is that this is a scientifically productive rule - finding a new representation that gets rid of epiphenomenal distinctions, often means a substantially different theory of physics with experimental consequences!

(Sure, what I just said is logically impossible, but it works.)

Sequence: Quantum Physics

Living in Many Worlds

Relative Configuration Space

Referenced by: Relative Configuration Space • Timeless Physics • Timeless Identity • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity

Original with comments: Mach's Principle: Anti-Epiphenomenal Physics

A Broken Koan

Eliezer Yudkowsky, 24 May 2008 07:04PM

At Baycon today and tomorrow. Physics series resumes tomorrow.

Meanwhile, here's a link to a page of Broken Koans and other Zen debris I ran across, which should amuse fans of ancient Eastern wisdom; and a koan of my own:

Two monks were arguing about a flag. One said, "The flag is moving."

The other said, "The wind is moving."

Julian Barbour happened to be passing by. He told them, "Not the wind, not the flag."

The first monk said, "Is the mind moving?"

Barbour replied, "Not even mind is moving."

The second monk said, "Is time moving?"

Barbour said, "There is no time. You could say that it is mu-ving."

"Then why do we think that flags flap, and wind blows, and minds change, and time moves?" inquired the first monk.

Barbour thought, and said, "Because you remember."

Original with comments: A Broken Koan

Relative Configuration Space

Eliezer Yudkowsky, 26 May 2008 09:25AM

This post is part of the Quantum Physics Sequence.
Previously in series: Mach's Principle: Anti-Epiphenomenal Physics
Followup to: Classical Configuration Spaces

Warning: The ideas in today's post are taken seriously by serious physicists, but they are not experimentally proven and are not taught as standard physics.

Today's post draws on the work of the physicist Julian Barbour, and contains diagrams stolen and/or modified from his book "The End of Time".

Previously, we saw Mach's idea (following in the earlier path of Leibniz) that inertia is resistance to relative motion. So that, if the whole universe was rotating, it would drag the inertial frame along with it. From the perspective of General Relativity, the rotating matter would generate gravitational waves.

All right: It's possible that you can't tell if the universe is rotating, because the laws of gravitation may be set up to make it look the same either way. But even if this turns out to be the case, it may not yet seem impossible to imagine that things could have been otherwise.

To expose Mach's Principle directly, we turn to Julian Barbour.

The diagrams that follow are stolen from Julian Barbour's The End of Time. I'd forgotten what an amazing book this was, or I would have stolen diagrams from it earlier to explain configuration space. Anyone interested in the nature of reality must read this book. Anyone interested in understanding modern quantum mechanics should read this book. "Must" and "should" are defined as in RFC 2119.

Suppose that we have three particles, A, B, and C, on a 2-dimensional plane; and suppose that these are the only 3 particles in the universe.

Let there be a classical configuration space which describes the 2D positions of A, B, and C. 3 classical 2D particles require a 6-dimensional configuration space.

If your monitor cannot display 6-dimensional space, I've set a 2D projection of a 3D cube to appear instead. If you see what looks like a window into an incomprehensible void, try using Firefox instead of Internet Explorer.

The thing about this 6-dimensional cube, is that it contains too much information. By looking at an exact point in this cube - supposedly corresponding to an exact state of reality - we can read off information that A, B, and C will never be able to observe.

The point (0, 1, 3, 4, 2, 5) corresponds to A at (0, 1), B at (3, 4), and C at (2, 5). Now consider the point (1, 1, 4, 4, 3, 5); which corresponds to moving A, B, and C one unit to the right, in unison.

Can A, B, and C ever detect any experimental difference? Supposing that A, B, and C can only see each other, as opposed to seeing "absolute space" in the background?

After we shift the universe to the right (shift the origin to the left), A looks around... and sees B and C at the same distance from itself as before. B and C can't detect any difference in the universe either.

Yet we have described (0, 1, 3, 4, 2, 5) and (1, 1, 4, 4, 3, 5) as two different points in the configuration space. Even though, to A, B, and C, the associated states of reality seem indistinguishable. We have postulated an epiphenomenal difference: This suggests that our physics is not over the true elements of reality. (Remember, this has been, historically, a highly productive line of reasoning! It is not just logic-chopping.)

Indeed, our classical configuration space has many epiphenomenal differences. We can rotate the three particles in unison, and end up with a different point in the configuration space; while A, B, and C again see themselves at the same distances from each other. The "rotation" that took place, was a matter of us looking at them from a different angle, from outside their universe. Which is to say the "rotation" was a choice of viewpoint for us, not an experimentally detectable fact within the ABC universe.

How can we rid the physics of mind projections and epiphenomena?

A and B and C cannot observe their absolute positions in space against a fixed background. Treating these absolute positions as elements of reality may be part of our problem.

What can A, B, and C observe? By hypothesis, they can observe their distances from each other. They can measure the distances AB, BC, and CA.

Why not use that as the dimensions of a configuration space?

At right is depicted a relative configuration space whose three dimensions are the distances AB, BC, and CA. It really is 3-dimensional, now!

If you're wondering why the configuration space looks pyramidal, it's because any point with e.g. AB + BC < CA is "outside the configuration space". It does not represent a realizable triangle, because one side is longer than the sum of the other two. Likewise AB + CA < BC and BC + CA < AB.

Every different point in this configuration space, corresponds to an experimentally different state of reality that A, B, and C can observe.

(Albeit this assumes that ABC can measure absolute, rather than relative, distances. Otherwise, different slices of pyramid-space would be observationally identical because they would describe the same triangle at different scales, as shown at left.)

(Oh, and we're assuming that A, B, and C can tell each other apart - perhaps they are different colors.)

The edges of each slice of the configuration space, are the configurations with A, B, and C on the same line. E.g., if AB + BC = CA, then B lies on a point between A and C.

The corners of each slice are the configurations in which two points coincide; e.g., AB=0, BC=CA.

At right (or possibly below, depending on your screen width), is a diagram showing a single slice in greater detail; Julian Barbour credits this to his friend Dierck Liebscher.

The point in the center of the slice corresponds to an equilateral triangle.

The dashed lines, which are axes of bilateral symmetry of the configuration space, contain points that correspond to isosceles triangles.

The curved lines are right-angled triangles.

Points "inside" the curved lines are acute triangles; points "outside" the curved lines are obtuse triangles.

What about three points coinciding?

There is no triangle at this scale where all three points coincide.

Remember, this is just one slice of the configuration space. Every point in the whole configuration space corresponds to what ABC experience as a different state of affairs.

The configuration where A, B, and C are all in the same place is unique in their experience. So it is only found in one slice of the configuration space: The slice that is a single point, at the tip of the infinite pyramid: The degenerate slice where the center and the corners are the same point: The slice that is the single point in configuration space: AB=BC=CA=0.

Julian Barbour calls this point Alpha.

But I'm getting ahead of myself, here - that sort of thing is the topic of tomorrow's post.

To see the power of a relative configuration space, observe how it makes it impossible to imagine certain epiphenomenal differences:

Put your Newtonian goggles back on: imagine A, B, and C as little billiard balls bouncing around in plain old space (not configuration space) and time. Perhaps A, B, and C attract each other via a kind of gravity, and so orbit around one another. If you were looking at the evolution of A, B, and C in plain old space and time, then a strobe-lit photograph of their motion might look like this:

In this time-series photograph, we've seen points A, B, and C forming a triangle. Not only do the points of the triangle orbit around each other, but they also seem to be heading down and to the right. It seems like you can imagine the triangle heading off up and to the right, or up and to the left, or perhaps spinning around much faster. Even though A, B, and C, who can only see their distance to each other, would never notice the difference.

Now we could also map that whole trajectory over time, onto the relative configuration space. If AB+BC+CA happens to be a constant throughout the evolution, then we could conveniently map the trajectory onto one slice of configuration space:

(This doesn't actually represent the triangle-series shown above it, but imagine that it does.)

If this is what you believe to be the reality - this trajectory in the relative configuration space - then, if I ask you to imagine, "Suppose that the triangle is heading up and to the left, instead of down and to the right", I have just uttered physical nonsense. Mapping that alternative trajectory in Newtonian space, onto the relative configuration space, would produce just the same curve. And if the laws of physics are over the relative configuration space, then this curve is all there is.

Imagine physics over trajectories in a relative configuration space like this one, but with many more particles, and perhaps 3 space dimensions. Sentient beings evolve in this universe, on some equivalent of a planet. They hunt across fields that do not seem to shift underfoot. They have a strong illusion of moving through an absolute space, against an absolute background; the relativity of motion is hidden from them.

But if the fundamental laws of their universe were over relative configurations, then it would not just be a contingent fact about their universe, that if all the particles were speeding or accelerating or rotating in unison, all the experiments would come out the same. Talking about "all the particles rotating in unison" would be physical nonsense. It only makes physical sense to talk about the velocity of some particles relative to other particles.

Your ancestors evolved on a savanna that seemed to stay put while they ran across it. You can, by an effort of mind, visualize a car that stays motionless as the world zips past, or alternatively, visualize a world that remains motionless as the car zips past. You can, by an effort of mind, see that the internal relations are the same. But it still seems to you that you are imagining two different things.

Your visual neurology is representing objects in terms of absolute positions against a fixed background. There is a web of cortical columns in your visual cortex that activate to create a mental picture. The particular columns that activate, are felt by you as positions in your visual field. That is how the algorithm feels from inside.

In a universe whose physics is over a relative configuration space, the absolute positions, and the fixed background, are not elements of reality. They are mind projection fallacies, the shadows of a point of view; as if your mind's eye were outside the universe, and the universe could move relative to that.

But if you could learn to visualize the relative configuration space, then, so long as you thought in terms of those elements of reality, it would no longer be imaginable that Mach's Principle could be false.

I am not entirely convinced of this notion of a relative configuration space. My soul as a computer programmer cries out against the idea of representing N particles with N^2 distances between them; it seems wasteful. On the other hand, I have no evidence that the Tao is prejudiced against redundant or overconstrained representations, in the same way that the Tao seems prejudiced against epiphenomena in representations. Though my soul as a programmer cries out against it, better an overconstrained representation than an epiphenomenal one. Still, it does not feel entirely satisfactory, to me. It seems like merely the best representation, not the true one.

Also, any position basis invokes an arbitrary space of simultaneity, and a relative position basis does so as well. As required by Special Relativity, the choice makes no difference - but this means that the relative position basis still contains epiphenomenal information. Perhaps the true representation will be more strictly local, in terms of invariant states of distant entanglement, as I've suggested before; and maybe, who knows, it won't be overconstrained?

Relativizing the position basis feels to me like an improvement, but it doesn't seem finished.

...

Of course, all this that we have said about the particles A, B, C and their trajectory through time, cannot possibly apply to our own universe.

In our own universe, as you may recall, there are no little billiard balls bouncing around.

In our own universe, if physics took place in a relative configuration space, it would be quantum physics in a relative configuration space. And a single moment of time, might look like this:

At right we see a cloud of red and blue mist, representing a complex amplitude distribution over the relative configuration space. You could imagine that redness is the real part and blueness is the imaginary part, or some such. But this is not a realistic amplitude distribution - just a representation of the general idea, "A cloud of complex amplitude in configuration space."

As for why only a sixth of the triangle is colored: If A, B, and C are the same species of particle, which is to say, identical particles, then the configuration space collapses along the sixfold symmetry corresponding to the six possible permutations of A, B, and C.

The whole cloud is a single static instant, in some arbitrary space of simultaneity. The quantum wavefunction is a distribution over configuration space, not a single point in configuration space. So to represent the state of the universe at a single moment, we need the whole cloud, which covers the entire collapsed configuration space.

You might naturally tend to assume that we could represent time using an animated version of this same diagram: and that the animated diagram would show the mist churning in the configuration space, the cloud's parts changing color, as amplitude flowed from volume to volume; and that as the quantum waves propagated, little blobs of amplitude density would move around through the configuration space, in trajectories much resembling the classical curve we saw earlier.

But that would be overcomplicating things.

Be aware: Churning mist in a non-relative configuration space, would be the metaphor that corresponds to the standard formulation of physics. That is, according to standard physics, the description I just gave above, would be correct (after we took it back out of the relative configuration space, which is not standard).

Yet tomorrow we shall discuss a certain further simplification of physics, which renders unimaginable still another epiphenomenal distinction, and deletes a further needless element of the laws.

Sequence: Quantum Physics

Mach's Principle: Anti-Epiphenomenal Physics

Timeless Physics

Referenced by: Timeless Physics • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity

Original with comments: Relative Configuration Space

Timeless Physics

Eliezer Yudkowsky, 27 May 2008 09:09AM

This post is part of the Quantum Physics Sequence.
Previously in series: Relative Configuration Space

Warning: The central idea in today's post is taken seriously by serious physicists; but it is not experimentally proven and is not taught as standard physics.

Today's post draws heavily on the work of the physicist Julian Barbour, and contains diagrams stolen and/or modified from his book "The End of Time". However, some of the arguments here are of my own devising, and Barbour might(?) not agree with them.

I shall begin by asking a incredibly deep question:

What time is it?

If you have the excellent habit of giving obvious answers to obvious questions, you will answer, "It is now 7:30pm [or whatever]."

How do you know?

"I know because I looked at the clock on my computer monitor."

Well, suppose I hacked into your computer and changed the clock. Would it then be a different time?

"No," you reply.

How do you know?

"Because I once used the 'Set Date and Time' facility on my computer to try and make it be the 22nd century, but it didn't work."

Ah. And how do you know that it didn't work?

"Because," you say, "I looked outside, and the buildings were still made of brick and wood and steel, rather than having been replaced by the gleaming crystal of diamondoid nanotechnological constructions; and gasoline was still only $4/gallon."

You have... interesting... expectations for the 22nd century; but let's not go into that. Suppose I replaced the buildings outside your home with confections of crystal, and raised the price of gas; then would it be 100 years later?

"No," you say, "I could look up at the night sky, and see the planets in roughly the same position as yesterday's night; with a powerful telescope I could measure the positions of the stars as they very slowly drift, relative to the Sun, and observe the rotation of distant galaxies. In these ways I would know exactly how much time had passed, no matter what you did here on Earth."

Ah. And suppose I snapped my fingers and caused all the stars and galaxies to move into the appropriate positions for 2108?

"You'd be arrested for violating the laws of physics."

But suppose I did it anyway.

"Then, still, 100 years would not have passed."

How would you know they had not passed?

"Because I would remember that, one night before, it had still been 2008. Though, realistically speaking, I would think it more likely that it was my memory at fault, not the galaxies."

Now suppose I snapped my fingers, and caused all the atoms in the universe to move into positions that would be appropriate for (one probable quantum branch) of 2108. Even the atoms in your brain.

Think carefully before you say, "It would still really be 2008." For does this belief of yours, have any observable consequences left? Or is it an epiphenomenon of your model of physics? Where is stored the fact that it is 'still 2008'? Can I snap my fingers one last time, and alter this last variable, and cause it to really be 2108?

Is it possible that Cthulhu could snap Its tentacles, and cause time for the whole universe to be suspended for exactly 10 million years, and then resume? How would anyone ever detect what had just happened?

A global suspension of time may seem imaginable, in the same way that it seems imaginable that you could "move all the matter in the whole universe ten meters to the left". To visualize the universe moving ten meters to the left, you imagine a little swirling ball of galaxies, and then it jerks leftward. Similarly, to imagine time stopping, you visualize a swirling ball of galaxies, and then it stops swirling, and hangs motionless for a while, and then starts up again.

But the sensation of passing time, in your visualization, is provided by your own mind's eye outside the system. You go on thinking, your brain's neurons firing, while, in your imagination, the swirling ball of galaxies stays motionless.

When you imagine the universe moving ten meters to the left, you are imagining motion relative to your mind's eye outside the universe. In the same way, when you imagine time stopping, you are imagining a motionless universe, frozen relative to a still-moving clock hidden outside: your own mind, counting the seconds of the freeze.

But what would it mean for 10 million "years" to pass, if motion everywhere had been suspended?

Does it make sense to say that the global rate of motion could slow down, or speed up, over the whole universe at once - so that all the particles arrive at the same final configuration, in twice as much time, or half as much time? You couldn't measure it with any clock, because the ticking of the clock would slow down too.

Do not say, "I could not detect it; therefore, who knows, it might happen every day."

Say rather, "I could not detect it, nor could anyone detect it even in principle, nor would any physical relation be affected except this one thing called 'the global rate of motion'. Therefore, I wonder what the phrase 'global rate of motion' really means."

All of that was a line of argument of Julian Barbour's, more or less, Let us pause here, and consider a second line of argument, this one my own. That is, I don't think it was in Barbour's The End of Time. (If I recall correctly, I reasoned thus even before I read Barbour, while I was coming up with my unpublished general decision theory of Newcomblike problems. Of course that does not mean the argument is novel; I have no idea whether it is novel. But if my argument is wrong, I do not want it blamed on an innocent bystander.) So:

"The future changes as we stand here, else we are the game pieces of the gods, not their heirs, as we have been promised."
-- Raistlin Majere

A fine sentiment; but what does it mean to change the future?

Suppose I have a lamp, with an old-style compact fluorescent bulb that takes a few seconds to warm up. At 7:00am, the lamp is off. At 7:01am, I flip the switch; the lamp flickers for a few moments, then begins to warm up. At 7:02am, the lamp is fully bright. Between 7:00am and 7:02am, the lamp changed from OFF to ON. This, certainly, is a change; but it is a change over time.

Change implies difference; difference implies comparison. Here, the two values being compared are (1) the state of "the lamp at 7:00am", which is OFF, and (2) the state of "the lamp at 7:02am", which is ON. So we say "the lamp" has changed from one time to another. At 7:00am, you wander by, and see the lamp is OFF; at 7:02am, you wander by, and see the lamp is ON.

But have you ever seen the future change from one time to another? Have you wandered by a lamp at exactly 7:02am, and seen that it is OFF; then, a bit later, looked in again on the "the lamp at exactly 7:02am", and discovered that it is now ON?

Naturally, we often feel like we are "changing the future". Logging on to your online bank account, you discover that your credit card bill comes due tomorrow, and, for some reason, has not been paid automatically. Imagining the future-by-default - extrapolating out the world as it would be without any further actions - you see that the bill not being paid, and interest charges accruing on your credit card. So you pay the bill online. And now, imagining tomorrow, it seems to you that the interest charges will not occur. So at 1:00pm, you imagined a future in which your credit card accrued interest charges, and at 1:02pm, you imagined a future in which it did not. And so your imagination of the future changed, from one time to another.

As I remarked previously: The way a belief feels from inside, is that you seem to be looking straight at reality. When it actually seems that you're looking at a belief, as such, you are really experiencing a belief about your beliefs.

When your extrapolation of the future changes, from one time to another, it feels like the future itself is changing. Yet you have never seen the future change. When you actually get to the future, you only ever see one outcome.

How could a single moment of time, change from one time to another?

I am not going to go into "free will" in today's blog post. Except to remark that if you have been reading Overcoming Bias all this time, and you are currently agonizing about whether or not you really have free will, instead of trying to understand where your own mind has become confused and generated an impossible question, you should probably go back and read it all again. For anyone who is just now joining us... perhaps I shall discuss the issue tomorrow.

Just remember Egan's Law: It all adds up to normality. Apples didn't stop falling when Einstein disproved Newton's theory of gravity, and anyone who jumped off a cliff would still go splat. Perhaps Time turns out to work differently than you thought; but tomorrow still lies ahead of you, and your choices, and their consequences. I wouldn't advise reworking your moral philosophy based on confusing arguments and strange-seeming physics, until the physics stops appearing strange and the arguments no longer seem confusing.

Now to physics we turn; and here I resume drawing my ideas from Julian Barbour.

For the benefit of anyone who hasn't followed the series on quantum mechanics, a very very quick summary:

- In classical physics - the mistaken physics that was developed first historically, and matches human intuitions all too well - a particle is like a little billiard ball. A particle is in a single place in 3D space, and we can describe its position with three real numbers. In quantum physics, we need an amplitude distribution over all possible positions for the particle - a complex number for the particle being here, a complex number for the particle being there, and so on through all the positions in space; a continuous distribution. (Configurations and Amplitude.)

- In classical physics, we can consider each particle independently. This particle is here, that particle is there. In quantum physics this is not possible; we can only assign amplitudes to configurations that describe the simultaneous positions of many particles. In fact, the only mathematical entities that actually have amplitudes are joint configurations of all the particles in the entire universe. (Joint Configurations.)

Above is a diagram that shows what a configuration space might look like for three particles, A, B, and C. ABC form a triangle in two-dimensional space. Every individual point in the configuration space corresponds to a simultaneous position of all the particles - above we see points that correspond to particular triangles i.e. joint positions of A, B, and C. (Classical Configuration Spaces; The Quantum Arena.)

The state of a quantum system is not a single point in this space; it is a distribution over this space. You could imagine it as a cloud, or a blob, or a colored mist within the space.

Here we see a relative configuration space, in which each axis is the distance between a pair of particles. This has some advantages I'm not going to recapitulate (it was covered in a previous post), so if you're dropping into the middle of the series, just pretend it's a regular configuration space.

We've just chopped up the pyramidal space you saw before, into a series of slices. In this configuration space, the slices near the bottom show all the particles close together (tiny triangles). As we rise up, the particles get further apart (larger triangles).

At the very bottom of the configuration space is a configuration where all the particles occupy the same position.

(But remember, it's nonsense to talk about an individual particle being anywhere in a configuration space - each point in the configuration space corresponds to a position of all the particles. Configuration space is not the 3D space you know. It's not that there are a bunch of particles resting in the same place at the bottom. The single bottom point corresponds to all the particles being in the same place in 3D space.)

Here we take a closer look at one of the slices of configuration space, and see a cloud of blue and red mist covering some of it. (Why am I only showing the cloud covering a sixth (exactly a sixth) of the triangle? This has to do with a symmetry in the space - exchanges of identical particles - which is not important to the present discussion.)

But there is your glimpse of some quantum mist - in two colors, because amplitudes are complex numbers with a real and imaginary part. An amplitude distribution or "wavefunction" assigns a complex number to every point in the continuous configuration space - a complex number to every possible configuration of all the particles.

Yesterday, I finished by asking how the state of a quantum system might evolve over time.

You might be tempted to visualize the mist churning and changing colors, as quantum amplitude flows within the configuration space.

And this is indeed the way that you would visualize standard physics.

Behold the standard Schrödinger Equation:

Here ψ(r, t) is the amplitude distribution over configuration space (r) and time (t). The left-hand side of the Schrödinger Equation is the change over time of the wavefunction ψ, and the right-hand-side shows how to calculate this change as the sum of two terms: The gradient of the wavefunction over configuration space (at that time), and the potential energy of each configuration.

Which is to say, the derivative in time of the wavefunction - the instantaneous rate of change - can be in terms of the wavefunction's derivative in space, plus a term for the potential energy.

If you tried to visualize Schrödinger's Equation - doesn't look too hard, right? - you'd see a blob of churning, complex mist in configuration space, with little blobs racing around and splitting into smaller blobs as waves propagated.

If you tried to calculate the quantum state of a single hydrogen atom over time, apart from the rest of the universe - which you can only really do if the hydrogen atom isn't entangled with anything - the atom's quantum state would evolve over time; the mist would churn.

But suppose you think about the whole universe at once, including yourself, of course. Because - even in the standard model of quantum physics! - that is exactly the arena in which quantum physics takes place: A wavefunction over all the particles, everywhere.

If you can sensibly talk about the quantum state of some particular hydrogen atom, it's only because the wavefunction happens to neatly factor into (hydrogen atom) * (rest of world).

Even if the hydrogen atom is behaving in a very regular way, the joint wavefunction for (hydrogen atom * rest of world) may not be so regular. Stars move into new positions, people are born and people die, digital watches tick, and the cosmos expands: The universe is non-recurrent.

Think of how the universal wavefunction ψ(r, t) might behave when r is the position of all the particles in the universe.

Let's call 9:00am the time t=0, measured in seconds.

At ψ(r, t=0), then, you are wondering what time it is: The particles making up the neurons in your brain, are in positions ryou that correspond to neurons firing in the thought-pattern "What time is it?" And the Earth, and the Sun, and the rest of the universe, have their own particles in the appropriate rrest-of-universe. Where the complete r roughly factorizes as the product (ryou * rrest-of-universe).

Over the next second, the joint wavefunction of the entire universe evolves into ψ(r, t=1). All the stars in the sky have moved a little bit onward, in whatever direction they're heading; the Sun has burned up a little more of its hydrogen; on Earth, an average of 1.8 people have died; and you've just glanced down at your watch.

At ψ(r, t=2), the stars have moved a little onward, the galaxies have rotated, the cosmos has expanded a little more (and its expansion has accelerated a little more), your watch has evolved into the state of showing 9:00:02 AM on its screen, and your own mind has evolved into the state of thinking the thought, "Huh, I guess it's nine o' clock."

Ready for the next big simplification in physics?

Here it is:

We don't need the t.

It's redundant.

The r never repeats itself. The universe is expanding, and in every instant, it gets a little bigger. We don't need a separate t to keep things straight. When you're looking at the whole universe, a unique function ψ of (r, t) is pretty much a unique function of r.

And the only way we know in the first place "what time it is", is by looking at clocks. And whether the clock is a wristwatch, or the expansion of the universe, or your own memories, that clock is encoded in the position of particles - in the r. We have never seen a t variable apart from the r.

We can recast the quantum wave equations, specifying the time evolution of ψ(r, t), as specifying relations within a wavefunction ψ(r).

Occam's Razor: Our equations don't need a t in them, so we can banish the t and make our ontology that much simpler.

An unchanging quantum mist hangs over the configuration space, not churning, not flowing.

But the mist has internal structure, internal relations; and these contain time implicitly.

The dynamics of physics - falling apples and rotating galaxies - is now embodied within the unchanging mist in the unchanging configuration space.

This landscape is not frozen like a cryonics patient suspended in liquid nitrogen. It is not motionless as an isolated system while the rest of the universe goes on without it.

The landscape is timeless; time exists only within it. To talk about time, you have to talk about relations inside the configuration space.

Asking "What happened before the Big Bang?" is revealed as a wrong question. There is no "before"; a "before" would be outside the configuration space. There was never a pre-existing emptiness into which our universe exploded. There is just this timeless mathematical object, time existing within it; and the object has a natural boundary at the Big Bang. You cannot ask "When did this mathematical object come into existence?" because there is no t outside it.

So that is Julian Barbour's proposal for the next great simplification project in physics.

(And yes, you can not only fit General Relativity into this paradigm, it actually comes out looking even more elegant than before. For which point I refer you to Julian Barbour's papers.)

Tomorrow, I'll go into some of my own thoughts and reactions to this proposal.

But one point seems worth noting immediately: I have spoken before on the apparently perfect universality of physical laws, that apply everywhere and everywhen. We have just raised this perfection to an even higher pitch: everything that exists is either perfectly global or perfectly local. There are points in configuration space that affect only their immediate neighbors in space and time; governed by universal laws of physics. Perfectly local, perfectly global. If the meaning and sheer beauty of this statement is not immediately obvious, I'll go into it tomorrow.

And a final intuition-pump, in case you haven't yet gotten timelessness on a gut level...

Think of this as a diagram of the many worlds of quantum physics. The branch points could be, say, your observation of a particle that seems to go either "left" or "right".

Looking back from the vantage point of the gold head, you only remember having been the two green heads.

So you seem to remember Time proceeding along a single line. You remember that the particle first went left, and then went right. You ask, "Which way will the particle go this time?"

You only remember one of the two outcomes that occurred on each occasion. So you ask, "When I make my next observation, which of the two possible worlds will I end up in?"

Remembering only a single line as your past, you try to extend that line into the future -

But both branches, both future versions of you, just exist. There is no fact of the matter as to "which branch you go down". Different versions of you experience both branches.

So that is many-worlds.

And to incorporate Barbour, we simply say that all of these heads, all these Nows, just exist. They do not appear and then vanish; they just are. From a global perspective, there is no answer to the question, "What time is it?" There are just different experiences at different Nows.

From any given vantage point, you look back, and remember other times - so that the question, "Why is it this time right now, rather than some other time?" seems to make sense. But there is no answer.

When I came to this understanding, I forgot the meaning that Time had once held for me.

Time has dissolved for me, has been reduced to something simpler that is not itself timeful.

I can no longer conceive that there might really be a universal time, which is somehow "moving" from the past to the future. This now seems like nonsense.

Something like Barbour's timeless physics has to be true, or I'm in trouble: I have forgotten how to imagine a universe that has "real genuine time" in it.

Sequence: Quantum Physics

Relative Configuration Space

Timeless Beauty

Referenced by: Timeless Beauty • Timeless Causality • Einstein's Superpowers • Class Project • Timeless Identity • Thou Art Physics • Timeless Control • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity • Heading Toward Morality • Math is Subjunctively Objective • Against Modal Logics

Original with comments: Timeless Physics

Timeless Beauty

Eliezer Yudkowsky, 28 May 2008 04:32AM

This post is part of the Quantum Physics Sequence.
Followup to: Timeless Physics

One of the great surprises of humanity's early study of physics was that there were universal laws, that the heavens were governed by the same order as the Earth: Laws that hold in all times, in all places, without known exception. Sometimes we discover a seeming exception to the old law, like Mercury's precession, but soon it turns out to perfectly obey a still deeper law, that once again is universal as far as the eye can see.

Every known law of fundamental physics is perfectly global. We know no law of fundamental physics that applies on Tuesdays but not Wednesdays, or that applies in the Northern hemisphere but not the Southern.

In classical physics, the laws are universal; but there are also other entities that are neither perfectly global nor perfectly local. Like the case I discussed yesterday, of an entity called "the lamp" where "the lamp" is OFF at 7:00am but ON at 7:02am; the lamp entity extends through time, and has different values at different times. The little billiard balls are like that in classical physics; a classical billiard ball is (alleged to be) a fundamentally existent entity, but it has a world-line, not a world-point.

In timeless physics, everything that exists is either perfectly global or perfectly local. The laws are perfectly global. The configurations are perfectly local - every possible arrangement of particles has a single complex amplitude assigned to it, which never changes from one time to another. Each configuration only affects, and is affected by, its immediate neighbors. Each actually existent thing is perfectly unique, as a mathematical entity.

Newton, first to combine the Heavens and the Earth with a truly universal generalization, saw a clockwork universe of moving billiard balls and their world-lines, governed by perfect exceptionless laws. Newton was the first to look upon a greater beauty than any mere religion had ever dreamed.

But the beauty of classical physics doesn't begin to compare to the beauty of timeless quantum physics.

Timeful quantum physics is pretty, but it's not all that much prettier than classical physics. In timeful physics the "same configuration" can still have different values at different times, its own little world-line, like a lamp switching from OFF to ON. There's that ugly t complicating the equations.

You can see the beauty of timeless quantum physics by noticing how much easier it is to mess up the perfection, if you try to tamper with Platonia.

Consider the collapse interpretation of quantum mechanics. To people raised on timeful quantum physics, "the collapse of the wavefunction" sounds like it might be a plausible physical mechanism.

If you step back and look upon the timeless mist over the entire configuration space, all dynamics manifest in its perfectly local relations, then the "pruning" process of collapse suddenly shows up as a hugely ugly discontinuity in the timeless object. Instead of a continuous mist, we have something that looks like a maimed tree with branches hacked off and sap-bleeding stumps left behind. The perfect locality is ruined, because whole branches are hacked off in one operation. Likewise, collapse destroys the perfect global uniformity of the laws that relate each configuration to its neighborhood; sometimes we have the usual relation of amplitude flow, and then sometimes we have the collapsing-relation instead.

This is the power of beauty: The more beautiful something is, the more obvious it becomes when you mess it up.

I was surprised that many of yesterday's commenters seemed to think that Barbour's timeless physics was nothing new, relative to the older idea of a Block Universe. 3+1D Minkowskian spacetime has no privileged space of simultaneity, which, in its own way, seems to require you to throw out the concept of a global now. From Minkowskian 3+1, I had the idea of "time as a single perfect 4D crystal" - I didn't know the phrase "Block Universe", but seemed evident enough.

Nonetheless, I did not really get timelessness until I read Barbour. Saying that the t coordinate was just another coordinate, didn't have nearly the same impact on me as tossing the t coordinate out the window.

Special Relativity is widely accepted, but that doesn't stop people from talking about "nonlocal collapse" or "retrocausation" - relativistic timeful QM isn't beautiful enough to protect itself from complication.

Shane Legg's reaction is the effect I was looking for:

"Stop it! If I intuitively took on board your timeless MWI view of the world... well, I'm worried that this might endanger my illusion of consciousness. Thinking about it is already making me feel a bit weird."

I wish I knew whether the unimpressed commenters got what Shane Legg did, just from hearing about Special Relativity; or if they still haven't gotten it yet from reading my brief summary of Barbour.

But in any case, let me talk in principle about why it helps to toss out the t coordinate:

To reduce a thing, you must reduce it to something that does not itself have the property you want to explain.

In old-school Artificial Intelligence, a researcher wonders where the meaning of a word like "apple" comes from. They want to get knowledge about "apples" into their beloved AI system, so they create a LISP token named apple. They realize that if they claim the token is meaningful of itself, they have not really reduced the nature of meaning... So they assert that "the apple token is not meaningful by itself", and then go on to say, "The meaning of the apple token emerges from its network of connections to other tokens." This is not true reductionism. It is wrapping up your confusion in a gift-box.

To reduce time, you must reduce it to something that is not time. It is not enough to take the t coordinate, and say that it is "just another dimension". So long as the t coordinate is there, it acts as a mental sponge that can soak up all the time-ness that you want to explain. If you toss out the t coordinate, you are forced to see time as something else, and not just see time as "time".

Tomorrow (if I can shake today's cold) I'll talk about one of my points of departure from Barbour: Namely, I have no problem with discarding time and keeping causality. The commenters who complained about Barbour grinding up the universe into disconnected slices, may be reassured: On this point, I think Barbour is trying too hard. We can discard t, and still keep causality within r.

I dare to disagree with Barbour, on this point, because it seems plausible that Barbour has not studied Judea Pearl and colleagues' formulation of causality -

- which likewise makes no use of a t coordinate.

Pearl et. al.'s formulation of "causality" would not be anywhere near as enlightening, if they had to put t coordinates on everything for the math to make sense. Even if the authors insisted that t was "just another property" or "just another number"... well, if you've read Pearl, you see my point. It would correspond to a much weaker understanding.

Sequence: Quantum Physics

Timeless Physics

Timeless Causality

Referenced by: Timeless Identity • Timeless Control • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity

Original with comments: Timeless Beauty

Timeless Causality

Eliezer Yudkowsky, 29 May 2008 06:45AM

This post is part of the Quantum Physics Sequence.
Followup to: Timeless Physics

Julian Barbour believes that each configuration, each individual point in configuration space, corresponds individually to an experienced Now - that each instantaneous time-slice of a brain is the carrier of a subjective experience.

On this point, I take it upon myself to disagree with Barbour.

There is a timeless formulation of causality, known to Bayesians, which may glue configurations together even in a timeless universe. Barbour may not have studied this; it is not widely studied.

Such causal links could be required for "computation" and "consciousness" - whatever those are. If so, we would not be forced to conclude that a single configuration, encoding a brain frozen in time, can be the bearer of an instantaneous experience. We could throw out time, and keep the concept of causal computation.

There is an old saying: "Correlation does not imply causation." I don't know if this is my own thought, or something I remember hearing, but on seeing this saying, a phrase ran through my mind: If correlation does not imply causation, what does?

Suppose I'm at the top of a canyon, near a pile of heavy rocks. I throw a rock over the side, and a few seconds later, I hear a crash. I do this again and again, and it seems that the rock-throw, and the crash, tend to correlate; to occur in the presence of each other. Perhaps the sound of the crash is causing me to throw a rock off the cliff? But no, this seems unlikely, for then an effect would have to precede its cause. It seems more likely that throwing the rock off the cliff is causing the crash. If, on the other hand, someone observed me on the cliff, and saw a flash of light, and then immediately afterward saw me throw a rock off the cliff, they would suspect that flashes of light caused me to throw rocks.

Perhaps correlation, plus time, can suggest a direction of causality?

But we just threw out time.

You see the problem here.

Once, sophisticated statisticians believed this problem was unsolvable. Many thought it was unsolvable even with time. Time-symmetrical laws of physics didn't seem to leave room for asymmetrical causality. And in statistics, nobody thought there was any way to define causality. They could measure correlation, and that was enough. Causality was declared dead, and the famous statistician R. A. Fisher testified that it was impossible to prove that smoking cigarettes actually caused cancer.

Anyway...

Let's say we have a data series, generated by taking snapshots over time of two variables 1 and 2. We have a large amount of data from the series, laid out on a track, but we don't know the direction of time on the track. On each round, the past values of 1 and 2 probabilistically generate the future value of 1, and then separately probabilistically generate the future value of 2. We know this, but we don't know the actual laws. We can try to infer the laws by gathering statistics about which values of 1 and 2 are adjacent to which other values of 1 and 2. But we don't know the global direction of time, yet, so we don't know if our statistic relates the effect to the cause, or the cause to the effect.

When we look at an arbitrary value-pair and its neighborhood, let's call the three slices L, M, and R for Left, Middle, and Right.

We are considering two hypotheses. First, that causality could be flowing from L to M to R:

Second, that causality could be flowing from R to M to L:

As good Bayesians, we realize that to distinguish these two hypotheses, we must find some kind of observation that is more likely in one case than in the other. But what might such an observation be?

We can try to look at various slices M, and try to find correlations between the values of M, and the values of L and R. For example, we could find that when M1 is in the + state, that R2 is often also in the + state. But is this because R2 causes M1 to be +, or because M1 causes R2 to be +?

If throwing a rock causes the sound of a crash, then the throw and the crash will tend to occur in each other's presence. But this is also true if the sound of the crash causes me to throw a rock. So observing these correlations does not tell us the direction of causality, unless we already know the direction of time.

From looking at this undirected diagram, we can guess that M1 will correlate to L1, M2 will correlate to R1, R2 will correlate to M2, and so on; and all this will be true because there are lines between the two nodes, regardless of which end of the line we try to draw the arrow upon. You can see the problem with trying to derive causality from correlation!

Could we find that when M1 is +, R2 is always +, but that when R2 is +, M1 is not always +, and say, "M1 must be causing R2"? But this does not follow. We said at the beginning that past values of 1 and 2 were generating future values of 1 and 2 in a probabilistic way; it was nowhere said that we would give preference to laws that made the future deterministic given the past, rather than vice versa. So there is nothing to make us prefer the hypothesis, "A + at M1 always causes R2 to be +" to the hypothesis, "M1 can only be + in cases where its parent R2 is +".

Ordinarily, at this point, I would say: "Now I am about to tell you the answer; so if you want to try to work out the problem on your own, you should do so now." But in this case, some of the greatest statisticians in history did not get it on their own, so if you do not already know the answer, I am not really expecting you to work it out. Maybe if you remember half a hint, but not the whole answer, you could try it on your own. Or if you suspect that your era will support you, you could try it on your own; I have given you a tremendous amount of help by asking exactly the correct question, and telling you that an answer is possible.

...

So! Instead of thinking in terms of observations we could find, and then trying to figure out if they might distinguish asymmetrically between the hypotheses, let us examine a single causal hypothesis and see if it implies any asymmetrical observations.

Say the flow of causality is from left to right:

Suppose that we do know L1 and L2, but we do not know R1 and R2. Will learning M1 tell us anything about M2?

That is, will we observe the conditional dependence

P(M2|L1,L2) ≠ P(M2|M1,L1,L2)

to hold? The answer, on the assumption that causality flows to the right, and on the other assumptions previously given, is no. "On each round, the past values of 1 and 2 probabilistically generate the future value of 1, and then separately probabilistically generate the future value of 2." So once we have L1 and L2, they generate M1 independently of how they generate M2.

But if we did know R1 or R2, then, on the assumptions, learning M1 would give us information about M2. Suppose that there are siblings Alpha and Betty, cute little vandals, who throw rocks when their parents are out of town. If the parents are out of town, then either Alpha or Betty might each, independently, decide to throw a rock through my window. If I don't know whether a rock has been thrown through my window, and I know that Alpha didn't throw a rock through my window, that doesn't affect my probability estimate that Betty threw a rock through my window - they decide independently. But if I know my window is broken, and I know Alpha didn't do it, then I can guess Betty is the culprit. So even though Alpha and Betty throw rocks independently of each other, knowing the effect can epistemically entangle my beliefs about the causes.

Similarly, if we didn't know L1 or L2, then M1 should give us information about M2, because from the effect M1 we can infer the state of its causes L1 and L2, and thence the effect of L1/L2 on M2. If I know that Alpha threw a rock, then I can guess that Alpha and Betty's parents are out of town, and that makes it more likely that Betty will throw a rock too.

Which all goes to say that, if causality is flowing from L to M to R, we may indeed expect the conditional dependence

P(M2|R1,R2) ≠ P(M2|M1,R1,R2)

to hold.

So if we observe, statistically, over many time slices:

P(M2|L1,L2) = P(M2|M1,L1,L2)
P(M2|R1,R2) ≠ P(M2|M1,R1,R2)

Then we know causality is flowing from left to right; and conversely if we see:

P(M2|L1,L2) ≠ P(M2|M1,L1,L2)
P(M2|R1,R2) = P(M2|M1,R1,R2)

Then we can guess causality is flowing from right to left.

This trick used the assumption of probabilistic generators. We couldn't have done it if the series had been generated by bijective mappings, i.e., if the future was deterministic given the past and only one possible past was compatible with each future.

So this trick does not directly apply to reading causality off of Barbour's Platonia (which is the name Barbour gives to the timeless mathematical object that is our universe).

However, think about the situation if humanity sent off colonization probes to distant superclusters, and then the accelerating expansion of the universe put the colonies over the cosmological horizon from us. There would then be distant human colonies that could not speak to us again: Correlations in a case where light, going forward, could not reach one colony from another, or reach any common ground.

On the other hand, we would be very surprised to reach a distant supercluster billions of light-years away, and find a spaceship just arriving from the other side of the universe, sent from another independently evolved Earth, which had developed genetically compatible indistinguishable humans who speak English. (A la way too much horrible sci-fi television.) We would not expect such extraordinary similarity of events, in a historical region where a ray of light could not yet have reached there from our Earth, nor a ray of light reached our Earth from there, nor could a ray of light reached both Earths from any mutual region between. On the assumption, that is, that rays of light travel in the direction we call "forward".

When two regions of spacetime are timelike separated, we cannot deduce any direction of causality from similarities between them; they could be similar because one is cause and one is effect, or vice versa. But when two regions of spacetime are spacelike separated, and far enough apart that they have no common causal ancestry assuming one direction of physical causality, but would have common causal ancestry assuming a different direction of physical causality, then similarity between them... is at least highly suggestive.

I am not skilled enough in causality to translate probabilistic theorems into bijective deterministic ones. And by calling certain similarities "surprising" I have secretly imported a probabilistic view; I have made myself uncertain so that I can be surprised.

But Judea Pearl himself believes that the arrows of his graphs are more fundamental than the statistical correlations they produce; he has said so in an essay entitled "Why I Am Only A Half-Bayesian". Pearl thinks that his arrows reflect reality, and hence, that there is more to inference than just raw probability distributions. If Pearl is right, then there is no reason why you could not have directedness in bijective deterministic mappings as well, which would manifest in the same sort of similarity/dissimilarity rules I have just described.

This does not bring back time. There is no t coordinate, and no global now sweeping across the universe. Events do not happen in the past or the present or the future, they just are. But there may be a certain... asymmetric locality of relatedness... that preserves "cause" and "effect", and with it, "therefore". A point in configuration space would never be "past" or "present" or "future", nor would it have a "time" coordinate, but it might be "cause" or "effect" to another point in configuration space.

I am aware of the standard argument that anything resembling an "arrow of time" should be made to stem strictly from the second law of thermodynamics and the low-entropy initial condition. But if you throw out causality along with time, it is hard to see how a low-entropy terminal condition and high-entropy initial condition could produce the same pattern of similar and dissimilar regions. Look at in another way: To compute a consistent universe with a low-entropy terminal condition and high-entropy initial condition, you have to simulate lots and lots of universes, then throw away all but a tiny fraction of them that end up with low entropy at the end. With a low-entropy initial condition, you can compute it out locally, without any global checks. So I am not yet ready to throw out the arrowheads on my arrows.

And, if we have "therefore" back, if we have "cause" and "effect" back - and science would be somewhat forlorn without them - then we can hope to retrieve the concept of "computation". We are not forced to grind up reality into disconnected configurations; there can be glue between them. We can require the amplitude relations between connected volumes of configuration space, to carry out some kind of timeless computation, before we decide that it contains the timeless Now of a conscious mind. We are not forced to associate experience with an isolated point in configuration space - which is a good thing from my perspective, because it doesn't seem to me that a frozen brain with all the particles in fixed positions ought to be having experiences. I would sooner associate experience with the arrows than the nodes, if I had to pick one or the other! I would sooner associate consciousness with the change in a brain than with the brain itself, if I had to pick one or the other.

This also lets me keep, for at least a little while longer, the concept of a conscious mind being connected to its future Nows, and anticipating some future experiences rather than others. Perhaps I will have to throw out this idea eventually, because I cannot seem to formulate it consistently; but for now, at least, I still cannot do without the notion of a "conditional probability". It still seems to me that there is some actual connection that makes it more likely for me to wake up tomorrow as Eliezer Yudkowsky, than as Britney Spears. If I am in the arrows even more than the nodes, that gives me a direction, a timeless flow. This may possibly be naive, but I am sticking with it until I can jump to an alternative that is less confusing than my present confused state of mind.

Don't think that any of this preserves time, though, or distinguishes the past from the future. I am just holding onto cause and effect and computation and even anticipation for a little while longer.

Sequence: Quantum Physics

Timeless Beauty

Timeless Identity

Referenced by: Timeless Identity • Thou Art Physics • Timeless Control • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity • The Ultimate Source • Dreams of AI Design • Against Modal Logics

Original with comments: Timeless Causality

Einstein's Superpowers

Eliezer Yudkowsky, 30 May 2008 06:40AM

Followup to: Einstein's Speed, My Childhood Role Model, Timeless Physics

There is a widespread tendency to talk (and think) as if Einstein, Newton, and similar historical figures had superpowers - something magical, something sacred, something beyond the mundane. (Remember, there are many more ways to worship a thing than lighting candles around its altar.)

Once I unthinkingly thought this way too, with respect to Einstein in particular, until reading Julian Barbour's The End of Time cured me of it.

Barbour laid out the history of anti-epiphenomenal physics and Mach's Principle; he described the historical controversies that predated Mach - all this that stood behind Einstein and was known to Einstein, when Einstein tackled his problem...

And maybe I'm just imagining things - reading too much of myself into Barbour's book - but I thought I heard Barbour very quietly shouting, coded between the polite lines:

What Einstein did isn't magic, people! If you all just looked at how he actually did it, instead of falling to your knees and worshiping him, maybe then you'd be able to do it too!

Maybe I'm mistaken, or extrapolating too far... but I kinda suspect that Barbour once tried to explain to people how you move further along Einstein's direction to get timeless physics; and they sniffed scornfully and said, "Oh, you think you're Einstein, do you?"

John Baez's Crackpot Index, item 18:

10 points for each favorable comparison of yourself to Einstein, or claim that special or general relativity are fundamentally misguided (without good evidence).

Item 30:

30 points for suggesting that Einstein, in his later years, was groping his way towards the ideas you now advocate.

Barbour never bothers to compare himself to Einstein, of course; nor does he ever appeal to Einstein in support of timeless physics. I mention these items on the Crackpot Index by way of showing how many people compare themselves to Einstein, and what society generally thinks of them.

The crackpot sees Einstein as something magical, so they compare themselves to Einstein by way of praising themselves as magical; they think Einstein had superpowers and they think they have superpowers, hence the comparison.

But it is just the other side of the same coin, to think that Einstein is sacred, and the crackpot is not sacred, therefore they have committed blasphemy in comparing themselves to Einstein.

Suppose a bright young physicist says, "I admire Einstein's work, but personally, I hope to do better." If someone is shocked and says, "What! You haven't accomplished anything remotely like what Einstein did; what makes you think you're smarter than him?" then they are the other side of the crackpot's coin.

The underlying problem is conflating social status and research potential.

Einstein has extremely high social status: because of his record of accomplishments; because of how he did it; and because he's the physicist whose name even the general public remembers, who brought honor to science itself.

And we tend to mix up fame with other quantities, and we tend to attribute people's behavior to dispositions rather than situations.

So there's this tendency to think that Einstein, even before he was famous, already had an inherent disposition to be Einstein - a potential as rare as his fame and as magical as his deeds. So that if you claim to have the potential to do what Einstein did, it is just the same as claiming Einstein's rank, rising far above your assigned status in the tribe.

I'm not phrasing this well, but then, I'm trying to dissect a confused thought: Einstein belongs to a separate magisterium, the sacred magisterium. The sacred magisterium is distinct from the mundane magisterium; you can't set out to be Einstein in the way you can set out to be a full professor or a CEO. Only beings with divine potential can enter the sacred magisterium - and then it is only fulfilling a destiny they already have. So if you say you want to outdo Einstein, you're claiming to already be part of the sacred magisterium - you claim to have the same aura of destiny that Einstein was born with, like a royal birthright...

"But Eliezer," you say, "surely not everyone can become Einstein."

You mean to say, not everyone can do better than Einstein.

"Um... yeah, that's what I meant."

Well... in the modern world, you may be correct. You probably should remember that I am a transhumanist, going around looking around at people thinking, "You know, it just sucks that not everyone has the potential to do better than Einstein, and this seems like a fixable problem." It colors one's attitude.

But in the modern world, yes, not everyone has the potential to be Einstein.

Still... how can I put this...

There's a phrase I once heard, can't remember where: "Just another Jewish genius." Some poet or author or philosopher or other, brilliant at a young age, doing something not tremendously important in the grand scheme of things, not all that influential, who ended up being dismissed as "Just another Jewish genius."

If Einstein had chosen the wrong angle of attack on his problem - if he hadn't chosen a sufficiently important problem to work on - if he hadn't persisted for years - if he'd taken any number of wrong turns - or if someone else had solved the problem first - then dear Albert would have ended up as just another Jewish genius.

Geniuses are rare, but not all that rare. It is not all that implausible to lay claim to the kind of intellect that can get you dismissed as "just another Jewish genius" or "just another brilliant mind who never did anything interesting with their life". The associated social status here is not high enough to be sacred, so it should seem like an ordinarily evaluable claim.

But what separates people like this from becoming Einstein, I suspect, is no innate defect of brilliance. It's things like "lack of an interesting problem" - or, to put the blame where it belongs, "failing to choose an important problem". It is very easy to fail at this because of the cached thought problem: Tell people to choose an important problem and they will choose the first cache hit for "important problem" that pops into their heads, like "global warming" or "string theory".

The truly important problems are often the ones you're not even considering, because they appear to be impossible, or, um, actually difficult, or worst of all, not clear how to solve. If you worked on them for years, they might not seem so impossible... but this is an extra and unusual insight; naive realism will tell you that solvable problems look solvable, and impossible-looking problems are impossible.

Then you have to come up with a new and worthwhile angle of attack. Most people who are not allergic to novelty, will go too far in the other direction, and fall into an affective death spiral.

And then you've got to bang your head on the problem for years, without being distracted by the temptations of easier living. "Life is what happens while we are making other plans," as the saying goes, and if you want to fulfill your other plans, you've often got to be ready to turn down life.

Society is not set up to support you while you work, either.

The point being, the problem is not that you need an aura of destiny and the aura of destiny is missing. If you'd met Albert before he published his papers, you would have perceived no aura of destiny about him to match his future high status. He would seem like just another Jewish genius.

This is not because the royal birthright is concealed, but because it simply is not there. It is not necessary. There is no separate magisterium for people who do important things.

I say this, because I want to do important things with my life, and I have a genuinely important problem, and an angle of attack, and I've been banging my head on it for years, and I've managed to set up a support structure for it; and I very frequently meet people who, in one way or another, say: "Yeah? Let's see your aura of destiny, buddy."

What impressed me about Julian Barbour was a quality that I don't think anyone would have known how to fake without actually having it: Barbour seemed to have seen through Einstein - he talked about Einstein as if everything Einstein had done was perfectly understandable and mundane.

Though even having realized this, to me it still came as a shock, when Barbour said something along the lines of, "Now here's where Einstein failed to apply his own methods, and missed the key insight -" But the shock was fleeting, I knew the Law: No gods, no magic, and ancient heroes are milestones to tick off in your rearview mirror.

This seeing through is something one has to achieve, an insight one has to discover. You cannot see through Einstein just by saying, "Einstein is mundane!" if his work still seems like magic unto you. That would be like declaring "Consciousness must reduce to neurons!" without having any idea of how to do it. It's true, but it doesn't solve the problem.

I'm not going to tell you that Einstein was an ordinary bloke oversold by the media, or that deep down he was a regular schmuck just like everyone else. That would be going much too far. To walk this path, one must acquire abilities some consider to be... unnatural. I take a special joy in doing things that people call "humanly impossible", because it shows that I'm growing up.

Yet the way that you acquire magical powers is not by being born with them, but by seeing, with a sudden shock, that they really are perfectly normal.

This is a general principle in life.

Sequence: Quantum Physics

My Childhood Role Model

Class Project

Sequence: Challenging the Difficult

Something to Protect

Trying to Try

Referenced by: Class Project • Principles of Disagreement • Timeless Identity • The Quantum Physics Sequence • The Level Above Mine • Competent Elites • Nonperson Predicates • ...And Say No More Of It • Teaching the Unteachable • Working Mantras

Original with comments: Einstein's Superpowers

Class Project

Eliezer Yudkowsky, 31 May 2008 12:23AM

Followup to: The Failures of Eld Science, Einstein's Superpowers

"Do as well as Einstein?" Jeffreyssai said, incredulously. "Just as well as Einstein? Albert Einstein was a great scientist of his era, but that was his era, not this one! Einstein did not comprehend the Bayesian methods; he lived before the cognitive biases were discovered; he had no scientific grasp of his own thought processes. Einstein spoke nonsense of an impersonal God - which tells you how well he understood the rhythm of reason, to discard it outside his own field! He was too caught up in the drama of rejecting his era's quantum mechanics to actually fix it. And while I grant that Einstein reasoned cleanly in the matter of General Relativity - barring that matter of the cosmological constant - he took ten years to do it. Too slow!"

"Too slow?" repeated Taji incredulously.

"Too slow! If Einstein were in this classroom now, rather than Earth of the negative first century, I would rap his knuckles! You will not try to do as well as Einstein! You will aspire to do BETTER than Einstein or you may as well not bother!"

Jeffreyssai shook his head. "Well, I've given you enough hints. It is time to test your skills. Now, I know that the other beisutsukai don't think much of my class projects..." Jeffreyssai paused significantly.

Brennan inwardly sighed. He'd heard this line many times before, in the Bardic Conspiracy, the Competitive Conspiracy: The other teachers think my assignments are too easy, you should be grateful, followed by some ridiculously difficult task -

"They say," Jeffreyssai said, "that my projects are too hard; insanely hard; that they pass from the realm of madness into the realm of Sparta; that Laplace himself would catch on fire; they accuse me of trying to tear apart my students' souls -"

Oh, crap.

"But there is a reason," Jeffreyssai said, "why many of my students have achieved great things; and by that I do not mean high rank in the Bayesian Conspiracy. I expected much of them, and they came to expect much of themselves. So..."

Jeffreyssai took a moment to look over his increasingly disturbed students, "Here is your assignment. Of quantum mechanics, and General Relativity, you have been told. This is the limit of Eld science, and hence, the limit of public knowledge. The five of you, working on your own, are to produce the correct theory of quantum gravity. Your time limit is one month."

"What?" said Brennan, Taji, Styrlyn, and Yin. Hiriwa gave them a puzzled look.

"Should you succeed," Jeffreyssai continued, "you will be promoted to beisutsukai of the second dan and sixth level. We will see if you have learned speed. Your clock starts - now."

And Jeffreyssai strode out of the room, slamming the door behind him.

"This is crazy!" Taji cried.

Hiriwa looked at Taji, bemused. "The solution is not known to us. How can you know it is so difficult?"

"Because we knew about this problem back in the Eld days! Eld scientists worked on this problem for a lot longer than one month."

Hiriwa shrugged. "They were still arguing about many-worlds too, weren't they?"

"Enough! There's no time!"

The other four students looked to Styrlyn, remembering that he was said to rank high in the Cooperative Conspiracy. There was a brief moment of weighing, of assessing, and then Styrlyn was their leader.

Styrlyn took a great breath. "We need a list of approaches. Write down all the angles you can think of. Independently - we need your individual components before we start combining. In five minutes, I'll ask each of you for your best idea first. No wasted thoughts! Go!"

Brennan grabbed a sheet and his tracer, set the tip to the surface, and then paused. He couldn't think of anything clever to say about unifying general relativity and quantum mechanics...

The other students were already writing.

Brennan tapped the tip, once, twice, thrice. General relativity and quantum mechanics...

Taji put his first sheet aside, grabbed another.

Finally, Brennan, for lack of anything clever to say, wrote down the obvious.

Minutes later, when Styrlyn called time, it was still all he had written.

"All right," Styrlyn said, "your best idea. Or the idea you most want the rest of us to take into account in our second components. Taji, go!"

Taji looked over his sheets. "Okay, I think we've got to assume that every avenue that Eld science was trying is a blind alley, or they would have found it. And if this is possible to do in one month, the answer must be, in some sense, elegant. So no multiple dimensions. If we start doing anything that looks like we should call it 'string theory', we'd better stop. Maybe begin by considering how failure to understand decoherence could have led Eld science astray in quantizing gravity."

"The opposite of folly is folly," Hiriwa said. "Let us pretend that Eld science never existed."

"No criticisms yet!" said Styrlyn. "Hiriwa, your suggestion?"

"Get rid of the infinities," said Hiriwa, "extirpate that which permits them. It should not be a matter of cleverness with integrals. A representation that allows infinity must be false-to-fact."

"Yin."

"We know from common sense," Yin said, "that if we stepped outside the universe, we would see time laid out all at once, reality like a crystal. But I once encountered a hint that physics is timeless in a deeper sense than that." Yin's eyes were distant, remembering. "Years ago, I found an abandoned city; it had been uninhabited for eras, I think. And behind a door whose locks were broken, carved into one wall: quote .ua sai .ei mi vimcu ty bu le mekso unquote."

Brennan translated: Eureka! Eliminate t from the equations. And written in Lojban, the sacred language of science, which meant the unknown writer had thought it to be true.

"The 'timeless physics' of which we've all heard rumors," Yin said, "may be timeless in a very literal sense."

"My own contribution," Styrlyn said. "The quantum physics we've learned is over joint positional configurations. It seems like we should be able to take that apart into a spatially local representation, in terms of invariant distant entanglements. Finding that representation might help us integrate with general relativity, whose curvature is local."

"A strangely individualist perspective," Taji murmured, "for one of the Cooperative Conspiracy."

Styrlyn shook his head. "You misunderstand us, then. The first lesson we learn is that groups are made of people... no, there is no time for politics. Brennan!"

Brennan shrugged. "Not much, I'm afraid, only the obvious. Inertial mass-energy was always observed to equal gravitational mass-energy, and Einstein showed that they were necessarily the same. So why is the 'energy' that is an eigenvalue of the quantum Hamiltonian, necessarily the same as the 'energy' quantity that appears in the equations of General Relativity? Why should spacetime curve at the same rate that the little arrows rotate?"

There was a brief pause.

Yin frowned. "That seems too obvious. Wouldn't Eld science have figured it out already?"

"Forget Eld science existed," Hiriwa said. "The question stands: we need the answer, whether it was known in ancient times or not. It cannot possibly be coincidence."

Taji's eyes were abstracted. "Perhaps it would be possible to show that an exception to the equality would violate some conservation law..."

"That is not where Brennan pointed," Hiriwa interrupted. "He did not ask for a proof that they must be set equal, given some appealing principle; he asked for a view in which the two are one and cannot be divided even conceptually, as was accomplished for inertial mass-energy and gravitational mass-energy. For we must assume that the beauty of the whole arises from the fundamental laws, and not the other way around. Fair-rephrasing?"

"Fair-rephrasing," Brennan replied.

Silence reigned for thirty-seven seconds, as the five pondered the five suggestions.

"I have an idea..."

Sequence: Quantum Physics

Einstein's Superpowers

Why Quantum?

Referenced by: The Rhythm of Disagreement • The Quantum Physics Sequence • Grasping Slippery Things • Against Modal Logics • Fighting a Rearguard Action Against the Truth • Mandatory Secret Identities

Original with comments: Class Project

A Premature Word on AI

Eliezer Yudkowsky, 31 May 2008 05:48PM

Followup to: A.I. Old-Timers, Do Scientists Already Know This Stuff?

In response to Robin Hanson's post on the disillusionment of old-time AI researchers such as Roger Schank, I thought I'd post a few premature words on AI, even though I'm not really ready to do so:

Anyway:

I never expected AI to be easy. I went into the AI field because I thought it was world-crackingly important, and I was willing to work on it if it took the rest of my whole life, even though it looked incredibly difficult.

I've noticed that folks who actively work on Artificial General Intelligence, seem to have started out thinking the problem was much easier than it first appeared to me.

In retrospect, if I had not thought that the AGI problem was worth a hundred and fifty thousand human lives per day - that's what I thought in the beginning - then I would not have challenged it; I would have run away and hid like a scared rabbit. Everything I now know about how to not panic in the face of difficult problems, I learned from tackling AGI, and later, the superproblem of Friendly AI, because running away wasn't an option.

Try telling one of these AGI folks about Friendly AI, and they reel back, surprised, and immediately say, "But that would be too difficult!" In short, they have the same run-away reflex as anyone else, but AGI has not activated it. (FAI does.)

Roger Schank is not necessarily in this class, please note. Most of the people currently wandering around in the AGI Dungeon are those too blind to see the warning signs, the skulls on spikes, the flaming pits. But e.g. John McCarthy is a warrior of a different sort; he ventured into the AI Dungeon before it was known to be difficult. I find that in terms of raw formidability, the warriors who first stumbled across the Dungeon, impress me rather more than most of the modern explorers - the first explorers were not self-selected for folly. But alas, their weapons tend to be extremely obsolete.

There are many ways to run away from difficult problems. Some of them are exceedingly subtle.

What makes a problem seem impossible? That no avenue of success is available to mind. What makes a problem seem scary? That you don't know what to do next.

Let's say that the problem of creating a general intelligence seems scary, because you have no idea how to do it.You could run away by working on chess-playing programs instead. Or you could run away by saying, "All past AI projects failed due to lack of computing power." Then you don't have to face the unpleasant prospect of staring at a blank piece of paper until drops of blood form on your forehead - the best description I've ever heard of the process of searching for core insight. You have avoided placing yourself into a condition where your daily work may consist of not knowing what to do next.

But "Computing power!" is a mysterious answer to a mysterious question. Even after you believe that all past AI projects failed "due to lack of computing power", it doesn't make intelligence any less mysterious. "What do you mean?" you say indignantly, "I have a perfectly good explanation for intelligence: it emerges from lots of computing power! Or knowledge! Or complexity!" And this is a subtle issue to which I must probably devote more posts. But if you contrast the rush of insight into details and specifics that follows from learning about, say, Pearlian causality, you may realize that "Computing power causes intelligence" does not constrain detailed anticipation of phenomena even in retrospect.

People are not systematically taught what to do when they're scared; everyone's got to work it out on their own. And so the vast majority stumble into simple traps like mysterious answers or affective death spirals. I too stumbled, but I managed to recover and get out alive; and realized what it was that I'd learned; and then I went back into the Dungeon, because I had something to protect.

I've recently discussed how scientists are not taught to handle chaos, so I'm emphasizing that aspect in this particular post, as opposed to a dozen other aspects... If you want to appreciate the inferential distances here, think of how odd all this would sound without the Einstein sequence. Then think of how odd the Einstein sequence would have sounded without the many-worlds sequence... There's plenty more where that came from.

What does progress in AGI/FAI look like, if not bigger and faster computers?

It looks like taking down the real barrier, the scary barrier, the one where you have to sweat blood: understanding things that seem mysterious, and not by declaring that they're "emergent" or "complex", either.

If you don't understand the family of Cox's Theorems and the Dutch Book argument, you can go round and round with "certainty factors" and "fuzzy logics" that seem sorta appealing, but that can never quite be made to work right. Once you understand the structure of probability - not just probability as an explicit tool, but as a forced implicit structure in cognitive engines - even if the structure is only approximate - then you begin to actually understand what you're doing; you are not just trying things that seem like good ideas. You have achieved core insight. You are not even limited to floating-point numbers between 0 and 1 to represent probability; you have seen through to structure, and can use log odds or smoke signals if you wish.

If you don't understand graphical models of conditional independence, you can go round and round inventing new "default logics" and "defeasible logics" that get more and more complicated as you try to incorporate an infinite number of special cases. If you know the graphical structure, and why the graphical model works, and the regularity of the environment that it exploits, and why it is efficient as well as correct, then you really understand the problem; you are not limited to explicit Bayesian networks, you just know that you have to exploit a certain kind of mathematical regularity in the environment.

Unfortunately, these two insights - Bayesian probability and Pearlian causality - are far from sufficient to solve general AI problems. If you try to do anything with these two theories that requires an additional key insight you do not yet possess, you will fail just like any other AGI project, and build something that grows more and more complicated and patchworky but never quite seems to work the way you hoped.

These two insights are examples of what "progress in AI" looks like.

Most people who say they intend to tackle AGI do not understand Bayes or Pearl. Most of the people in the AI Dungeon are there because they think they found the Sword of Truth in an old well, or, even worse, because they don't realize the problem is difficult. They are not polymaths; they are not making a convulsive desperate effort to solve the unsolvable. They are optimists who have their Great Idea that is the best idea ever even though they can't say exactly how it will produce intelligence, and they want to do the scientific thing and test their hypothesis. If they hadn't started out thinking they already had the Great Idea, they would have run away from the Dungeon; but this does not give them much of a motive to search for other master keys, even the ones already found.

The idea of looking for an "additional insight you don't already have" is something that the academic field of AI is just not set up to do. As a strategy, it does not result in a reliable success (defined as a reliable publication). As a strategy, it requires additional study and large expenditures of time. It ultimately amounts to "try to be Judea Pearl or Laplace" and that is not something that professors have been reliably taught to teach undergraduates; even though it is often what a field in a state of scientific chaos needs.

John McCarthy said quite well what Artificial Intelligence needs: 1.7 Einsteins, 2 Maxwells, 5 Faradays and .3 Manhattan Projects. From this I am forced to subtract the "Manhattan project", because security considerations of FAI prohibit using that many people; but I doubt it'll take more than another 1.5 Maxwells and 0.2 Faradays to make up for it.

But, as said, the field of AI is not set up to support this - it is set up to support explorations with reliable payoffs.

You would think that there would be genuinely formidable people going into the Dungeon of Generality, nonetheless, because they wanted to test their skills against true scientific chaos. Even if they hadn't yet realized that their little sister is down there. Well, that sounds very attractive in principle, but I guess it sounds a lot less attractive when you have to pay the rent. Or they're all off doing string theory, because AI is well-known to be impossible, not the sort of chaos that looks promising - why, it's genuinely scary! You might not succeed, if you went in there!

But I digress. This began as a response to Robin Hanson's post "A.I. Old-Timers", and Roger Shank's very different idea of what future AI progress will look like.

Okay, let's take a look at Roger Schank's argument:

I have not soured on AI. I still believe that we can create very intelligent machines. But I no longer believe that those machines will be like us... What AI can and should build are intelligent special purpose entities. (We can call them Specialized Intelligences or SI's.) Smart computers will indeed be created. But they will arrive in the form of SI's, ones that make lousy companions but know every shipping accident that ever happened and why (the shipping industry's SI) or as an expert on sales (a business world SI.)

I ask the fundamental question of rationality: Why do you believe what you believe?

Schank would seem to be talking as if he knows something about the course of future AI research - research that hasn't happened yet. What is it that he thinks he knows? How does he think he knows it?

As John McCarthy said: "Your statements amount to saying that if AI is possible, it should be easy. Why is that?"

There is a master strength behind all human arts: Human intelligence can, without additional adaptation, create the special-purpose systems of a skyscraper, a gun, a space shuttle, a nuclear weapon, a DNA synthesizer, a high-speed computer...

If none of what the human brain does is magic, the combined trick of it can be recreated in purer form.

If this can be done, someone will do it. The fact that shipping-inventory programs can be built as well, does not mean that it is sensible to talk about people only building shipping-inventory programs. If it is also possible to build something of human+ power. In a world where both events occur, the course of history is dominated by the latter.

So what is it that Roger Schank learned, as Bayesian evidence, which confirms some specific hypothesis over its alternatives - and what is the hypothesis, exactly? - that reveals to him the future course of AI research? Namely, that AI will not succeed in creating anything of general capability?

It would seem rather difficult to predict the future course of research you have not yet done. Wouldn't Schank have to know the best solution in order to know the minimum time the best solution would take?

Of course I don't think Schank is actually doing a Bayesian update here. I think Roger Schank gives the game away when he says:

When reporters interviewed me in the 70's and 80's about the possibilities for Artificial Intelligence I would always say that we would have machines that are as smart as we are within my lifetime. It seemed a safe answer since no one could ever tell me I was wrong.

There is careful futurism, where you try to consider all the biases you know, and separate your analysis into logical parts, and put confidence intervals around things, and use wider confidence intervals where you have less constraining knowledge, and all that other stuff rationalists do. Then there is sloppy futurism, where you just make something up that sounds neat. This sounds like sloppy futurism to me.

So, basically, Schank made a fantastic amazing futuristic prediction about machines "as smart as we are" "within my lifetime" - two phrases that themselves reveal some shaky assumptions.

Then Schank got all sad and disappointed because he wasn't making progress as fast as he hoped.

So Schank made a different futuristic prediction, about special-purpose AIs that will answer your questions about shipping disasters. It wasn't quite as shiny and futuristic, but it matched his new saddened mood, and it gave him something to say to reporters when they asked him where AI would be in 2050.

This is how the vast majority of futurism is done. So until I have reason to believe there is something more to Schank's analysis than this, I don't feel very guilty about disagreeing with him when I make "predictions" like:

If you don't know much about a problem, you should widen your confidence intervals in both directions. AI seems very hard because you don't know how to do it. But translating this into a confident prediction of a very long time interval would express your ignorance as if it were positive knowledge. So even though AI feels very hard to you, this is an expression of ignorance that should translate into a confidence interval wide in both directions: the less you know, the broader that confidence interval should be, in both directions.

Or:

You don't know what theoretical insights will be required for AI, or you would already have them. Theoretical breakthroughs can happen without advance warning (the warning is perceived in retrospect, of course, but not in advance); and they can be arbitrarily large. We know it is difficult to build a star from hydrogen atoms in the obvious way - because we understand how stars work, so we know that the work required is a huge amount of drudgery.

Or:

Looking at the anthropological trajectory of hominids seems to strongly contradict the assertion that exponentially increasing amounts of processing power or programming time are required for the production of intelligence in the vicinity of human; even when using an evolutionary algorithm that runs on blind mutations, random recombination, and selection with zero foresight.

But if I don't want this post to go on forever, I had better stop it here. See this paper, however.

Referenced by: The Rhythm of Disagreement • Detached Lever Fallacy • Failure By Affective Analogy • Nonperson Predicates • ...And Say No More Of It

Original with comments: A Premature Word on AI

The Rhythm of Disagreement

Eliezer Yudkowsky, 01 June 2008 08:18PM

Followup to: A Premature Word on AI, The Modesty Argument

Once, during the year I was working with Marcello, I passed by a math book he was reading, left open on the table. One formula caught my eye (why?); and I thought for a moment and said, "This... doesn't look like it can be right..."

Then we had to prove it couldn't be right.

Why prove it? It looked wrong; why take the time for proof?

Because it was in a math book. By presumption, when someone publishes a book, they run it past some editors and double-check their own work; then all the readers get a chance to check it, too. There might have been something we missed.

But in this case, there wasn't. It was a misprinted standard formula, off by one.

I once found an error in Judea Pearl's Causality - not just a misprint, but an actual error invalidating a conclusion in the text. I double and triple-checked, the best I was able, and then sent an email to Pearl describing what I thought the error was, and what I thought was the correct answer. Pearl confirmed the error, but he said my answer wasn't right either, for reasons I didn't understand and that I'd have to have gone back and done some rereading and analysis to follow. I had other stuff to do at the time, unfortunately, and couldn't expend the energy. And by the time Pearl posted an expanded explanation to the website, I'd forgotten the original details of the problem... Okay, so my improved answer was wrong.

Why take Pearl's word for it? He'd gotten the original problem wrong, and I'd caught him on it - why trust his second thought over mine?

Because he was frikkin' Judea Pearl. I mean, come on! I might dare to write Pearl with an error, when I could understand the error well enough that it would have seemed certain, if not for the disagreement. But it didn't seem likely that Pearl would concentrate his alerted awareness on the problem, warned of the mistake, and get it wrong twice. If I didn't understand Pearl's answer, that was my problem, not his. Unless I chose to expend however much work was required to understand it, I had to assume he was right this time. Not just as a matter of fairness, but of probability - that, in the real world, Pearl's answer really was right.

In IEEE Spectrum's sad little attempt at Singularity coverage, one bright spot is Paul Wallich's "Who's Who In The Singularity", which (a) actually mentions some of the real analysts like Nick Bostrom and myself and (b) correctly identifies me as an advocate of the "intelligence explosion", whereas e.g. Ray Kurzweil is designated as "technotopia - accelerating change". I.e., Paul Wallich actually did his homework instead of making everything up as he went along. Sad that it's just a little PDF chart.

Wallich's chart lists Daniel Dennett's position on the Singularity as:

Human-level AI may be inevitable, but don’t expect it anytime soon. "I don’t deny the possibility a priori; I just think it is vanishingly unlikely in the foreseeable future."

That surprised me. "Vanishingly unlikely"? Why would Dennett think that? He has no obvious reason to share any of the standard prejudices. I would be interested in knowing Dennett's reason for this opinion, and mildly disappointed if it turns out to be the usual, "We haven't succeeded in the last fifty years, therefore we definitely won't succeed in the next hundred years."

Also in IEEE Spectrum, Steven Pinker, author of The Blank Slate - a popular introduction to evolutionary psychology that includes topics like heuristics and biases - is quoted:

When machine consciousness will occur: "In one sense—information routing—they already have. In the other sense—first-person experience—we'll never know."

Whoa, said I to myself, Steven Pinker is a mysterian? "We'll never know"? How bizarre - I just lost some of the respect I had for him.

I disagree with Dennett about Singularity time horizons, and with Pinker about machine consciousness. Both of these are prestigious researchers whom I started out respecting about equally. So why am I curious to hear Dennett's reasons; but outright dismissive of Pinker?

I would probably say something like, "There are many potential reasons to disagree about AI time horizons, and no respectable authority to correct you if you mess up. But if you think consciousness is everlastingly mysterious, you have completely missed the lesson of history; and respectable minds will give you many good reasons to believe so. Non-reductionism says something much deeper about your outlook on reality than AI timeframe skepticism; someone like Pinker really ought to have known better."

(But all this presumes that Pinker is the one who is wrong, and not me...)

Robert Aumann, Nobel laureate and original inventor of the no-disagreement-among-Bayesians theorem, is a believing Orthodox Jew. (I know I keep saying this, but it deserves repeating, for the warning it carries.) By the time I discovered this strange proclivity of Aumann's, I had long ago analyzed the issues. Discovering that Aumann was Jewish, did not cause me to revisit the issues even momentarily. I did not consider for even a fraction of a second that this Nobel laureate and Bayesian might be right, and myself wrong. I did draw the lesson, "You can teach people Bayesian math, but even if they're genuinely very good with the math, applying it to real life and real beliefs is a whole different story."

Scott Aaronson calls me a bullet-swallower; I disagree. I am very choosy about which bullets I dodge, and which bullets I swallow. Any view of disagreement that implies I should not disagree with Robert Aumann must be wrong.

Then there's the whole recent analysis of Many-Worlds. I felt very guilty, writing about physics when I am not a physicist; but dammit, there are physicists out there talking complete nonsense about Occam's Razor, and they don't seem to feel guilty for using words like "falsifiable" without being able to do the math.

On the other hand, if, hypothetically, Scott Aaronson should say, "Eliezer, your question about why 'energy' in the Hamiltonian and 'energy' in General Relativity are the same quantity, is complete nonsense, it doesn't even have an answer, I can't explain why because you know too little," I would be like "Okay."

Nearly everyone I meet knows how to solve the problem of Friendly AI. I don't hesitate to dismiss nearly all of these solutions out of hand; standard wrong patterns I dissected long since.

Nick Bostrom, however, once asked whether it would make sense to build an Oracle AI, one that only answered questions, and ask it our questions about Friendly AI. I explained some of the theoretical reasons why this would be just as difficult as building a Friendly AI: The Oracle AI still needs an internal goal system to allocate computing resources efficiently, and it has to have a goal of answering questions and updating your mind, so it's not harmless unless it knows what side effects shouldn't happen. It also needs to implement or interpret a full meta-ethics before it can answer our questions about Friendly AI. So the Oracle AI is not necessarily any simpler, theoretically, than a Friendly AI.

Nick didn't seem fully convinced of this. I knew that Nick knew that I'd been thinking about the problem for years, so I knew he wasn't just disregarding me; his continued disagreement meant something. And I also remembered that Nick had spotted the problem of Friendly AI itself, at least two years before I had (though I did not realize this until later, when I was going back and reading some of Nick's older work). So I pondered Nick's idea further. Maybe, whatever the theoretical arguments, an AI that was supposed to only answer questions, and designed to the full standards of Friendly AI without skipping any of the work, could end up a pragmatically safer starting point. Every now and then I prod Nick's Oracle AI in my mind, to check the current status of the idea relative to any changes in my knowledge. I remember Nick has been right on previous occasions where I doubted his rightness; and if I am an expert, so is he.

I was present at a gathering with Sebastian Thrun (leader of the team that won the DARPA Grand Challenge '06 for motorized vehicles). Thrun introduced the two-envelopes problem and then asked: "Can you find an algorithm that, regardless of how the envelope amounts are distributed, always has a higher probability of picking the envelope with more money?"

I thought and said, "No."

"No deterministic algorithm can do it," said Thrun, "but if you use a randomized algorithm, it is possible."

Now I was really skeptical; you cannot extract work from noise.

Thrun gave the solution: Just pick any function from dollars onto probability that decreases monotonically and continuously from 1 probability at 0 dollars, to a probability of 0 at infinity. Then if you open the envelope and find that amount of money, roll a die and switch the envelope at that probability. However much money was in both envelopes originally, and whatever the distribution, you will always have a higher probability of switching the envelope with the lower amount of money.

I said, "That can't possibly work... you can't derive useful work from an arbitrary function and a random number... maybe it involves an improper prior..."

"No it doesn't," said Thrun; and it didn't.

So I went away and thought about it overnight and finally wrote an email in which I argued that the algorithm did make use of prior knowledge about the envelope distribution. (As the density of the differential of the monotonic function, in the vicinity of the actual envelope contents, goes to zero, the expected benefit of the algorithm over random chance, goes to zero.) Moreover, once you realized how you were using your prior knowledge, you could see a derandomized version of the algorithm which was superior, even though it didn't make the exact guarantee Thrun had made.

But Thrun's solution did do what he said it did.

(In a remarkable coincidence, not too much later, Steve Omohundro presented me with an even more startling paradox. "That can't work," I said. "Yes it can," said Steve, and it could. Later I perceived, after some thought, that the paradox was a more complex analogue of Thrun's algorithm. "Why, this is analogous to Thrun's algorithm," I said, and explained Thrun's algorithm. "That's not analogous," said Steve. "Yes it is," I said, and it was.)

Why disagree with Thrun in the first place? He was a prestigious AI researcher who had just won the DARPA Grand Challenge, crediting his Bayesian view of probability - a formidable warrior with modern arms and armor. It wasn't a transhumanist question; I had no special expertise.

Because I had worked out, as a very general principle, that you ought not to be able to extract cognitive work from randomness; and Thrun's algorithm seemed to be defying that principle.

Okay, but what does that have to do with the disagreement? Why presume that it was his algorithm that was at fault, and not my foolish belief that you couldn't extract cognitive work from randomness?

Well, in point of fact, neither of these was the problem. The fault was in my notion that there was a conflict between Thrun's algorithm doing what he said it did, and the no-work-from-randomness principle. So if I'd just assumed I was wrong, I would have been wrong.

Yet surely I could have done better, if I had simply presumed Thrun to be correct, and managed to break down the possibilities for error on my part into "The 'no work from randomness' principle is incorrect" and "My understanding of what Thrun meant is incorrect" and "My understanding of the algorithm is incomplete; there is no conflict between it and 'no work from randomness'."

Well, yes, on that occasion, this would have given me a better probability distribution, if I had assigned probability 0 to a possibility that turned out, in retrospect, to be wrong.

But probability 0 is a strawman; could I have done better by assigning a smaller probability that Thrun had said anything mathematically wrong?

Yes. And if I meet Thrun again, or anyone who seems similar to Thrun, that's just what I'll do.

Just as I'll assign a slightly higher probability that I might be right, the next time I find what looks like an error in a famous math book. In fact, one of the reasons why I lingered on what looked like a problem in Pearl's Causality, was that I'd previously found an acknowledged typo in Probability Theory: The Logic of Science.

My rhythm of disagreement is not a fixed rule, it seems. A fixed rule would be beyond updating by experience.

I tried to explain why I disagreed with Roger Schank, and Robin said, "All else equal a younger person is more likely to be right in a disagreement?"

But all else wasn't equal. That was the point. Roger Schank is a partisan of what one might best describe as "old school" AI, i.e., suggestively named LISP tokens.

Is it good for the young to disagree with the old? Sometimes. Not all the time. Just some of the time. When? Ah, that's the question! Even in general, if you are disagreeing about the future course of AI with a famous old AI researcher, and the famous old AI researcher is of the school of suggestively named LISP tokens, and you yourself are 21 years old and have taken one undergraduate course taught with "Artificial Intelligence: A Modern Approach" that you thought was great... then I would tell you to go for it. Probably both of you are wrong. But if you forced me to bet money on one or the other, without hearing the specific argument, I'd go with the young upstart. Then again, the young upstart is not me, so how do they know that rule?

It's hard enough to say what the rhythm of disagreement should be in my own case. I would hesitate to offer general advice to others, save the obvious: Be less ready to disagree with a supermajority than a mere majority; be less ready to disagree outside than inside your expertise; always pay close attention to the object-level arguments; never let the debate become about tribal status.

Referenced by: Principles of Disagreement

Original with comments: The Rhythm of Disagreement

Principles of Disagreement

Eliezer Yudkowsky, 02 June 2008 07:04AM

Followup to: The Rhythm of Disagreement

At the age of 15, a year before I knew what a "Singularity" was, I had learned about evolutionary psychology. Even from that beginning, it was apparent to me that people talked about "disagreement" as a matter of tribal status, processing it with the part of their brain that assessed people's standing in the tribe. The peculiar indignation of "How dare you disagree with Einstein?" has its origins here: Even if the disagreer is wrong, we wouldn't apply the same emotions to an ordinary math error like "How dare you write a formula that makes e equal to 1.718?"

At the age of 15, being a Traditional Rationalist, and never having heard of Aumann or Bayes, I thought the obvious answer was, "Entirely disregard people's authority and pay attention to the arguments. Only arguments count."

Ha ha! How naive.

I can't say that this principle never served my younger self wrong.

I can't even say that the principle gets you as close as possible to the truth.

I doubt I ever really clung to that principle in practice. In real life, I judged my authorities with care then, just as I do now...

But my efforts to follow that principle, made me stronger. They focused my attention upon arguments; believing in authority does not make you stronger. The principle gave me freedom to find a better way, which I eventually did, though I wandered at first.

Yet both of these benefits were pragmatic and long-term, not immediate and epistemic. And you cannot say, "I will disagree today, even though I'm probably wrong, because it will help me find the truth later." Then you are trying to doublethink. If you know today that you are probably wrong, you must abandon the belief today. Period. No cleverness. Always use your truth-finding skills at their full immediate strength, or you have abandoned something more important than any other benefit you will be offered; you have abandoned the truth.

So today, I sometimes accept things on authority, because my best guess is that they are really truly true in real life, and no other criterion gets a vote.

But always in the back of my mind is that childhood principle, directing my attention to the arguments as well, reminding me that you gain no strength from authority; that you may not even know anything, just be repeating it back.

Earlier I described how I disagreed with a math book and looked for proof, disagreed humbly with Judea Pearl and was proven (half) right, disagreed immodestly with Sebastian Thrun and was proven wrong, had a couple of quick exchanges with Steve Omohundro in which modesty-reasoning would just have slowed us down, respectfully disagreed with Daniel Dennett and disrespectfully disagreed with Steven Pinker, disagreed with Robert Aumann without a second thought, disagreed with Nick Bostrom with second thoughts...

What kind of rule am I using, that covers all these cases?

Er... "try to get the actual issue really right"? I mean, there are other rules but that's the important one. It's why I disagree with Aumann about Orthodox Judaism, and blindly accept Judea Pearl's word about the revised version of his analysis. Any argument that says I should take Aumann seriously is wasting my time; any argument that says I should disagree with Pearl is wasting my truth.

There are all sorts of general reasons not to argue with physicists about physics, but the rules are all there to help you get the issue right, so in the case of Many-Worlds you have to ignore them.

Yes, I know that's not helpful as a general principle. But dammit, wavefunctions don't collapse! It's a massively stupid idea that sticks around due to sheer historical contingency! I'm more confident of that than any principle I would dare to generalize about disagreement.

Notions of "disagreement" are psychology-dependent pragmatic philosophy. Physics and Occam's razor are much simpler. Object-level stuff is often much clearer than meta-level stuff, even though this itself is a meta-level principle.

In theory, you have to make a prior decision whether to trust your own assessment of how obvious it is that wavefunctions don't collapse, before you can assess whether wavefunctions don't collapse. In practice, it's much more obvious that wavefunctions don't collapse, than that I should trust my disagreement. Much more obvious. So I just go with that.

I trust any given level of meta as far as I can throw it, but no further.

There's a rhythm to disagreement. And oversimplified rules about when to disagree, can distract from that rhythm. Even "Follow arguments, not people" can distract from the rhythm, because no one, including my past self, really uses that rule in practice.

The way it works in real life is that I just do the standard first-order disagreement analysis: Okay, in real life, how likely is it that this person knows stuff that I don't?

Not, Okay, how much of the stuff that I know that they don't, have they already taken into account in a revised estimate, given that they know I disagree with them, and have formed guesses about what I might know that they don't, based on their assessment of my and their relative rationality...

Why don't I try the higher-order analyses? Because I've never seen a case where, even in retrospect, it seems like I could have gotten real-life mileage out of it. Too complicated, too much of a tendency to collapse to tribal status, too distracting from the object-level arguments.

I have previously observed that those who genuinely reach upward as rationalists, have usually been broken of their core trust in the sanity of the people around them. In this world, we have to figure out who to trust, and who we have reasons to trust, and who might be right even when we believe they're wrong. But I'm kinda skeptical that we can - in this world of mostly crazy people and a few slightly-more-sane people who've spent their whole lives surrounded by crazy people who claim they're saner than average - get real-world mileage out of complicated reasoning that involves sane people assessing each other's meta-sanity. We've been broken of that trust, you see.

Does Robin Hanson really trust, deep down, that I trust him enough, that I would not dare to disagree with him, unless he were really wrong? I can't trust that he does... so I don't trust him so much... so he shouldn't trust that I wouldn't dare disagree...

It would be an interesting experiment: but I cannot literally commit to walking into a room with Robin Hanson and not walking out until we have the same opinion about the Singularity. So that if I give him all my reasons and hear all his reasons, and Hanson tells me, "I still think you're wrong," I must then agree (or disagree in a net direction Robin can't predict). I trust Robin but I don't trust him THAT MUCH. Even if I tried to promise, I couldn't make myself believe it was really true - and that tells me I can't make the promise.

When I think about who I would be willing to try this with, the name that comes to mind is Michael Vassar - which surprised me, and I asked my mind why. The answer that came back was, "Because Michael Vassar knows viscerally what's at stake if he makes you update the wrong way; he wouldn't use the power lightly." I'm not going anywhere in particular with this; but it points in an interesting direction - that a primary reason I don't always update when people disagree with me, is that I don't think they're taking that disagreement with the extraordinary gravity that would be required, on both sides, for two people to trust each other in an Aumann cage match.

Yesterday, Robin asked me why I disagree with Roger Schank about whether AI will be general in the foreseeable future.

Well, first, be it said that I am no hypocrite; I have been explicitly defending immodesty against modesty since long before this blog began.

Roger Schank is a famous old AI researcher who I learned about as the pioneer of yet another false idol, "scripts". He used suggestively named LISP tokens, and I'd never heard it said of him that he had seen the light of Bayes.

So I noted that the warriors of old are often more formidable intellectually than those who venture into the Dungeon of General AI today, but their arms and armor are obsolete. And I pointed out that Schank's prediction with its stated reasons seemed more like an emotional reaction to discouragement, than a painstakingly crafted general model of the future of AI research that had happened to yield a firm prediction in this case.

Ah, said Robin, so it is good for the young to disagree with the old.

No, but if the old guy is Roger Schank, and the young guy is me, and we are disagreeing about Artificial General Intelligence, then sure.

If the old guy is, I don't know, Murray Gell-Mann, and we're disagreeing about, like, particle masses or something, I'd have to ask what I was even doing in that conversation.

If the old fogey is Murray Gell-Mann and the young upstart is Scott Aaronson, I'd probably stare at them helplessly like a deer caught in the headlights. I've listed out the pros and cons here, and they balance as far as I can tell:

- Murray Gell-Mann won a Nobel Prize back in the eighteenth century for work he did when he was four hundred years younger, or something like that.

- Scott Aaronson has more recent training.

- ...but physics may not have changed all that much since Gell-Mann's reign of applicability, sad to say.

- Aaronson still has most of his neurons left.

- I know Aaronson is smart, but Gell-Mann doesn't have any idea who Aaronson is. Aaronson knows Gell-Mann is a Nobel Laureate and wouldn't disagree lightly.

- Gell-Mann is a strong proponent of many-worlds and Aaronson is not, which is one of the acid tests of a physicist's ability to choose correctly amid controversy.

It is traditional - not Bayesian, not even remotely realistic, but traditional - that when some uppity young scientist is pushing their chosen field as far they possibly can, going past the frontier, they have a right to eat any old scientists they come across, for nutrition.

I think there's more than a grain of truth in that ideal. It's not completely true. It's certainly not upheld in practice. But it's not wrong, either.

It's not that the young have a generic right to disagree with the old, but yes, when the young are pushing the frontiers they often end up leaving the old behind. Everyone knows that and what's more, I think it's true.

If someday I get eaten, great.

I still agree with my fifteen-year-old self about some things: The tribal-status part of our minds, that asks, "How dare you disagree?", is just a hindrance. The real issues of rational disagreement have nothing to do with that part of us; it exists for other reasons and works by other rhythms. "How dare you disagree with Roger Schank?" ends up as a no-win question if you try to approach it on the meta-level and think in terms of generic trustworthiness: it forces you to argue that you yourself are generically above Schank and of higher tribal status; or alternatively, accept conclusions that do not seem, er, carefully reasoned. In such a case there is a great deal to be said for simply focusing on the object-level arguments.

But if there are no simple rules that forbid disagreement, can't people always make up whatever excuse for disagreement they like, so they can cling to precious beliefs?

Look... it's never hard to shoot off your own foot, in this art of rationality. And the more art you learn of rationality, the more potential excuses you have. If you insist on disagreeing with Gell-Mann about physics, BLAM it goes. There is no set of rules you can follow to be safe. You will always have the opportunity to shoot your own foot off.

I want to push my era further than the previous ones: create an advanced art of rationality, to advise people who are trying to reach as high as they can in real life. They will sometimes have to disagree with others. If they are pushing the frontiers of their science they may have to disagree with their elders. They will have to develop the skill - learning from practice - of when to disagree and when not to. "Don't" is the wrong answer.

If others take that as a welcome excuse to shoot their own feet off, that doesn't change what's really the truly true truth.

I once gave a talk on rationality at Peter Thiel's Clarium Capital. I did not want anything bad to happen to Clarium Capital. So I ended my talk by saying, "And above all, if any of these reasonable-sounding principles turn out not to work, don't use them."

In retrospect, thinking back, I could have given the different caution: "And be careful to follow these principles consistently, instead of making special exceptions when it seems tempting." But it would not be a good thing for the Singularity Institute, if anything bad happened to Clarium Capital.

That's as close as I've ever come to betting on my high-minded advice about rationality in a prediction market - putting my skin in a game with near-term financial consequences. I considered just staying home - Clarium was trading successfully; did I want to disturb their rhythm with Centipede's Dilemmas? But because past success is no guarantee of future success in finance, I went, and offered what help I could give, emphasizing above all the problem of motivated skepticism - when I had skin in the game. Yet at the end I said: "Don't trust principles until you see them working," not "Be wary of the temptation to make exceptions."

I conclude with one last tale of disagreement:

Nick Bostrom and I once took a taxi and split the fare. When we counted the money we'd assembled to pay the driver, we found an extra twenty there.

"I'm pretty sure this twenty isn't mine," said Nick.

"I'd have been sure that it wasn't mine either," I said.

"You just take it," said Nick.

"No, you just take it," I said.

We looked at each other, and we knew what we had to do.

"To the best of your ability to say at this point, what would have been your initial probability that the bill was yours?" I said.

"Fifteen percent," said Nick.

"I would have said twenty percent," I said.

So we split it $8.57 / $11.43, and went happily on our way, guilt-free.

I think that's the only time I've ever seen an Aumann-inspired algorithm used in real-world practice.

Original with comments: Principles of Disagreement

Timeless Identity

Eliezer Yudkowsky, 03 June 2008 08:16AM

This post is part of the Quantum Physics Sequence.
Followup to: No Individual Particles, Identity Isn't In Specific Atoms, Timeless Physics, Timeless Causality

People have asked me, "What practical good does it do to discuss quantum physics or consciousness or zombies or personal identity? I mean, what's the application for me in real life?"

Before the end of today's post, we shall see a real-world application with practical consequences, for you, yes, you in today's world. It is built upon many prerequisites and deep foundations; you will not be able to tell others what you have seen, though you may (or may not) want desperately to tell them. (Short of having them read the last several months of OB.)

In No Individual Particles we saw that the intuitive conception of reality as little billiard balls bopping around, is entirely and absolutely wrong; the basic ontological reality, to the best of anyone's present knowledge, is a joint configuration space. These configurations have mathematical identities like "A particle here, a particle there", rather than "particle 1 here, particle 2 there" and the difference is experimentally testable. What might appear to be a little billiard ball, like an electron caught in a trap, is actually a multiplicative factor in a wavefunction that happens to approximately factor. The factorization of 18 includes two factors of 3, not one factor of 3, but this doesn't mean the two 3s have separate individual identities - quantum mechanics is sort of like that. (If that didn't make any sense to you, sorry; you need to have followed the series on quantum physics.)

In Identity Isn't In Specific Atoms, we took this counterintuitive truth of physical ontology, and proceeded to kick hell out of an intuitive concept of personal identity that depends on being made of the "same atoms" - the intuition that you are the same person, if you are made out of the same pieces. But because the brain doesn't repeat its exact state (let alone the whole universe), the joint configuration space which underlies you, is nonoverlapping from one fraction of a second to the next. Or even from one Planck interval to the next. I.e., "you" of now and "you" of one second later do not have in common any ontologically basic elements with a shared persistent identity.

Just from standard quantum mechanics, we can see immediately that some of the standard thought-experiments used to pump intuitions in philosophical discussions of identity, are physical nonsense. For example, there is a thought experiment that runs like this:

"The Scanner here on Earth will destroy my brain and body, while recording the exact states of all my cells. It will then transmit this information by radio. Travelling at the speed of light, the message will take three minutes to reach the Replicator on Mars. This will then create, out of new matter, a brain and body exactly like mine. It will be in this body that I shall wake up."

This is Derek Parfit in the excellent Reasons and Persons, p. 199 - note that Parfit is describing thought experiments, not necessarily endorsing them.

There is an argument which Parfit describes (but does not himself endorse), and which I have seen many people spontaneously invent, which says (not a quote):

Ah, but suppose an improved Scanner were invented, which scanned you non-destructively, but still transmitted the same information to Mars . Now, clearly, in this case, you, the original have simply stayed on Earth, and the person on Mars is only a copy. Therefore this teleporter is actually murder and birth, not travel at all - it destroys the original, and constructs a copy!

Well, but who says that if we build an exact copy of you, one version is the privileged original and the other is just a copy? Are you under the impression that one of these bodies is constructed out of the original atoms - that it has some kind of physical continuity the other does not possess? But there is no such thing as a particular atom, so the original-ness or new-ness of the person can't depend on the original-ness or new-ness of the atoms.

(If you are now saying, "No, you can't distinguish two electrons yet, but that doesn't mean they're the same entity -" then you have not been following the series on quantum mechanics, or you need to reread it. Physics does not work the way you think it does. There are no little billiard balls bouncing around down there.)

If you further realize that, as a matter of fact, you are splitting all the time due to ordinary decoherence, then you are much more likely to look at this thought experiment and say: "There is no copy; there are two originals."

Intuitively, in your imagination, it might seem that one billiard ball stays in the same place on Earth, and another billiard ball has popped into place on Mars; so one is the "original", and the other is the "copy". But at a fundamental level, things are not made out of billiard balls.

A sentient brain constructed to atomic precision, and copied with atomic precision, could undergo a quantum evolution along with its "copy", such that, afterward, there would exist no fact of the matter as to which of the two brains was the "original". In some Feynman diagrams they would exchange places, in some Feynman diagrams not. The two entire brains would be, in aggregate, identical particles with no individual identities.

Parfit, having discussed the teleportation thought experiment, counters the intuitions of physical continuity with a different set of thought experiments:

"Consider another range of possible cases: the Physical Spectrum. These cases involve all of the different possible degrees of physical continuity...

"In a case close to the near end, scientists would replace 1% of the cells in my brain and body with exact duplicates. In the case in the middle of the spectrum, they would replace 50%. In a case near the far end, they would replace 99%, leaving only 1% of my original brain and body. At the far end, the 'replacement' would involve the complete destruction of my brain and body, and the creation out of new organic matter of a Replica of me."

(Reasons and Persons, p. 234.)

Parfit uses this to argue against the intuition of physical continuity pumped by the first experiment: if your identity depends on physical continuity, where is the exact threshold at which you cease to be "you"?

By the way, although I'm criticizing Parfit's reasoning here, I really liked Parfit's discussion of personal identity. It really surprised me. I was expecting a rehash of the same arguments I've seen on transhumanist mailing lists over the last decade or more. Parfit gets much further than I've seen the mailing lists get. This is a sad verdict for the mailing lists. And as for Reasons and Persons, it well deserves its fame.

But although Parfit executed his arguments competently and with great philosophical skill, those two particular arguments (Parfit has lots more!) are doomed by physics.

There just is no such thing as "new organic matter" that has a persistent identity apart from "old organic matter". No fact of the matter exists, as to which electron is which, in your body on Earth or your body on Mars. No fact of the matter exists, as to how many electrons in your body have been "replaced" or "left in the same place". So both thought experiments are physical nonsense.

Parfit seems to be enunciating his own opinion here (not Devil's advocating) when he says:

"There are two kinds of sameness, or identity. I and my Replica are qualitatively identical, or exactly alike. But we may not be numerically identical, one and the same person. Similarly, two white billiard balls are not numerically but may be qualitatively identical. If I paint one of these balls red, it will cease to be qualitatively identical with itself as it was. But the red ball that I later see and the white ball that I painted red are numerically identical. They are one and the same ball." (p. 201.)

In the human imagination, the way we have evolved to imagine things, we can imagine two qualitatively identical billiard balls that have a further fact about them - their persistent identity - that makes them distinct.

But it seems to be a basic lesson of physics that "numerical identity" just does not exist. Where "qualitative identity" exists, you can set up quantum evolutions that refute the illusion of individuality - Feynman diagrams that sum over different permutations of the identicals.

We should always have been suspicious of "numerical identity", since it was not experimentally detectable; but physics swoops in and drop-kicks the whole argument out the window.

Parfit p. 241:

"Reductionists admit that there is a difference between numerical identity and exact similarity. In some cases, there would be a real difference between some person's being me, and his being someone else who is merely exactly like me."

This reductionist admits no such thing.

Parfit even describes a wise-seeming reductionist refusal to answer questions as to when one person becomes another, when you are "replacing" the atoms inside them. P. 235:

(The reductionist says:) "The resulting person will be psychologically continuous with me as I am now. This is all there is to know. I do not know whether the resulting person will be me, or will be someone else who is merely exactly like me. But this is not, here, a real question, which must have an answer. It does not describe two different possibilities, one of which must be true. It is here an empty question. There is not a real difference here between the resulting person's being me, and his being someone else. This is why, even though I do not know whether I am about to die, I know everything."

Almost but not quite reductionist enough! When you master quantum mechanics, you see that, in the thought experiment where your atoms are being "replaced" in various quantities by "different" atoms, nothing whatsoever is actually happening - the thought experiment itself is physically empty.

So this reductionist, at least, triumphantly says - not, "It is an empty question; I know everything that there is to know, even though I don't know if I will live or die" - but simply, "I will live; nothing happened."

This whole episode is one of the main reasons why I hope that when I really understand matters such as these, and they have ceased to be mysteries unto me, that I will be able to give definite answers to questions that seem like they ought to have definite answers.

And it is a reason why I am suspicious, of philosophies that too early - before the dispelling of mystery - say, "There is no answer to the question." Sometimes there is no answer, but then the absence of the answer comes with a shock of understanding, a click like thunder, that makes the question vanish in a puff of smoke. As opposed to a dull empty sort of feeling, as of being told to shut up and stop asking questions.

And another lesson: Though the thought experiment of having atoms "replaced" seems easy to imagine in the abstract, anyone knowing a fully detailed physical visualization would have immediately seen that the thought experiment was physical nonsense. Let zombie theorists take note!

Additional physics can shift our view of identity even further:

In Timeless Physics, we looked at a speculative, but even more beautiful view of quantum mechanics: We don't need to suppose the amplitude distribution over the configuration space is changing, since the universe never repeats itself. We never see any particular joint configuration (of the whole universe) change amplitude from one time to another; from one time to another, the universe will have expanded. There is just a timeless amplitude distribution (aka wavefunction) over a configuration space that includes compressed configurations of the universe (early times) and expanded configurations of the universe (later times).

Then we will need to discover people and their identities embodied within a timeless set of relations between configurations that never repeat themselves, and never change from one time to another.

As we saw in Timeless Beauty, timeless physics is beautiful because it would make everything that exists either perfectly global - like the uniform, exceptionless laws of physics that apply everywhere and everywhen - or perfectly local - like points in the configuration space that only affect or are affected by their immediate local neighborhood. Everything that exists fundamentally, would be qualitatively unique: there would never be two fundamental entities that have the same properties but are not the same entity.

(Note: The you on Earth, and the you on Mars, are not ontologically basic. You are factors of a joint amplitude distribution that is ontologically basic. Suppose the integer 18 exists: the factorization of 18 will include two factors of 3, not one factor of 3. This does not mean that inside the Platonic integer 18 there are two little 3s hanging around with persistent identities, living in different houses.)

We also saw in Timeless Causality that the end of time is not necessarily the end of cause and effect; causality can be defined (and detected statistically!) without mentioning "time". This is important because it preserves arguments about personal identity that rely on causal continuity rather than "physical continuity".

Previously I drew this diagram of you in a timeless, branching universe:

To understand many-worlds: The gold head only remembers the green heads, creating the illusion of a unique line through time, and the intuitive question, "Where does the line go next?" But it goes to both possible futures, and both possible futures will look back and see a single line through time. In many-worlds, there is no fact of the matter as to which future you personally will end up in. There is no copy; there are two originals.

To understand timeless physics: The heads are not popping in and out of existence as some Global Now sweeps forward. They are all just there, each thinking that now is a different time.

In Timeless Causality I drew this diagram:

This was part of an illustration of how we could statistically distinguish left-flowing causality from right-flowing causality - an argument that cause and effect could be defined relationally, even the absence of a changing global time. And I said that, because we could keep cause and effect as the glue that binds configurations together, we could go on trying to identify experiences with computations embodied in flows of amplitude, rather than having to identify experiences with individual configurations.

But both diagrams have a common flaw: they show discrete nodes, connected by discrete arrows. In reality, physics is continuous.

So if you want to know "Where is the computation? Where is the experience?" my best guess would be to point to something like a directional braid:

This is not a braid of moving particles. This is a braid of interactions within close neighborhoods of timeless configuration space.

Every point intersected by the red line is unique as a mathematical entity; the points are not moving from one time to another. However, the amplitude at different points is related by physical laws; and there is a direction of causality to the relations.

You could say that the amplitude is flowing, in a river that never changes, but has a direction.

Embodied in this timeless flow are computations; within the computations, experiences. The experiences' computations' configurations might even overlap each other:

In the causal relations covered by the rectangle 1, there would be one moment of Now; in the causal relations covered by the rectangle 2, another moment of Now. There is a causal direction between them: 1 is the cause of 2, not the other way around. The rectangles overlap - though I really am not sure if I should be drawing them with overlap or not - because the computations are embodied in some of the same configurations. Or if not, there is still causal continuity because the end state of one computation is the start state of another.

But on an ontologically fundamental level, nothing with a persistent identity moves through time.

Even the braid itself is not ontologically fundamental; a human brain is a factor of a larger wavefunction that happens to factorize.

Then what is preserved from one time to another? On an ontologically basic level, absolutely nothing.

But you will recall that I earlier talked about any perturbation which does not disturb your internal narrative, almost certainly not being able to disturb whatever is the true cause of your saying "I think therefore I am" - this is why you can't leave a person physically unaltered, and subtract their consciousness. When you look at a person on the level of organization of neurons firing, anything which does not disturb, or only infinitesimally disturbs, the pattern of neurons firing - such as flipping a switch from across the room - ought not to disturb your consciousness, or your personal identity.

If you were to describe the brain on the level of neurons and synapses, then this description of the factor of the wavefunction that is your brain, would have a very great deal in common, across different cross-sections of the braid. The pattern of synapses would be "almost the same" - that is, the description would come out almost the same - even though, on an ontologically basic level, nothing that exists fundamentally is held in common between them. The internal narrative goes on, and you can see it within the vastly higher-level view of the firing patterns in the connection of synapses. The computational pattern computes, "I think therefore I am". The narrative says, today and tomorrow, "I am Eliezer Yudkowsky, I am a rationalist, and I have something to protect." Even though, in the river that never flows, not a single drop of water is shared between one time and another.

If there's any basis whatsoever to this notion of "continuity of consciousness" - I haven't quite given up on it yet, because I don't have anything better to cling to - then I would guess that this is how it works.

Oh... and I promised you a real-world application, didn't I?

Well, here it is:

Many throughout time, tempted by the promise of immortality, have consumed strange and often fatal elixirs; they have tried to bargain with devils that failed to appear; and done many other silly things.

But like all superpowers, long-range life extension can only be acquired by seeing, with a shock, that some way of getting it is perfectly normal.

If you can see the moments of now braided into time, the causal dependencies of future states on past states, the high-level pattern of synapses and the internal narrative as a computation within it - if you can viscerally dispel the classical hallucination of a little billiard ball that is you, and see your nows strung out in the river that never flows - then you can see that signing up for cryonics, being vitrified in liquid nitrogen when you die, and having your brain nanotechnologically reconstructed fifty years later, is actually less of a change than going to sleep, dreaming, and forgetting your dreams when you wake up.

You should be able to see that, now, if you've followed through this whole series. You should be able to get it on a gut level - that being vitrified in liquid nitrogen for fifty years (around 3e52 Planck intervals) is not very different from waiting an average of 2e26 Planck intervals between neurons firing, on the generous assumption that there are a hundred trillion synapses firing a thousand times per second. You should be able to see that there is nothing preserved from one night's sleep to the morning's waking, which cryonic suspension does not preserve also. Assuming the vitrification technology is good enough for a sufficiently powerful Bayesian superintelligence to look at your frozen brain, and figure out "who you were" to the same resolution that your morning's waking self resembles the person who went to sleep that night.

Do you know what it takes to securely erase a computer's hard drive? Writing it over with all zeroes isn't enough. Writing it over with all zeroes, then all ones, then a random pattern, isn't enough. Someone with the right tools can still examine the final state of a section of magnetic memory, and distinguish the state, "This was a 1 written over by a 1, then a 0, then a 1" from "This was a 0 written over by a 1, then a 0, then a 1". The best way to securely erase a computer's hard drive is to destroy it with thermite.

I really don't think that carefully vitrifying a brain to prevent ice crystal formation and then freezing it in liquid nitrogen is going to be a secure erase procedure, if you can examine atomic-level differences in the synapses.

Someone hears about cryonics and thinks for 10 seconds and says, "But if you're frozen and then revived, are you really the same person?"

And if they happened to know all about quantum physics and could apply the abstract knowledge to real life, and they had followed the whole debate about zombies and resolved it against epiphenomenalism in general, then they would be able to visualize the braids in the river that never flows, and say, "Yes."

But this knowledge is not common.

So they die.

There are numerous other reasons that people seize on, when they search for a rationalization for a negative initial flinch against cryonics. And numerous other knowledges that would be required to answer those objections. "But wouldn't it be boring to live such a long time?" (Can be answered if you know hedonic psychology, and have developed a theory of fun, and can visualize accessible fun spaces that increase in volume with increasing intelligence.) "Why would future civilizations bother to revive me?" (Requires understanding either economic growth diminishing the cost, or knowledge of history and how societies have become kinder over time, or knowing about Friendly AI.) "Isn't it wrong to live so long?" (Requires knowing about the "sour grapes" bias. See also transhumanism as simplified humanism and the meaning that immortality gives to life.) Then there's the meta-knowledge of how to question all these deeply wise cached thoughts that pop into your head about the futility of life; and the ability to do things that might make people look at you weird, and so on...

Some of these are series of posts I haven't done yet. But if you anticipate updating your probabilities when you read those future posts, then you should update them now. Or, if you prefer, trust me:

If you would rather live happily ever after, than die, and you are willing to spend between $300 and $2000 per year(*) to express this preference, then sign up for cryonics.

If you've been cryocrastinating, putting off signing up for cryonics "until later", don't think that you've "gotten away with it so far". Many worlds, remember? There are branched versions of you that are dying of cancer, and not signed up for cryonics, and it's too late for them to get life insurance.

See, knowing about many worlds can help you visualize probabilities as frequencies, because they usually are.

It might encourage you to get around to getting health insurance, too, or wearing a helmet on your motorcycle, or whatever: don't think you've gotten away with it so far.

And if you're planning to play the lottery, don't think you might win this time. A vanishingly small fraction of you wins, every time. So either learn to discount small fractions of the future by shutting up and multiplying, or spend all your money on lottery tickets - your call.

It is a very important lesson in rationality, that at any time, the Environment may suddenly ask you almost any question, which requires you to draw on 7 different fields of knowledge. If you missed studying a single one of them, you may suffer arbitrarily large penalties up to and including capital punishment. You can die for an answer you gave in 10 seconds, without realizing that a field of knowledge existed of which you were ignorant.

This is why there is a virtue of scholarship.

150,000 people die every day. Some of those deaths are truly unavoidable, but most are the result of inadequate knowledge of cognitive biases, advanced futurism, and quantum mechanics.(**)

If you disagree with my premises or my conclusion, take a moment to consider nonetheless, that the very existence of an argument about life-or-death stakes, whatever position you take in that argument, constitutes a sufficient lesson on the sudden relevance of scholarship.

(*) The way cryonics works is that you get a life insurance policy, and the policy pays for your cryonic suspension. The Cryonics Institute is the cheapest provider, Alcor is the high-class one. Rudi Hoffman set up my own insurance policy, with CI. I have no affiliate agreements with any of these entities, nor, to my knowledge, do they have affiliate agreements with anyone. They're trying to look respectable, and so they rely on altruism and word-of-mouth to grow, instead of paid salespeople. So there's a vastly smaller worldwide market for immortality than lung-cancer-in-a-stick. Welcome to your Earth; it's going to stay this way until you fix it.

(**) Most deaths? Yes: If cryonics were widely seen in the same terms as any other medical procedure, economies of scale would considerably diminish the cost; it would be applied routinely in hospitals; and foreign aid would enable it to be applied even in poor countries. So children in Africa are dying because citizens and politicians and philanthropists in the First World don't have a gut-level understanding of quantum mechanics.

Added: For some of the questions that are being asked, see Alcor's FAQ for scientists and Ben Best's Cryonics FAQ.

Sequence: Quantum Physics

Timeless Causality

Thou Art Physics

Referenced by: Why Quantum? • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity • Heading Toward Morality • The Fear of Common Knowledge • Beyond the Reach of God • The Anthropic Trilemma

Original with comments: Timeless Identity

Why Quantum?

Eliezer Yudkowsky, 04 June 2008 05:34AM

This post is part of the Quantum Physics Sequence.
Followup to: Quantum Explanations

"Why are you doing these posts on quantum physics?" the one asked me.

"Quite a number of reasons," I said.

"For one thing," I said, "the many-worlds issue is just about the only case I know of where you can bring the principles of Science and Bayesianism into direct conflict." It's important to have different mental buckets for "science" and "rationality", as they are different concepts. Bringing the two principles into direct conflict is helpful for illustrating what science is and is not, and what rationality is and is not. Otherwise you end up trusting in what you call "science", which won't be strict enough.

"For another thing," I continued, "part of what goes into becoming a rationalist, is learning to live into a counterintuitive world - learning to find things underneath the surface that are unlike the world of surface forms." Quantum mechanics makes a good introduction to that, when done correctly without the horrible confusion and despair. It breaks you of your belief in an intuitive universe, counters naive realism, destroys your trust in the way that your cognitive algorithms look from inside - and then you're ready to start seeing your mind as a mind, not as a window onto reality.

"But you're writing about physics, without being a physicist," the one said, "isn't that... a little..."

"Yes," I said, "it is, and I felt guilty about it. But there were physicists talking complete nonsense about Occam's Razor without knowing the probability theory of it, so my hand was forced. Also the situation in teaching quantum mechanics is really awful - I saw the introductions to Bayesianism and they seemed unnecessarily difficult, but the situation in quantum mechanics is so much worse." It really is. I remember sitting there staring at the "linear operators", trying to figure out what the hell they physically did to the eigenvectors - trying to visualize the actual events that were going on in the physical evolution - before it dawned on me that it was just a math trick to extract the average of the eigenvalues. Okay, but... can't you just tell me that up front? Write it down somewhere? Oh, I forgot, the math doesn't mean anything, it just works.

"Furthermore," I added, "knowing about many worlds, helps you visualize probabilities as frequencies, which is helpful to many points I want to make."

"And furthermore," I said, "reducing time to non-time is a powerful example of the principle, in reductionism, that you should reduce something to something other than itself."

"And even furthermore," I said, "I had to break my readers of trust in Science, even trust in physicists, because it doesn't seem possible to think and trust at the same time."

"Many-worlds is really a very clear and simple problem," I said, "by comparison with the challenges you encounter in AI, which are around a hundred times less clear-cut. And many scientists can't even get many-worlds, in the absence of authority." So you are left with no choice but to aspire to do better than the average scientist; a hell of a lot better, in fact. This notion is one that you cannot just blurt out to people without showing them why it is necessary.

Another helpful advantage - I often do things with quite a few different purposes in mind, as you may have realized by this point - was that you can see various commenters who still haven't gotten it, who are still saying, "Oh, look, Eliezer is overconfident because he believes in many-worlds."

Well, if you can viscerally see the arguments I have laid forth, then you can see that I am not being careless in having an opinion about physics. The balance of arguments is overwhelmingly tipped; and physicists who deny it, are making specific errors of probability theory (which I have specifically laid out, and shown to you) that they might not be expected to know about. It is not just a matter of my forming strong opinions at random.

But would you believe that I had such strong support, if I had not shown it to you in full detail? Ponder this well. For I may have other strong opinions. And it may seem to you that you don't see any good reason to form such strong beliefs. Except this is not what you will see; you will see simply that there is no good reason for strong belief, that there is no strong support one way or the other. For our first-order beliefs are how the world seems to be. And you may think, "Oh, Eliezer is just opinionated - forming strong beliefs in the absence of lopsided support." And I will not have the time to do another couple of months worth of blog posts.

I am very far from infallible, but I do not hold strong opinions at random.

"And yet still furthermore," I said, "transhumanist mailing lists have been arguing about issues of personal identity for years, and a tremendous amount of time has been wasted on it." Probably most who argue, will not bother to read what I have set forth; but if it stops any intelligent folk from wasting further time, that too is a benefit.

I am sometimes accused of being overconfident and opinionated, for telling people that being composed of "the same atoms" has nothing to do with their personal continuity. Or for saying that an uploading scan performed to the same resolution as thermal noise, actually has less effect on your identity than a sneeze (because your eyes squeeze shut when you sneeze, and that actually alters the computational state of billions of neurons in your visual cortex). Yet if you can see your nows braided into causality of the river that never flows; and the synaptic connections computing your internal narrative, that remain the same from one time to another, though not a drop of water is shared; then you can see that I have reasons for this strong belief as well.

Perhaps the one says to me that the exact duplicate constructed on Mars, is just a copy. And I post a short comment saying, "Wrong. There is no copy, there are two originals. This is knowable and I know it." Would you have thought that I might have very strong support, that you might not be seeing?

I won't always have the time to write a month of blog posts. While I am enough of a Traditional Rationalist that I dislike trust, and will not lightly ask it, I may ask it of you if your life is at stake.

Another one once asked me: "What does quantum physics have to do with overcoming bias?"

Robin Hanson chose the name "Overcoming Bias"; but names are not steel chains. If I'd started my own personal blog for the material I'm now posting, I would have called it "Reinventing Rationality" or something along those lines - and even that wouldn't have been the real purpose, which would have been harder to explain.

What are these series of posts, really? Raw material for a popular book on rationality - but maybe a tenth of this material, or less, will make it into the book. One of the reasons I write long posts, is so that I can shorten them later with a good conscience, knowing that I did lay out the full argument somewhere. But the whole quantum physics sequence is probably not going to make it into the popular book at all - and neither will many other posts. So what's the rest of it for?

Sometimes I think wistfully of how much more I could have accomplished in my teenage years, if I had known a fraction of what I know now at age 15. (This is the age at which I was a Traditional Rationalist, and dedicated and bright as such ones go, but knew not the Way of Bayes.) You can think of these blog posts, perhaps, as a series of letters to my past self. Only not exactly, because some of what I now write, I did already know then.

It seems to me, looking back, that the road which took me to this Way, had a great deal of luck in it. I would like to eliminate that element of luck for those who come after. So some of what I post, is more formal explanations of matters which Eliezer-15 knew in his bones. And the rest, I only wish I had known.

Perhaps if you prime someone with enough material as a starting point, they can figure out the other 95% on their own, if they go on to study the relevant sciences at a higher technical level. That's what I hope.

Eliezer-15 was led far astray by the seeming mysteriousness of quantum mechanics. An antiproject in which he was aided and abetted by certain popular physicists - notably Sir Roger Penrose; but also all those physicists who told him that quantum physics was "mysterious" and that it was okay not to understand it.

This is something I wish I had known, so I explained it to me.

Why not just tell me to ignore quantum physics? Because I am not going to "just ignore" a question that large. It is not how we work.

If you are confronting real scientific chaos - not just some light matter of an experimental anomaly or the search for a better theory, but genuine fear and despair, as now exists in Artificial Intelligence - then it is necessary to be a polymath. Healthy fields have healthy ways of thinking; you cannot trust the traditions of the confused field you must reform, though you must learn them. You never know which way you'll need to draw upon, on venturing out into the unknown. You learn new sciences for the same reasons that programmers learn new programming languages: to change the way you think. If you want to never learn anything without knowing in advance how it will apply, you had best stay away from chaos.

If you want to tackle challenges on the order of AI, you can't just learn a bunch of AI stuff.

And finally...

Finally...

There finally comes a point where you get tired of trying to communicate across vast inferential distances. There comes a point where you get tired of not being able to say things to people without a month of preliminary explanation. There comes a point where you want to say something about branching Earths or identical particles or braids in the river that never flows, and you can't.

It is such a tremendous relief, to finally be able to say all these things. And all the other things, that I have said here; that people have asked me about for so long, and all I could do was wave my hands. I didn't have to explain the concept of "inferential distance" from scratch, I could just link to it. It is such a relief.

I have written hundreds of blog posts here. Think about what it would be like, to carry all that around inside your head.

If I can do all the long sequences on Overcoming Bias, then maybe after that, it will be possible to say most things that I want to say, in just one piece.

Sequence: Quantum Physics

Class Project

Referenced by: The Quantum Physics Sequence • Heading Toward Morality • Setting Up Metaethics

Original with comments: Why Quantum?

Living in Many Worlds

Eliezer Yudkowsky, 05 June 2008 02:24AM

This post is part of the Quantum Physics Sequence.
Followup to: Many Worlds, One Best Guess

Some commenters have recently expressed disturbance at the thought of constantly splitting into zillions of other people, as is the straightforward and unavoidable prediction of quantum mechanics.

Others have confessed themselves unclear as to the implications of many-worlds for planning: If you decide to buckle your seat belt in this world, does that increase the chance of another self unbuckling their seat belt? Are you being selfish at their expense?

Just remember Egan's Law: It all adds up to normality.

(After Greg Egan, in Quarantine.)

Frank Sulloway said: "Ironically, psychoanalysis has it over Darwinism precisely because its predictions are so outlandish and its explanations are so counterintuitive that we think, Is that really true? How radical! Freud’s ideas are so intriguing that people are willing to pay for them, while one of the great disadvantages of Darwinism is that we feel we know it already, because, in a sense, we do."

When Einstein overthrew the Newtonian version of gravity, apples didn't stop falling, planets didn't swerve into the Sun. Every new theory of physics must capture the successful predictions of the old theory it displaced; it should predict that the sky will be blue, rather than green.

So don't think that many-worlds is there to make strange, radical, exciting predictions. It all adds up to normality.

Then why should anyone care?

Because there was once asked the question, fascinating unto a rationalist: What all adds up to normality?

And the answer to this question turns out to be: quantum mechanics. It is quantum mechanics that adds up to normality.

If there were something else there instead of quantum mechanics, then the world would look strange and unusual.

Bear this in mind, when you are wondering how to live in the strange new universe of many worlds: You have always been there.

Religions, anthropologists tell us, usually exhibit a property called minimal counterintuitiveness; they are startling enough to be memorable, but not so bizarre as to be difficult to memorize. Anubis has the head of a dog, which makes him memorable, but the rest of him is the body of a man. Spirits can see through walls; but they still become hungry.

But physics is not a religion, set to surprise you just exactly enough to be memorable. The underlying phenomena are so counterintuitive that it takes long study for humans to come to grips with them. But the surface phenomena are entirely ordinary. You will never catch a glimpse of another world out of the corner of your eye. You will never hear the voice of some other self. That is unambiguously prohibited outright by the laws. Sorry, you're just schizophrenic.

The act of making decisions has no special interaction with the process that branches worlds. In your mind, in your imagination, a decision seems like a branching point where the world could go two different ways. But you would feel just the same uncertainty, visualize just the same alternatives, if there were only one world. That's what people thought for centuries before quantum mechanics, and they still visualized alternative outcomes that could result from their decisions.

Decision and decoherence are entirely orthogonal concepts. If your brain never became decoherent, then that single cognitive process would still have to imagine different choices and their different outcomes. And a rock, which makes no decisions, obeys the same laws of quantum mechanics as anything else, and splits frantically as it lies in one place.

You don't split when you come to a decision in particular, any more than you particularly split when you take a breath. You're just splitting all the time as the result of decoherence, which has nothing to do with choices.

There is a population of worlds, and in each world, it all adds up to normality: apples don't stop falling. In each world, people choose the course that seems best to them. Maybe they happen on a different line of thinking, and see new implications or miss others, and come to a different choice. But it's not that one world chooses each choice. It's not that one version of you chooses what seems best, and another version chooses what seems worst. In each world, apples go on falling and people go on doing what seems like a good idea.

Yes, you can nitpick exceptions to this rule, but they're normal exceptions. It all adds up to normality, in all the worlds.

You cannot "choose which world to end up in". In all the worlds, people's choices determine outcomes in the same way they would in just one single world.

The choice you make here does not have some strange balancing influence on some world elsewhere. There is no causal communication between decoherent worlds. In each world, people's choices control the future of that world, not some other world.

If you can imagine decisionmaking in one world, you can imagine decision-making in many worlds: just have the world constantly splitting while otherwise obeying all the same rules.

In no world does two plus two equal five. In no world can spaceships travel faster than light. All the quantum worlds obey our laws of physics; their existence is asserted in the first place by our laws of physics. Since the beginning, not one unusual thing has ever happened, in this or any other world. They are all lawful.

Are there horrible worlds out there, which are utterly beyond your ability to affect? Sure. And horrible things happened during the 12th century, which are also beyond your ability to affect. But the 12th century is not your responsibility, because it has, as the quaint phrase goes, "already happened". I would suggest that you consider every world which is not in your future, to be part of the "generalized past".

Live in your own world. Before you knew about quantum physics, you would not have been tempted to try living in a world that did not seem to exist. Your decisions should add up to this same normality: you shouldn't try to live in a quantum world you can't communicate with.

Your decision theory should (almost always) be the same, whether you suppose that there is a 90% probability of something happening, or if it will happen in 9 out of 10 worlds. Now, because people have trouble handling probabilities, it may be helpful to visualize something happening in 9 out of 10 worlds. But this just helps you use normal decision theory.

If you've been putting it off before, now is the time to learn to shut up and multiply. As I noted in Lotteries: A Waste of Hope:

The human brain doesn't do 64-bit floating-point arithmetic, and it can't devalue the emotional force of a pleasant anticipation by a factor of 0.00000001 without dropping the line of reasoning entirely.

And in New Improved Lottery:

Between zero chance of becoming wealthy, and epsilon chance, there is an order-of-epsilon difference. If you doubt this, let epsilon equal one over googolplex.

If you're thinking about a world that could arise in a lawful way, but whose probability is a quadrillion to one, and something very pleasant or very awful is happening in this world... well, it does probably exist, if it is lawful. But you should try to release one quadrillionth as many neurotransmitters, in your reward centers or your aversive centers, so that you can weigh that world appropriately in your decisions. If you don't think you can do that... don't bother thinking about it.

Otherwise you might as well go out and buy a lottery ticket using a quantum random number, a strategy that is guaranteed to result in a very tiny mega-win.

Or here's another way of thinking about it: Are you considering expending some mental energy on a world whose frequency in your future is less than a trillionth? Then go get a 10-sided die from your local gaming store, and, before you begin thinking about that strange world, start rolling the die. If the die comes up 9 twelve times in a row, then you can think about that world. Otherwise don't waste your time; thought-time is a resource to be expended wisely.

You can roll the dice as many times as you like, but you can't think about the world until 9 comes up twelve times in a row. Then you can think about it for a minute. After that you have to start rolling the die again.

This may help you to appreciate the concept of "trillion to one" on a more visceral level.

If at any point you catch yourself thinking that quantum physics might have some kind of strange, abnormal implication for everyday life - then you should probably stop right there.

Oh, there are a few implications of many-worlds for ethics. Average utilitarianism suddenly looks a lot more attractive - you don't need to worry about creating as many people as possible, because there are already plenty of people exploring person-space. You just want the average quality of life to be as high as possible, in the future worlds that are your responsibility.

And you should always take joy in discovery, as long as you personally don't know a thing. It is meaningless to talk of being the "first" or the "only" person to know a thing, when everything knowable is known within worlds that are in neither your past nor your future, and are neither before or after you.

But, by and large, it all adds up to normality. If your understanding of many-worlds is the tiniest bit shaky, and you are contemplating whether to believe some strange proposition, or feel some strange emotion, or plan some strange strategy, then I can give you very simple advice: Don't.

The quantum universe is not a strange place into which you have been thrust. It is the way things have always been.

Sequence: Quantum Physics

Many Worlds, One Best Guess

Mach's Principle: Anti-Epiphenomenal Physics

Referenced by: Thou Art Physics • The Quantum Physics Sequence • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity • My Kind of Reflection • Probability is Subjectively Objective • Changing Your Metaethics • Setting Up Metaethics • Inseparably Right; or, Joy in the Merely Good • Why Does Power Corrupt? • For The People Who Are Still Alive • Cynicism in Ev-Psych (and Econ?)

Original with comments: Living in Many Worlds

Thou Art Physics

Eliezer Yudkowsky, 06 June 2008 06:37AM

This post is part of the Quantum Physics Sequence and the Solution to "Free Will".
Followup to: Dissolving the Question, Hand vs. Fingers, Timeless Causality, Living in Many-Worlds

Three months ago - jeebers, has it really been that long? - I posed the following homework assignment: Do a stack trace of the human cognitive algorithms that produce debates about 'free will'. Note that this task is strongly distinguished from arguing that free will does, or does not exist.

Now, as expected, the notion of "timeless physics" is causing people to ask, "If the future is determined, how can our choices control it?" The wise reader can guess that it all adds up to normality; but this leaves the question of how.

People hear: "The universe runs like clockwork; physics is deterministic; the future is fixed." And their minds form an causal network that looks like this:

Here we see the causes "Me" and "Physics", competing to determine the state of the "Future" effect. If the "Future" is fully determined by "Physics", then obviously there is no room for it to be affected by "Me".

This causal network is not an explicit philosophical belief. It's implicit - a background representation of the brain, controlling which philosophical arguments seem "reasonable". It just seems like the way things are.

Every now and then, another neuroscience press release appears, claiming that, because researchers used an fMRI to spot the brain doing something-or-other during a decision process, it's not you who chooses, it's your brain.

Likewise that old chestnut, "Reductionism undermines rationality itself. Because then, every time you said something, it wouldn't be the result of reasoning about the evidence - it would be merely quarks bopping around."

Of course the actual diagram should be:

Or better yet:

Why is this not obvious? Because there are many levels of organization that separate our models of our thoughts - our emotions, our beliefs, our agonizing indecisions, and our final choices - from our models of electrons and quarks.

We can intuitively visualize that a hand is made of fingers (and thumb and palm). To ask whether it's really our hand that picks something up, or merely our fingers, thumb, and palm, is transparently a wrong question.

But the gap between physics and cognition cannot be crossed by direct visualization. No one can visualize atoms making up a person, the way they can see fingers making up a hand.

And so it requires constant vigilance to maintain your perception of yourself as an entity within physics.

This vigilance is one of the great keys to philosophy, like the Mind Projection Fallacy. You will recall that it is this point which I nominated as having tripped up the quantum physicists who failed to imagine macroscopic decoherence; they did not think to apply the laws to themselves.

Beliefs, desires, emotions, morals, goals, imaginations, anticipations, sensory perceptions, fleeting wishes, ideals, temptations... You might call this the "surface layer" of the mind, the parts-of-self that people can see even without science. If I say, "It is not you who determines the future, it is your desires, plans, and actions that determine the future", you can readily see the part-whole relations. It is immediately visible, like fingers making up a hand. There are other part-whole relations all the way down to physics, but they are not immediately visible.

"Compatibilism" is the philosophical position that "free will" can be intuitively and satisfyingly defined in such a way as to be compatible with deterministic physics. "Incompatibilism" is the position that free will and determinism are incompatible.

My position might perhaps be called "Requiredism." When agency, choice, control, and moral responsibility are cashed out in a sensible way, they require determinism - at least some patches of determinism within the universe. If you choose, and plan, and act, and bring some future into being, in accordance with your desire, then all this requires a lawful sort of reality; you cannot do it amid utter chaos. There must be order over at least over those parts of reality that are being controlled by you. You are within physics, and so you/physics have determined the future. If it were not determined by physics, it could not be determined by you.

Or perhaps I should say, "If the future were not determined by reality, it could not be determined by you," or "If the future were not determined by something, it could not be determined by you." You don't need neuroscience or physics to push naive definitions of free will into incoherence. If the mind were not embodied in the brain, it would be embodied in something else; there would be some real thing that was a mind. If the future were not determined by physics, it would be determined by something, some law, some order, some grand reality that included you within it.

But if the laws of physics control us, then how can we be said to control ourselves?

Turn it around: If the laws of physics did not control us, how could we possibly control ourselves?

How could thoughts judge other thoughts, how could emotions conflict with each other, how could one course of action appear best, how could we pass from uncertainty to certainty about our own plans, in the midst of utter chaos?

If we were not in reality, where could we be?

The future is determined by physics. What kind of physics? The kind of physics that includes the actions of human beings.

People's choices are determined by physics. What kind of physics? The kind of physics that includes weighing decisions, considering possible outcomes, judging them, being tempted, following morals, rationalizing transgressions, trying to do better...

There is no point where a quark swoops in from Pluto and overrides all this.

The thoughts of your decision process are all real, they are all something. But a thought is too big and complicated to be an atom. So thoughts are made of smaller things, and our name for the stuff that stuff is made of, is "physics".

Physics underlies our decisions and includes our decisions, it does not explain them away.

To be continued... when I take up the issue of control in a timeless universe. As always, I suggest that the reader should first try to solve the problem on their own. Remember, physics adds up to normality; it's your cognitive algorithms that generate confusion.

Sequence: Quantum Physics

Timeless Identity

Timeless Control

Referenced by: Timeless Control • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity • Causality and Moral Responsibility • The Ultimate Source • Heading Toward Morality • Does Your Morality Care What You Think? • Lawful Creativity • High Challenge • Free to Optimize • The Evolutionary-Cognitive Boundary • Hyakujo's Fox • Ingredients of Timeless Decision Theory

Original with comments: Thou Art Physics

Timeless Control

Eliezer Yudkowsky, 07 June 2008 05:16AM

This post is part of the Quantum Physics Sequence and the Solution to "Free Will".
Followup to: Timeless Physics, Timeless Causality, Thou Art Physics

People hear about many-worlds, which is deterministic, or about timeless physics, and ask:

If the future is determined by physics, how can anyone control it?

In Thou Art Physics, I pointed out that since you are within physics, anything you control is necessarily controlled by physics. Today we will talk about a different aspect of the confusion, the words "determined" and "control".

The "Block Universe" is the classical term for the universe considered from outside Time. Even without timeless physics, Special Relativity outlaws any global space of simultaneity, which is widely believed to suggest the Block Universe - spacetime as one vast 4D block.

When you take a perspective outside time, you have to be careful not to let your old, timeful intuitions run wild in the absence of their subject matter.

In the Block Universe, the future is not determined before you make your choice. "Before" is a timeful word. Once you descend so far as to start talking about time, then, of course, the future comes "after" the past, not "before" it.

If we're going to take a timeless perspective, then the past and the future have not always been there. The Block Universe is not something that hangs, motionless and static, lasting for a very long time. You might try to visualize the Block Universe hanging in front of your mind's eye, but then your mind's eye is running the clock while the universe stays still. Nor does the Block Universe exist for just a single second, and then disappear. It is not instantaneous. It is not eternal. It does not last for exactly 15 seconds. All these are timeful statements. The Block Universe is simply there.

Perhaps people imagine a Determinator - not so much an agent, perhaps, but a mysterious entity labeled "Determinism" - which, at "the dawn of time", say, 6:00am, writes down your choice at 7:00am, and separately, writes the outcome at 7:02am. In which case, indeed, the future would be determined before you made your decision...

In this model, the Determinator writes the script for the Block Universe at 6:00am. And then time - the global time of the universe - continues, running through the Block Universe and realizing the script.

At 7:00am you're trying to decide to turn on the light bulb. But the Determinator already decided at 6:00am whether the light bulb would be on or off at 7:02am. Back at the dawn of time when Destiny wrote out the Block Universe, which was scripted before you started experiencing it...

This, perhaps, is the kind of unspoken, intuitive mental model that might lead people to talk about "determinism" implying that the future is determined before you make your decision.

Even without the concept of the Block Universe or timeless physics, this is probably what goes on when people start talking about "deterministic physics" in which "the whole course of history" was fixed at "the dawn of time" and therefore your choices have no effect on the "future".

As described in Timeless Causality, "cause" and "effect" are things we talk about by pointing to relations within the Block Universe. E.g., we might expect to see human colonies separated by an expanding cosmological horizon; we can expect to find correlation between two regions that communicate with a mutual point in the "past", but have no light-lines to any mutual points in their "future". But we wouldn't expect to find a human colony in a distant supercluster, having arrived from the other side of the universe; we should not find correlation between regions with a shared "future" but no shared "past". This is how we can experimentally observe the orientation of the Block Universe, the direction of the river that never flows.

If you are going to talk about causality at all - and personally, I think we should, because the universe doesn't make much sense without it - then causality applies to relations within the Block Universe, not outside it.

The Past is just there, and the Future is just there, but the relations between them have a certain kind of structure - whose ultimate nature, I do not conceive myself to understand - but which we do know a bit about mathematically; the structure is called "causality".

(I am not ruling out the possibility of causality that extends outside the Block Universe - say, some reason why the laws of physics are what they are. We can have timeless causal relations, remember? But the causal relations between, say, "dropping a glass" and "water spilling out", or between "deciding to do something" and "doing it", are causal relations embedded within the Block.)

One of the things we can do with graphical models of causality - networks of little directed arrows - is construe counterfactuals: Statements about "what would have happened if X had occurred, instead of Y".

These counterfactuals are untestable, unobservable, and do not actually exist anywhere that I've been able to find. Counterfactuals are not facts, unless you count them as mathematical properties of certain causal diagrams. We can define statistical properties we expect to see, given a causal hypothesis; but counterfactuals themselves are not observable. We cannot see what "would have happened, if I hadn't dropped the glass".

Nonetheless, if you draw the causal graph that the statistics force you to draw, within our Block Universe, and you construct the counterfactual, then you get statements like: "If I hadn't dropped the glass, the water wouldn't have spilled."

If your mind contains the causal model that has "Determinism" as the cause of both the "Past" and the "Future", then you will start saying things like, But it was determined before the dawn of time that the water would spill - so not dropping the glass would have made no difference. This would be the standard counterfactual, on the causal graph in which "Past" and "Future" are both children of some mutual ancestor, but have no connection between them.

And then there's the idea that, if you can predict the whole course of the universe by looking at the state at the beginning of time, the present must have no influence on the future...

Surely, if you can determine the Future just by looking at the Past, there's no need to look at the Present?

The problem with the right-side graph is twofold: First, it violates the beautiful locality of reality; we're supposing causal relations that go outside the immediate neighborhoods of space/time/configuration. And second, you can't compute the Future from the Past, except by also computing something that looks exactly like the Present; which computation just creates another copy of the Block Universe (if that statement even makes any sense), it does not affect any of the causal relations within it.

One must avoid mixing up timeless and timeful thinking. E.g., trying to have "Determinism" acting on things before they happen. Determinism is a timeless viewpoint, so it doesn't mix well with words like "before".

The same thing happens if you try to talk about how the Past at 6:30am determines the Future at 7:30am, and therefore, the state at 7:30am is already determined at 6:30am, so you can't control it at 7:00am, because it was determined at 6:30am earlier...

What is determined is a timeless mathematical structure whose interior includes 7:00am and 7:30am. That which you might be tempted to say "already exists" at 6:00am, does not exist before 7:00am, it is something whose existence includes the Now of 7:00am and the Now of 7:30am.

If you imagine a counterfactual surgery on the interior of the structure at 7:00am, then, according to the statistically correct way to draw the arrows of causality within the structure, the 7:30am part would be affected as well.

So it is exactly correct to say, on the one hand, "The whole future course of the universe was determined by its state at 6:30am this morning," and, on the other, "If I hadn't dropped the glass, the water wouldn't have spilled." In the former case you're talking about a mathematical object outside time; in the latter case you're talking about cause and effect inside the mathematical object. Part of what is determined is that dropping the glass in the Now of 7:00:00am, causes the water to spill in the Now of 7:00:01am.

And as pointed out in Thou Art Physics, you are inside that mathematical object too. So are your thoughts, emotions, morals, goals, beliefs, and all else that goes into the way you determine your decisions.

To say "the future is already written" is a fine example of mixed-up timeful and timeless thinking. The future is. It is not "already". What is it that writes the future? In the timeless causal relations, we do. That is what is written: that our choices control the future.

But how can you "control" something without changing it?

"Change" is a word that makes sense within time, and only within time. One observes a macroscopically persistent object, like, say, a lamp, and compares its state at 7:00am to its state at 7:02am. If the two states are different, then we say that "the lamp" changed over time.

In Timeless Physics, I observed that, while things can change from one time to another, a single moment of time is never observed to change:

At 7:00am, the lamp is off. At 7:01am, I flip the switch... At 7:02am, the lamp is fully bright. Between 7:00am and 7:02am, the lamp changed from OFF to ON.

But have you ever seen the future change from one time to another? Have you wandered by a lamp at exactly 7:02am, and seen that it is OFF; then, a bit later, looked in again on the "the lamp at exactly 7:02am", and discovered that it is now ON?

But if you have to change a single moment of time, in order to be said to "control" something, you really are hosed.

Forget this whole business of deterministic physics for a moment.

Let's say there was some way to change a single moment of time.

We would then need some kind of meta-time over which time could "change".

The lamp's state would need to change from "OFF at 7:02am at 3:00meta-am" to "ON at 7:02am at 3:01meta-am".

But wait! Have you ever seen a lamp change from OFF at 7:02am at 3:00meta-am, to ON at 7:02am at 3:00meta-am? No! A single instant of meta-time never changes, so you cannot change it, and you have no control.

Now we need meta-meta time.

So if we're going to keep our concepts of "cause" and "control" and "choose" - and to discard them would leave a heck of a lot observations unexplained - then we've got to figure out some way to define them within time, within that which is written, within the Block Universe, within... well... reality.

Control lets you change things from one time to another; you can turn on a lamp that was previously off. That's one kind of control, and a fine sort of control it is to have. But trying to pull this stunt on a single moment of time, is a type error.

If you isolate a subsystem of reality, like a rock rolling down hill, then you can mathematically define the future-in-isolation of that subsystem; you can take the subsystem in isolation, and compute what would happen to it if you did not act on it. In this case, what would happen is that the rock would reach the bottom of the hill. This future-in-isolation is not something that actually happens in the Block Universe; it is a computable property of the subsystem as it exists at some particular moment. If you reach in from outside the isolation, you can stop the rock from rolling. Now if you walk away, and again leave the system isolated, the future-in-isolation will be that the rock just stays there. But perhaps someone will reach in, and tip the rock over and start it rolling again. The hill is not really isolated - the universe is a continuous whole - but we can imagine what would happen if the hill were isolated. This is a "counterfactual", so called because they are not factual.

The future-in-isolation of a subsystem can change from one time to another, as the subsystem itself changes over time as the result of actions from outside. The future of the Grand System that includes everything, cannot change as the result of outside action.

People want to place themselves outside the System, see themselves separated from it by a Cartesian boundary. But even if free will could act outside physics to change the Block Universe, we would just have a Grand System that included free-will+physics and the future would be fully determined by that. If you have "freer will" we just have an Even Grander System, and so on.

It's hard to put yourself outside Reality. Whatever is, is real.

Control lets you determine single moments of time (though they do not change from one meta-time to another). You can change what would have happened, from one time to another. But you cannot change what does happen - just determine it. Control means that you are what writes the written future, according to the laws of causality as they exist within the writing.

Or maybe look at it this way: Pretend, for a moment, that naive views of free will were correct. The future "doesn't exist yet" and can be "changed". (Note: How are these two statements compatible?) Suppose that you exercise your "free will" at 6:30am to rescue three toddlers from a burning orphanage, changing their future from horrible flamey death at 7:00am, to happy gurgling contentment at 7:00am.

But now it is 7:30am, and I say:

"Aha! The past is fixed and can never be altered! So now you cannot ever have chosen any differently than you did choose. Furthermore, the actual outcome of your actions can never change either; the outcome is now fixed, so even if your past choice did now change, the past outcome wouldn't, because they are both just determined by "The Past". While your will was once free at 6:30am to change the future at 7:00am, it is now 7:30am and this freedom no longer exists. So now your will at 6:30am is no longer free. How can your past will have been free, now that there is only one past? Therefore I do not now assign you any moral credit for saving the orphanage; you no longer could have chosen differently from how you chose."

In the Block Universe, the "past" and the "future" are just perspectives, taken from some point within the Block. So, if the fixation of the past doesn't prevent the embedded decisions from having (had?) the property of freedom, why should the determination of the future prevent those embedded decisions from having the same property?

In the Block Universe, the Future is just like the Past: it contains the Nows of people making choices that determine their outcomes, which do not change from one meta-time to another.

And given the way we draw the causal arrows, it is correct to form the (un-observable) counterfactuals, "If I hadn't saved those children from the orphanage, they would be dead," and "If I don't think carefully, my thoughts will end up in Outer Mongolia." One is a counterfactual over the past, and one is a counterfactual over the future; but they are both as correct as a counter-factual can be.

The next step in analyzing the cognitive issues surrounding free will, is to take apart the word "could" - as in "I could have decided not to save the children from the orphanage." As always, I encourage the reader to try to get it in advance - this one is easier if you know a certain simple algorithm from Artificial Intelligence.

PPS: It all adds up to normality.

Sequence: Quantum Physics

Thou Art Physics

The Failures of Eld Science

Referenced by: Thou Art Physics • The Quantum Physics Sequence • Quantum Mechanics and Personal Identity • Causality and Moral Responsibility • Possibility and Could-ness • The Ultimate Source • Heading Toward Morality • Setting Up Metaethics • Free to Optimize • Cynical About Cynicism • Ingredients of Timeless Decision Theory

Original with comments: Timeless Control

Bloggingheads: Yudkowsky and Horgan

Eliezer Yudkowsky, 07 June 2008 10:09PM

I appear today on Bloggingheads.tv, in "Science Saturday: Singularity Edition", speaking with John Horgan about the Singularity. I talked too much. This episode needed to be around two hours longer.

One question I fumbled at 62:30 was "What's the strongest opposition you've seen to Singularity ideas?" The basic problem is that nearly everyone who attacks the Singularity is either completely unacquainted with the existing thinking, or they attack Kurzweil, and in any case it's more a collection of disconnected broadsides (often mostly ad hominem) than a coherent criticism. There's no equivalent in Singularity studies of Richard Jones's critique of nanotechnology - which I don't agree with, but at least Jones has read Drexler. People who don't buy the Singularity don't put in the time and hard work to criticize it properly.

What I should have done, though, was interpreted the question more charitably as "What's the strongest opposition to strong AI or transhumanism?" in which case there's Sir Roger Penrose, Jaron Lanier, Leon Kass, and many others. None of these are good arguments - or I would have to accept them! - but at least they are painstakingly crafted arguments, and something like organized opposition.

Original with comments: Bloggingheads: Yudkowsky and Horgan

Against Devil's Advocacy

Eliezer Yudkowsky, 09 June 2008 04:15AM

From an article by Michael Ruse:

Richard Dawkins once called me a "creep." He did so very publicly but meant no personal offense, and I took none: We were, and still are, friends. The cause of his ire—his anguish, even—was that, in the course of a public discussion, I was defending a position I did not truly hold. We philosophers are always doing this; it's a version of the reductio ad absurdum argument. We do so partly to stimulate debate (especially in the classroom), partly to see how far a position can be pushed before it collapses (and why the collapse), and partly (let us be frank) out of sheer bloody-mindedness, because we like to rile the opposition.

Dawkins, however, has the moral purity—some would say the moral rigidity—of the evangelical Christian or the committed feminist. Not even for the sake of argument can he endorse something that he thinks false. To do so is not just mistaken, he feels; in some deep sense, it is wrong. Life is serious, and there are evils to be fought. There must be no compromise or equivocation, even for pedagogical reasons. As the Quakers say, "Let your yea be yea, and your nay, nay."

Michael Ruse doesn't get it.

When I was a kid and my father was teaching me about skepticism -

(Dad was an avid skeptic and Martin Gardner / James Randi fan, as well as being an Orthodox Jew. Let that be a lesson on the anti-healing power of compartmentalization.)

- he used the example of the hypothesis: "There is an object one foot across in the asteroid belt composed entirely of chocolate cake." You would have to search the whole asteroid belt to disprove this hypothesis. But though this hypothesis is very hard to disprove, there aren't good arguments for it.

And the child-Eliezer asked his mind to search for arguments that there was a chocolate cake in the asteroid belt. Lo, his mind returned the reply: "Since the asteroid-belt-chocolate-cake is one of the classic examples of a bad hypothesis, if anyone ever invents a time machine, some prankster will toss a chocolate cake back into the 20th-century asteroid belt, making it true all along."

Thus - at a very young age - I discovered that my mind could, if asked, invent arguments for anything.

I know people whose sanity has been destroyed by this discovery. They conclude that Reason can be used to argue for anything. And so there is no point in arguing that God doesn't exist, because you could just as well argue that God does exist. Nothing left but to believe whatever you want.

Having given up, they develop whole philosophies of self-inflation to make their despair seem Deeply Wise. If they catch you trying to use Reason, they will smile and pat you on the head and say, "Oh, someday you'll discover that you can argue for anything."

Perhaps even now, my readers are thinking, "Uh-oh, Eliezer can rationalize anything, that's not a good sign."

But you know... being mentally agile doesn't always doom you to disaster. I mean, you might expect that it would. Yet sometimes practice turns out to be different from theory.

Rationalization came too easily to me. It was visibly just a game.

If I had been less imaginative and more easily stumped - if I had not found myself able to argue for any proposition no matter how bizarre - then perhaps I would have confused the activity with thinking.

But I could argue even for chocolate cake in the asteroid belt. It wasn't even difficult; my mind coughed up the argument immediately. It was very clear that this was fake thinking and not real thinking. I never for a moment confused the game with real life. I didn't start thinking there might really be a chocolate cake in the asteroid belt.

You might expect that any child with enough mental agility to come up with arguments for anything, would surely be doomed. But intelligence doesn't always do so much damage as you might think. In this case, it just set me up, at a very early age, to distinguish "reasoning" from "rationalizing". They felt different.

Perhaps I'm misremembering... but it seems to me that, even at that young age, I looked at my mind's amazing clever argument for a time-traveling chocolate cake, and thought: I've got to avoid doing that.

(Though there are much more subtle cognitive implementations of rationalizing processes, than blatant, obvious, conscious search for favorable arguments. A wordless flinch away from an idea can undo you as surely as a deliberate search for arguments against it. Those subtler processes, I only began to notice years later.)

I picked up an intuitive sense that real thinking was that which could force you into an answer whether you liked it or not, and fake thinking was that which could argue for anything.

This was an incredibly valuable lesson -

(Though, like many principles that my young self obtained by reversing stupidity, it gave good advice on specific concrete problems; but went wildly astray when I tried to use it to make abstract deductions, e.g. about the nature of morality.)

- which was one of the major drivers behind my break with Judaism. The elaborate arguments and counterarguments of ancient rabbis, looked like the kind of fake thinking I did to argue that there was chocolate cake in the asteroid belt. Only the rabbis had forgotten it was a game, and were actually taking it seriously.

Believe me, I understand the Traditional argument behind Devil's Advocacy. By arguing the opposing position, you increase your mental flexibility. You shake yourself out of your old shoes. You get a chance to gather evidence against your position, instead of arguing for it. You rotate things around, see them from a different viewpoint. Turnabout is fair play, so you turn about, to play fair.

Perhaps this is what Michael Rose was thinking, when he accused Richard Dawkins of "moral rigidity".

I surely don't mean to teach people to say: "Since I believe in fairies, I ought not to expect to find any good arguments against their existence, therefore I will not search because the mental effort has a low expected utility." That comes under the heading of: If you want to shoot your foot off, it is never the least bit difficult to do so.

Maybe there are some stages of life, or some states of mind, in which you can be helped by trying to play Devil's Advocate. Students who have genuinely never thought of trying to search for arguments on both sides of an issue, may be helped by the notion of "Devil's Advocate".

But with anyone in this state of mind, I would sooner begin by teaching them that policy debates should not appear one-sided. There is no expectation against having strong arguments on both sides of a policy debate; single actions have multiple consequences. If you can't think of strong arguments against your most precious favored policies, or strong arguments for policies that you hate but which other people endorse, then indeed, you very likely have a problem that could be described as "failing to see the other points of view".

You, dear reader, are probably a sophisticated enough reasoner that if you manage to get yourself stuck in an advanced rut, dutifully playing Devil's Advocate won't get you out of it. You'll just subconsciously avoid any Devil's arguments that make you genuinely nervous, and then congratulate yourself for doing your duty. People at this level need stronger medicine. (So far I've only covered medium-strength medicine.)

If you can bring yourself to a state of real doubt and genuine curiosity, there is no need for Devil's Advocacy. You can investigate the contrary position because you think it might be really genuinely true, not because you are playing games with time-traveling chocolate cakes. If you cannot find this trace of true doubt within yourself, can merely playing Devil's Advocate help you?

I have no trouble thinking of arguments for why the Singularity won't happen for another 50 years. With some effort, I can make a case for why it might not happen in 100 years. I can also think of plausible-sounding scenarios in which the Singularity happens in two minutes, i.e., someone ran a covert research project and it is finishing right now. I can think of plausible arguments for 10-year, 20-year, 30-year, and 40-year timeframes.

This is not because I am good at playing Devil's Advocate and coming up with clever arguments. It's because I really don't know. A true doubt exists in each case, and I can follow my doubt to find the source of a genuine argument. Or if you prefer: I really don't know, because I can come up with all these plausible arguments.

On the other hand, it is really hard for me to visualize the proposition that there is no kind of mind substantially stronger than a human one. I have trouble believing that the human brain, which just barely suffices to run a technological civilization that can build a computer, is also the theoretical upper limit of effective intelligence. I cannot argue effectively for that, because I do not believe it. Or if you prefer, I do not believe it, because I cannot argue effectively for it. If you want that idea argued, find someone who really believes it. Since a very young age, I've been endeavoring to get away from those modes of thought where you can argue for just anything.

In the state of mind and stage of life where you are trying to distinguish rationality from rationalization, and trying to tell the difference between weak arguments and strong arguments, Devil's Advocate cannot lead you to unfake modes of reasoning. Its only power is that it may perhaps show you the fake modes which operate equally well on any side, and tell you when you are uncertain.

There is no chess grandmaster who can play only black, or only white; but in the battles of Reason, a soldier who fights with equal strength on any side has zero force.

That's what Richard Dawkins understands that Michael Ruse doesn't - that Reason is not a game.

Added: Brandon argues that Devil's Advocacy is most importantly a social rather than individual process, which aspect I confess I wasn't thinking about.

Referenced by: LA-602 vs. RHIC Review • Fundamental Doubts • Contaminated by Optimism

Original with comments: Against Devil's Advocacy

Eliezer's Post Dependencies; Book Notification; Graphic Designer Wanted

Eliezer Yudkowsky, 10 June 2008 01:18AM

I'm going to try and produce summaries of the quantum physics series today or tomorrow.

Andrew Hay has produced a neat graph of (explicit) dependencies among my Overcoming Bias posts - an automatically generated map of the "Followup to" structure:

Eliezer's Post Dependencies (includes only posts with dependencies)
All of my posts (including posts without dependencies)

Subscribe here to future email notifications for when the popular book comes out (which may be a year or two later), and/or I start producing e-books:

Notifications for the rationality book, or for any other stuff I produce

(Thanks to Christian Rovner for setting up PHPList.)

Sometime in the next two weeks, I need to get at least one Powerpoint presentation of mine re-produced to professional standards of graphic design. Ideally, in a form that will let me make small modifications myself. This is likely to lead into other graphic design work on producing the ebooks, redesigning my personal website, creating Bayesian Conspiracy T-shirts, etc.

I am not looking for an unpaid volunteer. I am looking for a professional graphic designer who can do sporadic small units of work quickly.

Desired style for the presentation: Professional-looking and easy-to-read (as opposed to flamboyant / elaborate). I already have the presentation content, in black text on white background. I would like it to look like it was produced by a grownup, which is beyond my own skill. Emails to sentience@pobox.com, please include your fee schedule and a link to your portfolio.

Original with comments: Eliezer's Post Dependencies; Book Notification; Graphic Designer Wanted

The Quantum Physics Sequence

Eliezer Yudkowsky, 11 June 2008 03:42AM

This is an inclusive guide to the series of posts on quantum mechanics that began on April 9th, 2008, including the digressions into related topics (such as the difference between Science and Bayesianism) and some of the preliminary reading.

You may also be interested in one of the less inclusive post guides, such as:

- An Intuitive Explanation of Quantum Mechanics (just the science, for students confused by their physics textbooks)

- Quantum Physics Revealed As Non-Mysterious (quantum physics does not make the universe any more mysterious than it was previously)

- And the Winner is... Many-Worlds! (the many-worlds interpretations wins outright given the current state of evidence)

- Quantum Mechanics and Personal Identity (the ontology of quantum mechanics, in which there are no particles with individual identities, rules out theories of personal continuity that invoke "the same atoms" as a concept)

My current plan calls for the quantum physics series to eventually be turned into one or more e-books.

Preliminaries:

- Probability is in the Mind: If you are uncertain about a phenomenon, this is a fact about your state of mind, not a fact about the phenomenon itself. There are mysterious questions but not mysterious answers. The map is not the territory.

- Reductionism: We build models of the universe that have many different levels of description. But so far as anyone has been able to determine, the universe itself has only the single level of fundamental physics - reality doesn't explicitly compute protons, only quarks.

- Joy in the Merely Real: If you can't take joy in things that turn out to be explicable, you're going to set yourself up for eternal disappointment. Don't worry if quantum physics turns out to be normal.

- Zombies! Zombies? and The Generalized Anti-Zombie Principle: Don't try to put your consciousness or your personal identity outside physics. Whatever makes you say "I think therefore I am", causes your lips to move; it is within the chains of cause and effect that produce our observed universe.

- Belief in the Implied Invisible: If a spaceship goes over the cosmological horizon relative to us, so that it can no longer communicate with us, should we believe that the spaceship instantly ceases to exist?

Basic Quantum Mechanics:

- Quantum Explanations: Quantum mechanics doesn't deserve its fearsome reputation. If you tell people something is supposed to be mysterious, they won't understand it. It's human intuitions that are "strange" or "weird"; physics itself is perfectly normal. Talking about historical erroneous concepts like "particles" or "waves" is just asking to confuse people; present the real, unified quantum physics straight out. The series will take a strictly realist perspective - quantum equations describe something that is real and out there. Warning: Although a large faction of physicists agrees with this, it is not universally accepted. Stronger warning: I am not even going to present non-realist viewpoints until later, because I think this is a major source of confusion.

- Configurations and Amplitude: A preliminary glimpse at the stuff reality is made of. The classic split-photon experiment with half-silvered mirrors. Alternative pathways the photon can take, can cancel each other out. The mysterious measuring tool that tells us the relative squared moduli.

- Joint Configurations: The laws of physics are inherently over mathematical entities, configurations, that involve multiple particles. A basic, ontologically existent entity, according to our current understanding of quantum mechanics, does not look like a photon - it looks like a configuration of the universe with "A photon here, a photon there." Amplitude flows between these configurations can cancel or add; this gives us a way to detect which configurations are distinct. It is an experimentally testable fact that "Photon 1 here, photon 2 there" is the same configuration as "Photon 2 here, photon 1 there".

- Distinct Configurations: Since configurations are over the combined state of all the elements in a system, adding a sensor that detects whether a particle went one way or the other, becomes a new element of the system that can make configurations "distinct" instead of "identical". This confused the living daylights out of early quantum experimenters, because it meant that things behaved differently when they tried to "measure" them. But it's not only measuring instruments that do the trick - any sensitive physical element will do - and the distinctness of configurations is a physical fact, not a fact about our knowledge. There is no need to suppose that the universe cares what we think.

- Where Philosophy Meets Science: In retrospect, supposing that quantum physics had anything to do with consciousness was a big mistake. Could philosophers have told the physicists so? But we don't usually see philosophers sponsoring major advances in physics; why not?

- Can You Prove Two Particles Are Identical?: You wouldn't think that it would be possible to do an experiment that told you that two particles are completely identical - not just to the limit of experimental precision, but perfectly. You could even give a precise-sounding philosophical argument for why it was not possible - but the argument would have a deeply buried assumption. Quantum physics violates this deep assumption, making the experiment easy.

- Classical Configuration Spaces: How to visualize the state of a system of two 1-dimensional particles, as a single point in 2-dimensional space. A preliminary step before moving into...

- The Quantum Arena: Instead of a system state being associated with a single point in a classical configuration space, the instantaneous real state of a quantum system is a complex amplitude distribution over a quantum configuration space. What creates the illusion of "individual particles", like an electron caught in a trap, is a plaid distribution - one that happens to factor into the product of two parts. It is the whole distribution that evolves when a quantum system evolves. Individual configurations don't have physics; amplitude distributions have physics. Quantum entanglement is the general case; quantum independence is the special case.

- Feynman Paths: Instead of thinking that a photon takes a single straight path through space, we can regard it as taking all possible paths through space, and adding the amplitudes for every possible path. Nearly all the paths cancel out - unless we do clever quantum things, so that some paths add instead of canceling out. Then we can make light do funny tricks for us, like reflecting off a mirror in such a way that the angle of incidence doesn't equal the angle of reflection. But ordinarily, nearly all the paths except an extremely narrow band, cancel out - this is one of the keys to recovering the hallucination of classical physics.

- No Individual Particles: One of the chief ways to confuse yourself while thinking about quantum mechanics, is to think as if photons were little billiard balls bouncing around. The appearance of little billiard balls is a special case of a deeper level on which there are only multiparticle configurations and amplitude flows. It is easy to set up physical situations in which there exists no fact of the matter as to which electron was originally which.

- Identity Isn't In Specific Atoms, Three Dialogues on Identity: Given that there's no such thing as "the same atom", whether you are "the same person" from one time to another can't possibly depend on whether you're made out of the same atoms.

- Decoherence: A quantum system that factorizes can evolve into a system that doesn't factorize, destroying the illusion of independence. But entangling a quantum system with its environment, can appear to destroy entanglements that are already present. Entanglement with the environment can separate out the pieces of an amplitude distribution, preventing them from interacting with each other. Decoherence is fundamentally symmetric in time, but appears asymmetric because of the second law of thermodynamics.

- The So-Called Heisenberg Uncertainty Principle: Unlike classical physics, in quantum physics it is not possible to separate out a particle's "position" from its "momentum". The evolution of the amplitude distribution over time, involves things like taking the second derivative in space and multiplying by i to get the first derivative in time. The end result of this time evolution rule is that blobs of particle-presence appear to race around in physical space. The notion of "an exact particular momentum" or "an exact particular position" is not something that can physically happen, it is a tool for analyzing amplitude distributions by taking them apart into a sum of simpler waves. This uses the assumption and fact of linearity: the evolution of the whole wavefunction seems to always be the additive sum of the evolution of its pieces. Using this tool, we can see that if you take apart the same distribution into a sum of positions and a sum of momenta, they cannot both be sharply concentrated at the same time. When you "observe" a particle's position, that is, decohere its positional distribution by making it interact with a sensor, you take its wave packet apart into two pieces; then the two pieces evolve differently. The Heisenberg Principle definitely does not say that knowing about the particle, or consciously seeing it, will make the universe behave differently.

- Which Basis Is More Fundamental?: The position basis can be computed locally in the configuration space; the momentum basis is not local. Why care about locality? Because it is a very deep principle; reality itself seems to favor it in some way.

- Where Physics Meets Experience: Meet the Ebborians, who reproduce by fission. The Ebborian brain is like a thick sheet of paper that splits down its thickness. They frequently experience dividing into two minds, and can talk to their other selves. It seems that their unified theory of physics is almost finished, and can answer every question, when one Ebborian asks: When exactly does one Ebborian become two people?

- Where Experience Confuses Physicists: It then turns out that the entire planet of Ebbore is splitting along a fourth-dimensional thickness, duplicating all the people within it. But why does the apparent chance of "ending up" in one of those worlds, equal the square of the fourth-dimensional thickness? Many mysterious answers are proposed to this question, and one non-mysterious one.

- On Being Decoherent: When a sensor measures a particle whose amplitude distribution stretches over space - perhaps seeing if the particle is to the left or right of some dividing line - then the standard laws of quantum mechanics call for the sensor+particle system to evolve into a state of (particle left, sensor measures LEFT) + (particle right, sensor measures RIGHT). But when we humans look at the sensor, it only seems to say "LEFT" or "RIGHT", never a mixture like "LIGFT". This, of course, is because we ourselves are made of particles, and subject to the standard quantum laws that imply decoherence. Under standard quantum laws, the final state is (particle left, sensor measures LEFT, human sees "LEFT") + (particle right, sensor measures RIGHT, human sees "RIGHT").

- The Conscious Sorites Paradox: Decoherence is implicit in quantum physics, not an extra law on top of it. Asking exactly when "one world" splits into "two worlds" may be like asking when, if you keep removing grains of sand from a pile, it stops being a "heap". Even if you're inside the world, there may not be a definite answer. This puzzle does not arise only in quantum physics; the Ebborians could face it in a classical universe, or we could build sentient flat computers that split down their thickness. Is this really a physicist's problem?

- Decoherece is Pointless: There is no exact point at which decoherence suddenly happens. All of quantum mechanics is continuous and differentiable, and decoherent processes are no exception to this.

- Decoherent Essences: Decoherence is implicit within physics, not an extra law on top of it. You can choose representations that make decoherence harder to see, just like you can choose representations that make apples harder to see, but exactly the same physical process still goes on; the apple doesn't disappear and neither does decoherence. If you could make decoherence magically go away by choosing the right representation, we wouldn't need to shield quantum computers from the environment.

- The Born Probabilities: The last serious mysterious question left in quantum physics: When a quantum world splits in two, why do we seem to have a greater probability of ending up in the larger blob, exactly proportional to the integral of the squared modulus? It's an open problem, but non-mysterious answers have been proposed. Try not to go funny in the head about it.

- Decoherence as Projection: Since quantum evolution is linear and unitary, decoherence can be seen as projecting a wavefunction onto orthogonal subspaces. This can be neatly illustrated using polarized photons and the angle of the polarized sheet that will absorb or transmit them.

- Entangled Photons: Using our newly acquired understanding of photon polarizations, we see how to construct a quantum state of two photons in which, when you measure one of them, the person in the same world as you, will always find that the opposite photon has opposite quantum state. This is not because any influence is transmitted; it is just decoherence that takes place in a very symmetrical way, as can readily be observed in our calculations.

Many Worlds:

(At this point in the sequence, most of the mathematical background has been built up, and we are ready to evaluate interpretations of quantum mechanics.)

- Bell's Theorem: No EPR "Reality": (Note: This post was designed to be read as a stand-alone, if desired.) Originally, the discoverers of quantum physics thought they had discovered an incomplete description of reality - that there was some deeper physical process they were missing, and this was why they couldn't predict exactly the results of quantum experiments. The math of Bell's Theorem is surprisingly simple, and we walk through it. Bell's Theorem rules out being able to locally predict a single, unique outcome of measurements - ruling out a way that Einstein, Podolsky, and Rosen once defined "reality". This shows how deep implicit philosophical assumptions can go. If worlds can split, so that there is no single unique outcome, then Bell's Theorem is no problem. Bell's Theorem does, however, rule out the idea that quantum physics describes our partial knowledge of a deeper physical state that could locally produce single outcomes - any such description will be inconsistent.

- Spooky Action at a Distance: The No-Communication Theorem: As Einstein argued long ago, the quantum physics of his era - that is, the single-global-world interpretation of quantum physics, in which experiments have single unique random results - violates Special Relativity; it imposes a preferred space of simultaneity and requires a mysterious influence to be transmitted faster than light; which mysterious influence can never be used to transmit any useful information. Getting rid of the single global world dispels this mystery and puts everything back to normal again.

- Decoherence is Simple, Decoherence is Falsifiable and Testable: (Note: Designed to be standalone readable.) An epistle to the physicists. To probability theorists, words like "simple", "falsifiable", and "testable" have exact mathematical meanings, which are there for very strong reasons. The (minority?) faction of physicists who say that many-worlds is "not falsifiable" or that it "violates Occam's Razor" or that it is "untestable", are committing the same kind of mathematical crime as non-physicists who invent their own theories of gravity that go as inverse-cube. This is one of the reasons why I, a non-physicist, dared to talk about physics - because I saw (some!) physicists using probability theory in a way that was simply wrong. Not just criticizable, but outright mathematically wrong: 2 + 2 = 3.

- Quantum Non-Realism: "Shut up and calculate" is the best approach you can take when none of your theories are very good. But that is not the same as claiming that "Shut up!" actually is a theory of physics. Saying "I don't know what these equations mean, but they seem to work" is a very different matter from saying: "These equations definitely don't mean anything, they just work!"

- Collapse Postulates: Early physicists simply didn't think of the possibility of more than one world - it just didn't occur to them, even though it's the straightforward result of applying the quantum laws at all levels. So they accidentally invented a completely and strictly unnecessary part of quantum theory to ensure there was only one world - a law of physics that says that parts of the wavefunction mysteriously and spontaneously disappear when decoherence prevents us from seeing them any more. If such a law really existed, it would be the only non-linear, non-unitary, non-differentiable, non-local, non-CPT-symmetric, acausal, faster-than-light phenomenon in all of physics.

- If Many-Worlds Had Come First: If early physicists had never made the mistake, and thought immediately to apply the quantum laws at all levels to produce macroscopic decoherence, then "collapse postulates" would today seem like a completely crackpot theory. In addition to their other problems, like FTL, the collapse postulate would be the only physical law that was informally specified - often in dualistic (mentalistic) terms - because it was the only fundamental law adopted without precise evidence to nail it down. Here, we get a glimpse at that alternate Earth.

- Many Worlds, One Best Guess: Summarizes the arguments that nail down macroscopic decoherence, aka the "many-worlds interpretation". Concludes that many-worlds wins outright given the current state of evidence. The argument should have been over fifty years ago. New physical evidence could reopen it, but we have no particular reason to expect this.

- Living in Many Worlds: The many worlds of quantum mechanics are not some strange, alien universe into which you have been thrust. They are where you have always lived. Egan's Law: "It all adds up to normality." Then why care about quantum physics at all? Because there's still the question of what adds up to normality, and the answer to this question turns out to be, "Quantum physics." If you're thinking of building any strange philosophies around many-worlds, you probably shouldn't - that's not what it's for.

Timeless Physics:

(Now we depart from what is nailed down in standard physics, and enter into more speculative realms - particularly Julian Barbour's Machian timeless physics.)

- Mach's Principle: Anti-Epiphenomenal Physics: Could you tell if the whole universe were shifted an inch to the left? Could you tell if the whole universe was traveling left at ten miles per hour? Could you tell if the whole universe was accelerating left at ten miles per hour? Could you tell if the whole universe was rotating?

- Relative Configuration Space: Maybe the reason why we can't observe absolute speeds, absolute positions, absolute accelerations, or absolute rotations, is that particles don't have absolute positions - only positions relative to each other. That is, maybe quantum physics takes place in a relative configuration space.

- Timeless Physics: What time is it? How do you know? The question "What time is it right now?" may make around as much sense as asking "Where is the universe?" Not only that, our physics equations may not need a t in them!

- Timeless Beauty: To get rid of time you must reduce it to nontime. In timeless physics, everything that exists is perfectly global or perfectly local. The laws of physics are perfectly global; the configuration space is perfectly local. Every fundamentally existent ontological entity has a unique identity and a unique value. This beauty makes ugly theories much more visibly ugly; a collapse postulate becomes a visible scar on the perfection.

- Timeless Causality: Using the modern, Bayesian formulation of causality, we can define causality without talking about time - define it purely in terms of relations. The river of time never flows, but it has a direction.

- Timeless Identity: How can you be the same person tomorrow as today, in the river that never flows, when not a drop of water is shared between one time and another? Having used physics to completely trash all naive theories of identity, we reassemble a conception of persons and experiences from what is left. With a surprising practical application...

- Thou Art Physics: If the laws of physics control everything we do, then how can our choices be meaningful? Because you are physics. You aren't competing with physics for control of the universe, you are within physics. Anything you control is necessarily controlled by physics.

- Timeless Control: We throw away "time" but retain causality, and with it, the concepts "control" and "decide". To talk of something as having been "always determined" is mixing up a timeless and a timeful conclusion, with paradoxical results. When you take a perspective outside time, you have to be careful not to let your old, timeful intuitions run wild in the absence of their subject matter.

Rationality and Science:

(Okay, so it was many-worlds all along and collapse theories are silly. Did first-half-of-20th-century physicists really screw up that badly? How did they go wrong? Why haven't modern physicists unanimously endorsed many-worlds, if the issue is that clear-cut? What lessons can we learn from this whole debacle?)

- The Failures of Eld Science: A short story set in the same world as Initiation Ceremony. Future physics students look back on the cautionary tale of quantum physics.

- The Dilemma: Science or Bayes?: The failure of first-half-of-20th-century-physics was not due to straying from the scientific method. Science and rationality - that is, Science and Bayesianism - aren't the same thing, and sometimes they give different answers.

- Science Doesn't Trust Your Rationality: The reason Science doesn't always agree with the exact, Bayesian, rational answer, is that Science doesn't trust you to be rational. It wants you to go out and gather overwhelming experimental evidence.

- When Science Can't Help: If you have an idea, Science tells you to test it experimentally. If you spend 10 years testing the idea and the result comes out negative, Science slaps you on the back and says, "Better luck next time." If you want to spend 10 years testing a hypothesis that will actually turn out to be right, you'll have to try to do the thing that Science doesn't trust you to do: think rationally, and figure out the answer before you get clubbed over the head with it.

- Science Isn't Strict Enough: Science lets you believe any damn stupid idea that hasn't been refuted by experiment. Bayesianism says there is always an exactly rational degree of belief given your current evidence, and this does not shift a nanometer to the left or to the right depending on your whims. Science is a social freedom - we let people test whatever hypotheses they like, because we don't trust the village elders to decide in advance - but you shouldn't confuse that with an individual standard of rationality.

- Do Scientists Already Know This Stuff?: No. Maybe someday it will be part of standard scientific training, but for now, it's not, and the absence is visible.

- No Safe Defense, Not Even Science: Why am I trying to break your trust in Science? Because you can't think and trust at the same time. The social rules of Science are verbal rather than quantitative; it is possible to believe you are following them. With Bayesianism, it is never possible to do an exact calculation and get the exact rational answer that you know exists. You are visibly less than perfect, and so you will not be tempted to trust yourself.

- Changing the Definition of Science: Many of these ideas are surprisingly conventional, and being floated around by other thinkers. I'm a good deal less of a lonely iconoclast than I seem; maybe it's just the way I talk.

- Faster Than Science: Is it really possible to arrive at the truth faster than Science does? Not only is it possible, but the social process of science relies on scientists doing so - when they choose which hypotheses to test. In many answer spaces it's not possible to find the true hypothesis by accident. Science leaves it up to experiment to socially declare who was right, but if there weren't some people who could get it right in the absence of overwhelming experimental proof, science would be stuck.

- Einstein's Speed: Albert was unusually good at finding the right theory in the presence of only a small amount of experimental evidence. Even more unusually, he admitted it - he claimed to know the theory was right, even in advance of the public proof. It's possible to arrive at the truth by thinking great high-minded thoughts of the sort that Science does not trust you to think, but it's a lot harder than arriving at the truth in the presence of overwhelming evidence.

- That Alien Message: Einstein used evidence more efficiently than other physicists, but he was still extremely inefficient in an absolute sense. If a huge team of cryptographers and physicists were examining a interstellar transmission, going over it bit by bit, we could deduce principles on the order of Galilean gravity just from seeing one or two frames of a picture. As if the very first human to see an apple fall, had, on the instant, realized that its position went as the square of the time and that this implied constant acceleration.

- My Childhood Role Model: I looked up to the ideal of a Bayesian superintelligence, not Einstein.

- Einstein's Superpowers: There's an unfortunate tendency to talk as if Einstein had superpowers - as if, even before Einstein was famous, he had an inherent disposition to be Einstein - a potential as rare as his fame and as magical as his deeds. Yet the way you acquire superpowers is not by being born with them, but by seeing, with a sudden shock, that they are perfectly normal.

- Class Project: From the world of Initiation Ceremony. Brennan and the others are faced with their midterm exams.

- Why Quantum?: Why do a series on quantum mechanics? Some of the many morals that are best illustrated by the tale of quantum mechanics and its misinterpretation.

Sequence: Challenging the Difficult

The Failures of Eld Science

Something to Protect

Referenced by: Reversed Stupidity Is Not Intelligence • Quantum Explanations • Configurations and Amplitude • Joint Configurations • Distinct Configurations • Where Philosophy Meets Science • Can You Prove Two Particles Are Identical? • Classical Configuration Spaces • The Quantum Arena • Feynman Paths • No Individual Particles • Identity Isn't In Specific Atoms • Three Dialogues on Identity • Decoherence • The So-Called Heisenberg Uncertainty Principle • Which Basis Is More Fundamental? • Where Physics Meets Experience • Where Experience Confuses Physicists • On Being Decoherent • The Conscious Sorites Paradox • Decoherence is Pointless • Decoherent Essences • The Born Probabilities • Decoherence as Projection • Entangled Photons • Bell's Theorem: No EPR "Reality" • Spooky Action at a Distance: The No-Communication Theorem • Decoherence is Simple • Decoherence is Falsifiable and Testable • Quantum Non-Realism • Collapse Postulates • If Many-Worlds Had Come First • Many Worlds, One Best Guess • Mach's Principle: Anti-Epiphenomenal Physics • Relative Configuration Space • Timeless Physics • Timeless Beauty • Timeless Causality • Timeless Identity • Why Quantum? • Living in Many Worlds • Thou Art Physics • Timeless Control • An Intuitive Explanation of Quantum Mechanics • Quantum Physics Revealed As Non-Mysterious • And the Winner is... Many-Worlds! • Quantum Mechanics and Personal Identity • Heading Toward Morality • Setting Up Metaethics • Dreams of Friendliness • For The People Who Are Still Alive • ...And Say No More Of It • The Sacred Mundane • Ingredients of Timeless Decision Theory

Original with comments: The Quantum Physics Sequence

An Intuitive Explanation of Quantum Mechanics

Eliezer Yudkowsky, 12 June 2008 03:45AM

This is one of several shortened indices into the Quantum Physics Sequence. It is intended for students who are having trouble grasping the meaning of quantum math; or for people who want to learn the simple math of everything and are getting around to quantum mechanics.

There's a widespread belief that quantum mechanics is supposed to be confusing. This is not a good frame of mind for either a teacher or a student. Complicated math can be difficult but it is never, ever allowed to be confusing.

And I find that legendarily "confusing" subjects often are not really all that complicated as math, particularly if you just want a very basic - but still mathematical - grasp on what goes on down there.

This series takes you as far into quantum mechanics as you can go with only algebra. Any further and you should get a real physics textbook - once you know what all the math means.

- Quantum Explanations: Quantum mechanics doesn't deserve its fearsome reputation. If you tell people something is supposed to be mysterious, they won't understand it. It's human intuitions that are "strange" or "weird"; physics itself is perfectly normal. Talking about historical erroneous concepts like "particles" or "waves" is just asking to confuse people; present the real, unified quantum physics straight out. The series will take a strictly realist perspective - quantum equations describe something that is real and out there.

- Configurations and Amplitude: A preliminary glimpse at the stuff reality is made of. The classic split-photon experiment with half-silvered mirrors. Alternative pathways the photon can take, can cancel each other out. The mysterious measuring tool that tells us the relative squared moduli.

- Joint Configurations: The laws of physics are inherently over mathematical entities, configurations, that involve multiple particles. A basic, ontologically existent entity, according to our current understanding of quantum mechanics, does not look like a photon - it looks like a configuration of the universe with "A photon here, a photon there." Amplitude flows between these configurations can cancel or add; this gives us a way to detect which configurations are distinct.

- Distinct Configurations: Since configurations are over the combined state of all the elements in a system, adding a sensor that detects whether a particle went one way or the other, becomes a new element of the system that can make configurations "distinct" instead of "identical". This confused the living daylights out of early quantum experimenters, because it meant that things behaved differently when they tried to "measure" them. But it's not only measuring instruments that do the trick - any sensitive physical element will do. There is no need to suppose that the universe cares what we think.

- Can You Prove Two Particles Are Identical?: You wouldn't think that it would be possible to do an experiment that told you that two particles are completely identical - not just to the limit of experimental precision, but perfectly. But quantum mechanics makes it easy.

- Classical Configuration Spaces: How to visualize the state of a system of two 1-dimensional particles, as a single point in 2-dimensional space. A preliminary step before moving into...

- The Quantum Arena: Instead of a system state being associated with a single point in a classical configuration space, the instantaneous real state of a quantum system is a complex amplitude distribution over a quantum configuration space. What creates the illusion of "individual particles", like an electron caught in a trap, is a plaid distribution - one that happens to factor into the product of two parts. It is the whole distribution that evolves when a quantum system evolves. Individual configurations don't have physics; amplitude distributions have physics.

- Feynman Paths: Instead of thinking that a photon takes a single straight path through space, we can regard it as taking all possible paths through space, and adding the amplitudes for every possible path. Nearly all the paths cancel out - unless we do clever quantum things, so that some paths add instead of canceling out. Then we can make light do funny tricks for us, like reflecting off a mirror in such a way that the angle of incidence doesn't equal the angle of reflection. But ordinarily, nearly all the paths except an extremely narrow band, cancel out - this is one of the keys to recovering the hallucination of classical physics.

- No Individual Particles: One of the chief ways to confuse yourself while thinking about quantum mechanics, is to think as if photons were little billiard balls bouncing around.

- Decoherence: A quantum system that factorizes can evolve into a system that doesn't factorize, destroying the illusion of independence. But entangling a quantum system with its environment, can appear to destroy entanglements that are already present. Entanglement with the environment can separate out the pieces of an amplitude distribution, preventing them from interacting with each other.

- The So-Called Heisenberg Uncertainty Principle: Unlike classical physics, in quantum physics it is not possible to separate out a particle's "position" from its "momentum". The evolution of the amplitude distribution over time, involves things like taking the second derivative in space and multiplying by i to get the first derivative in time. The end result of this time evolution rule is that blobs of particle-presence appear to race around in physical space. The notion of "an exact particular momentum" or "an exact particular position" is not something that can physically happen, it is a tool for analyzing amplitude distributions by taking them apart into a sum of simpler waves. This uses the assumption and fact of linearity: the evolution of the whole wavefunction seems to always be the additive sum of the evolution of its pieces. Using this tool, we can see that if you take apart the same distribution into a sum of positions and a sum of momenta, they cannot both be sharply concentrated at the same time. When you "observe" a particle's position, that is, decohere its positional distribution by making it interact with a sensor, you take its wave packet apart into two pieces; then the two pieces evolve differently. The Heisenberg Principle definitely does not say that knowing about the particle, or consciously seeing it, will make the universe behave differently.

- On Being Decoherent: When a sensor measures a particle whose amplitude distribution stretches over space - perhaps seeing if the particle is to the left or right of some dividing line - then the standard laws of quantum mechanics call for the sensor+particle system to evolve into a state of (particle left, sensor measures LEFT) + (particle right, sensor measures RIGHT). But when we humans look at the sensor, it only seems to say "LEFT" or "RIGHT", never a mixture like "LIGFT". This, of course, is because we ourselves are made of particles, and subject to the standard quantum laws that imply decoherence. Under standard quantum laws, the final state is (particle left, sensor measures LEFT, human sees "LEFT") + (particle right, sensor measures RIGHT, human sees "RIGHT").

- Decoherece is Pointless: There is no exact point at which decoherence suddenly happens. All of quantum mechanics is continuous and differentiable, and decoherent processes are no exception to this.

- The Born Probabilities: The last serious mysterious question left in quantum physics: When a quantum world splits in two, why do we seem to have a greater probability of ending up in the larger blob, exactly proportional to the integral of the squared modulus? It's an open problem, but non-mysterious answers have been proposed. Try not to go funny in the head about it.

- Decoherence as Projection: Since quantum evolution is linear and unitary, decoherence can be seen as projecting a wavefunction onto orthogonal subspaces. This can be neatly illustrated using polarized photons and the angle of the polarized sheet that will absorb or transmit them.

- Entangled Photons: Using our newly acquired understanding of photon polarizations, we see how to construct a quantum state of two photons in which, when you measure one of them, the person in the same world as you, will always find that the opposite photon has opposite quantum state. This is not because any influence is transmitted; it is just decoherence that takes place in a very symmetrical way, as can readily be observed in our calculations.

- Bell's Theorem: No EPR "Reality": Originally, the discoverers of quantum physics thought they had discovered an incomplete description of reality - that there was some deeper physical process they were missing, and this was why they couldn't predict exactly the results of quantum experiments. But Bell's Theorem rules out any local deterministic physics with single unique outcomes. The math of Bell's Theorem is surprisingly simple, and we walk through it.

Referenced by: The Quantum Physics Sequence

Original with comments: An Intuitive Explanation of Quantum Mechanics

Quantum Physics Revealed As Non-Mysterious

Eliezer Yudkowsky, 12 June 2008 05:20AM

This is one of several shortened indices into the Quantum Physics Sequence.

Hello! You may have been directed to this page because you said something along the lines of "Quantum physics shows that reality doesn't exist apart from our observation of it," or "Science has disproved the idea of an objective reality," or even just "Quantum physics is one of the great mysteries of modern science; no one understands how it works."

There was a time, roughly the first half-century after quantum physics was invented, when this was more or less true. Certainly, when quantum physics was just being discovered, scientists were very confused indeed! But time passed, and science moved on. If you're confused about a phenomenon, that's a fact about your own state of mind, not a fact about the phenomenon itself - there are mysterious questions, but not mysterious answers. Science eventually figured out what was going on, and why things looked so strange at first.

The series of posts indexed below will show you - not just tell you - what's really going on down there. To be honest, you're not going to be able to follow this if algebra scares you. But there won't be any calculus, either.

Some optional preliminaries you might want to read:

- Reductionism: We build models of the universe that have many different levels of description. But so far as anyone has been able to determine, the universe itself has only the single level of fundamental physics - reality doesn't explicitly compute protons, only quarks.

- Joy in the Merely Real: If you can't take joy in things that turn out to be explicable, you're going to set yourself up for eternal disappointment. Don't worry if quantum physics turns out to be normal.

And here's the main sequence:

- Quantum Explanations: Quantum mechanics doesn't deserve its fearsome reputation. If you tell people something is supposed to be mysterious, they won't understand it. It's human intuitions that are "strange" or "weird"; physics itself is perfectly normal. Talking about historical erroneous concepts like "particles" or "waves" is just asking to confuse people; present the real, unified quantum physics straight out.

- Configurations and Amplitude: A preliminary glimpse at the stuff reality is made of. The classic split-photon experiment with half-silvered mirrors. Alternative pathways the photon can take, can cancel each other out. The mysterious measuring tool that tells us the relative squared moduli.

- Joint Configurations: The laws of physics are inherently over mathematical entities, configurations, that involve multiple particles. A basic, ontologically existent entity, according to our current understanding of quantum mechanics, does not look like a photon - it looks like a configuration of the universe with "A photon here, a photon there." Amplitude flows between these configurations can cancel or add; this gives us a way to detect which configurations are distinct.

- Distinct Configurations: Since configurations are over the combined state of all the elements in a system, adding a sensor that detects whether a particle went one way or the other, becomes a new element of the system that can make configurations "distinct" instead of "identical". This confused the living daylights out of early quantum experimenters, because it meant that things behaved differently when they tried to "measure" them. But it's not only measuring instruments that do the trick - any sensitive physical element will do - and the distinctness of configurations is a physical fact, not a fact about our knowledge. There is no need to suppose that the universe cares what we think.

- Where Philosophy Meets Science: In retrospect, supposing that quantum physics had anything to do with consciousness was a big mistake. Could philosophers have told the physicists so? But we don't usually see philosophers sponsoring major advances in physics; why not?

- Classical Configuration Spaces: How to visualize the state of a system of two 1-dimensional particles, as a single point in 2-dimensional space. A preliminary step before moving into...

- The Quantum Arena: Instead of a system state being associated with a single point in a classical configuration space, the instantaneous real state of a quantum system is a complex amplitude distribution over a quantum configuration space. What creates the illusion of "individual particles", like an electron caught in a trap, is a plaid distribution - one that happens to factor into the product of two parts. It is the whole distribution that evolves when a quantum system evolves. Individual configurations don't have physics; amplitude distributions have physics.

- Feynman Paths: Instead of thinking that a photon takes a single straight path through space, we can regard it as taking all possible paths through space, and adding the amplitudes for every possible path. Nearly all the paths cancel out - unless we do clever quantum things, so that some paths add instead of canceling out. Then we can make light do funny tricks for us, like reflecting off a mirror in such a way that the angle of incidence doesn't equal the angle of reflection. But ordinarily, nearly all the paths except an extremely narrow band, cancel out - this is one of the keys to recovering the hallucination of classical physics.

- No Individual Particles: One of the chief ways to confuse yourself while thinking about quantum mechanics, is to think as if photons were little billiard balls bouncing around. The appearance of little billiard balls is a special case of a deeper level on which there are only multiparticle configurations and amplitude flows.

- Decoherence: A quantum system that factorizes can evolve into a system that doesn't factorize, destroying the illusion of independence. But entangling a quantum system with its environment, can appear to destroy entanglements that are already present. Entanglement with the environment can separate out the pieces of an amplitude distribution, preventing them from interacting with each other.

- The So-Called Heisenberg Uncertainty Principle: Unlike classical physics, in quantum physics it is not possible to separate out a particle's "position" from its "momentum". The evolution of the amplitude distribution over time, involves things like taking the second derivative in space and multiplying by i to get the first derivative in time. The end result of this time evolution rule is that blobs of particle-presence appear to race around in physical space. The notion of "an exact particular momentum" or "an exact particular position" is not something that can physically happen, it is a tool for analyzing amplitude distributions by taking them apart into a sum of simpler waves. This uses the assumption and fact of linearity: the evolution of the whole wavefunction seems to always be the additive sum of the evolution of its pieces. Using this tool, we can see that if you take apart the same distribution into a sum of positions and a sum of momenta, they cannot both be sharply concentrated at the same time. When you "observe" a particle's position, that is, decohere its positional distribution by making it interact with a sensor, you take its wave packet apart into two pieces; then the two pieces evolve differently. The Heisenberg Principle definitely does not say that knowing about the particle, or consciously seeing it, will make the universe behave differently.

- Where Physics Meets Experience: Meet the Ebborians, who reproduce by fission. The Ebborian brain is like a thick sheet of paper that splits down its thickness. They frequently experience dividing into two minds, and can talk to their other selves. It seems that their unified theory of physics is almost finished, and can answer every question, when one Ebborian asks: When exactly does one Ebborian become two people?

- Where Experience Confuses Physicists: It then turns out that the entire planet of Ebbore is splitting along a fourth-dimensional thickness, duplicating all the people within it. But why does the apparent chance of "ending up" in one of those worlds, equal the square of the fourth-dimensional thickness? Many mysterious answers are proposed to this question, and one non-mysterious one.

- On Being Decoherent: When a sensor measures a particle whose amplitude distribution stretches over space - perhaps seeing if the particle is to the left or right of some dividing line - then the standard laws of quantum mechanics call for the sensor+particle system to evolve into a state of (particle left, sensor measures LEFT) + (particle right, sensor measures RIGHT). But when we humans look at the sensor, it only seems to say "LEFT" or "RIGHT", never a mixture like "LIGFT". This, of course, is because we ourselves are made of particles, and subject to the standard quantum laws that imply decoherence. Under standard quantum laws, the final state is (particle left, sensor measures LEFT, human sees "LEFT") + (particle right, sensor measures RIGHT, human sees "RIGHT").

- The Conscious Sorites Paradox: Decoherence is implicit in quantum physics, not an extra law on top of it. Asking exactly when "one world" splits into "two worlds" may be like asking when, if you keep removing grains of sand from a pile, it stops being a "heap". Even if you're inside the world, there may not be a definite answer. This puzzle does not arise only in quantum physics; the Ebborians could face it in a classical universe, or we could build sentient flat computers that split down their thickness.

- Decoherent Essences: Decoherence is implicit within physics, not an extra law on top of it. You can choose representations that make decoherence harder to see, just like you can choose representations that make apples harder to see, but exactly the same physical process still goes on; the apple doesn't disappear and neither does decoherence. If you could make decoherence magically go away by choosing the right representation, we wouldn't need to shield quantum computers from the environment.

- The Born Probabilities: The last serious mysterious question left in quantum physics: When a quantum world splits in two, why do we seem to have a greater probability of ending up in the larger blob, exactly proportional to the integral of the squared modulus? It's an open problem, but non-mysterious answers have been proposed. Try not to go funny in the head about it.

- Decoherence as Projection: Since quantum evolution is linear and unitary, decoherence can be seen as projecting a wavefunction onto orthogonal subspaces. This can be neatly illustrated using polarized photons and the angle of the polarized sheet that will absorb or transmit them.

- Entangled Photons: Using our newly acquired understanding of photon polarizations, we see how to construct a quantum state of two photons in which, when you measure one of them, the person in the same world as you, will always find that the opposite photon has opposite quantum state. This is not because any influence is transmitted; it is just decoherence that takes place in a very symmetrical way, as can readily be observed in our calculations.

- Bell's Theorem: No EPR "Reality": (Note: This post was designed to be read as a stand-alone, if desired.) Originally, the discoverers of quantum physics thought they had discovered an incomplete description of reality - that there was some deeper physical process they were missing, and this was why they couldn't predict exactly the results of quantum experiments. The math of Bell's Theorem is surprisingly simple, and we walk through it. Bell's Theorem rules out being able to locally predict a single, unique outcome of measurements - ruling out a way that Einstein, Podolsky, and Rosen once defined "reality". This shows how deep implicit philosophical assumptions can go. If worlds can split, so that there is no single unique outcome, then Bell's Theorem is no problem. Bell's Theorem does, however, rule out the idea that quantum physics describes our partial knowledge of a deeper physical state that could locally produce single outcomes - any such description will be inconsistent.

- Spooky Action at a Distance: The No-Communication Theorem: As Einstein argued long ago, the quantum physics of his era - that is, the single-global-world interpretation of quantum physics, in which experiments have single unique random results - violates Special Relativity; it imposes a preferred space of simultaneity and requires a mysterious influence to be transmitted faster than light; which mysterious influence can never be used to transmit any useful information. Getting rid of the single global world dispels this mystery and puts everything back to normal again.

- Quantum Non-Realism: "Shut up and calculate" is the best approach you can take when none of your theories are very good. But that is not the same as claiming that "Shut up!" actually is a theory of physics. Saying "I don't know what these equations mean, but they seem to work" is a very different matter from saying: "These equations definitely don't mean anything, they just work!"

- Collapse Postulates: Early physicists simply didn't think of the possibility of more than one world - it just didn't occur to them, even though it's the straightforward result of applying the quantum laws at all levels. So they accidentally invented a completely and strictly unnecessary part of quantum theory to ensure there was only one world - a law of physics that says that parts of the wavefunction mysteriously and spontaneously disappear when decoherence prevents us from seeing them any more. If such a law really existed, it would be the only non-linear, non-unitary, non-differentiable, non-local, non-CPT-symmetric, acausal, faster-than-light phenomenon in all of physics.

- Many Worlds, One Best Guess: Summarizes the arguments that nail down macroscopic decoherence, aka the "many-worlds interpretation". Concludes that many-worlds wins outright given the current state of evidence. The argument should have been over fifty years ago. New physical evidence could reopen it, but we have no particular reason to expect this.

- Living in Many Worlds: The many worlds of quantum mechanics are not some strange, alien universe into which you have been thrust. They are where you have always lived. Egan's Law: "It all adds up to normality." Then why care about quantum physics at all? Because there's still the question of what adds up to normality, and the answer to this question turns out to be, "Quantum physics." If you're thinking of building any strange philosophies around many-worlds, you probably shouldn't - that's not what it's for.

Referenced by: The Quantum Physics Sequence

Original with comments: Quantum Physics Revealed As Non-Mysterious

And the Winner is... Many-Worlds!

Eliezer Yudkowsky, 12 June 2008 06:05AM

This is one of several shortened indices into the Quantum Physics Sequence.

Macroscopic quantum superpositions, a.k.a. the "many-worlds interpretation" or MWI, was proposed in 1957 and brought to the general attention of the scientific community in 1970. Ever since, MWI has steadily gained in popularity. As of 2008, MWI may or may not be endorsed by a majority of theoretical physicists (attempted opinion polls conflict on this point). Of course, Science is not supposed to be an opinion poll, but anyone who tells you that MWI is "science fiction" is simply ignorant.

When a theory is slowly persuading scientists despite all academic inertia, and more and more graduate students grow up familiar with it, at what point should one go ahead and declare a temporary winner pending new evidence?

Reading through the referenced posts will give you a very basic introduction to quantum mechanics - algebra is involved, but no calculus - by which you may nonetheless gain an understanding sufficient to see, and not just be told, that the modern case for many-worlds has become overwhelming. Not just plausible, not just strong, but overwhelming. Single-world versions of quantum mechanics just don't work, and all the legendary confusingness and mysteriousness of quantum mechanics stems from this essential fact. But enough telling - let me show you.

- Quantum Explanations: Quantum mechanics doesn't deserve its fearsome reputation. If you tell people something is supposed to be mysterious, they won't understand it. It's human intuitions that are "strange" or "weird"; physics itself is perfectly normal. Talking about historical erroneous concepts like "particles" or "waves" is just asking to confuse people; present the real, unified quantum physics straight out. The series will take a strictly realist perspective - quantum equations describe something that is real and out there.

- Configurations and Amplitude: A preliminary glimpse at the stuff reality is made of. The classic split-photon experiment with half-silvered mirrors. Alternative pathways the photon can take, can cancel each other out. The mysterious measuring tool that tells us the relative squared moduli.

- Joint Configurations: The laws of physics are inherently over mathematical entities, configurations, that involve multiple particles. A basic, ontologically existent entity, according to our current understanding of quantum mechanics, does not look like a photon - it looks like a configuration of the universe with "A photon here, a photon there." Amplitude flows between these configurations can cancel or add; this gives us a way to detect which configurations are distinct.

- Distinct Configurations: Since configurations are over the combined state of all the elements in a system, adding a sensor that detects whether a particle went one way or the other, becomes a new element of the system that can make configurations "distinct" instead of "identical". This confused the living daylights out of early quantum experimenters, because it meant that things behaved differently when they tried to "measure" them. But it's not only measuring instruments that do the trick - any sensitive physical element will do - and the distinctness of configurations is a physical fact, not a fact about our knowledge. There is no need to suppose that the universe cares what we think.

- Where Philosophy Meets Science: In retrospect, supposing that quantum physics had anything to do with consciousness was a big mistake. Could philosophers have told the physicists so? But we don't usually see philosophers sponsoring major advances in physics; why not?

- Can You Prove Two Particles Are Identical?: You wouldn't think that it would be possible to do an experiment that told you that two particles are completely identical - not just to the limit of experimental precision, but perfectly. You could even give a precise-sounding philosophical argument for why it was not possible - but the argument would have a deeply buried assumption. Quantum physics violates this deep assumption, making the experiment easy.

- Classical Configuration Spaces: How to visualize the state of a system of two 1-dimensional particles, as a single point in 2-dimensional space. A preliminary step before moving into...

- The Quantum Arena: Instead of a system state being associated with a single point in a classical configuration space, the instantaneous real state of a quantum system is a complex amplitude distribution over a quantum configuration space. What creates the illusion of "individual particles", like an electron caught in a trap, is a plaid distribution - one that happens to factor into the product of two parts. It is the whole distribution that evolves when a quantum system evolves. Individual configurations don't have physics; amplitude distributions have physics. Quantum entanglement is the general case; quantum independence is the special case.

- Feynman Paths: Instead of thinking that a photon takes a single straight path through space, we can regard it as taking all possible paths through space, and adding the amplitudes for every possible path. Nearly all the paths cancel out - unless we do clever quantum things, so that some paths add instead of canceling out. Then we can make light do funny tricks for us, like reflecting off a mirror in such a way that the angle of incidence doesn't equal the angle of reflection. But ordinarily, nearly all the paths except an extremely narrow band, cancel out - this is one of the keys to recovering the hallucination of classical physics.

- No Individual Particles: One of the chief ways to confuse yourself while thinking about quantum mechanics, is to think as if photons were little billiard balls bouncing around. The appearance of little billiard balls is a special case of a deeper level on which there are only multiparticle configurations and amplitude flows. It is easy to set up physical situations in which there exists no fact of the matter as to which electron was originally which.

- Decoherence: A quantum system that factorizes can evolve into a system that doesn't factorize, destroying the illusion of independence. But entangling a quantum system with its environment, can appear to destroy entanglements that are already present. Entanglement with the environment can separate out the pieces of an amplitude distribution, preventing them from interacting with each other. Decoherence is fundamentally symmetric in time, but appears asymmetric because of the second law of thermodynamics.

- The So-Called Heisenberg Uncertainty Principle: Unlike classical physics, in quantum physics it is not possible to separate out a particle's "position" from its "momentum". The evolution of the amplitude distribution over time, involves things like taking the second derivative in space and multiplying by i to get the first derivative in time. The end result of this time evolution rule is that blobs of particle-presence appear to race around in physical space. The notion of "an exact particular momentum" or "an exact particular position" is not something that can physically happen, it is a tool for analyzing amplitude distributions by taking them apart into a sum of simpler waves. This uses the assumption and fact of linearity: the evolution of the whole wavefunction seems to always be the additive sum of the evolution of its pieces. Using this tool, we can see that if you take apart the same distribution into a sum of positions and a sum of momenta, they cannot both be sharply concentrated at the same time. When you "observe" a particle's position, that is, decohere its positional distribution by making it interact with a sensor, you take its wave packet apart into two pieces; then the two pieces evolve differently. The Heisenberg Principle definitely does not say that knowing about the particle, or consciously seeing it, will make the universe behave differently.

- Belief in the Implied Invisible: If a spaceship goes over the cosmological horizon relative to us, so that it can no longer communicate with us, should we believe that the spaceship instantly ceases to exist?

- Where Physics Meets Experience: Meet the Ebborians, who reproduce by fission. The Ebborian brain is like a thick sheet of paper that splits down its thickness. They frequently experience dividing into two minds, and can talk to their other selves. It seems that their unified theory of physics is almost finished, and can answer every question, when one Ebborian asks: When exactly does one Ebborian become two people?

- Where Experience Confuses Physicists: It then turns out that the entire planet of Ebbore is splitting along a fourth-dimensional thickness, duplicating all the people within it. But why does the apparent chance of "ending up" in one of those worlds, equal the square of the fourth-dimensional thickness? Many mysterious answers are proposed to this question, and one non-mysterious one.

- On Being Decoherent: When a sensor measures a particle whose amplitude distribution stretches over space - perhaps seeing if the particle is to the left or right of some dividing line - then the standard laws of quantum mechanics call for the sensor+particle system to evolve into a state of (particle left, sensor measures LEFT) + (particle right, sensor measures RIGHT). But when we humans look at the sensor, it only seems to say "LEFT" or "RIGHT", never a mixture like "LIGFT". This, of course, is because we ourselves are made of particles, and subject to the standard quantum laws that imply decoherence. Under standard quantum laws, the final state is (particle left, sensor measures LEFT, human sees "LEFT") + (particle right, sensor measures RIGHT, human sees "RIGHT").

- The Conscious Sorites Paradox: Decoherence is implicit in quantum physics, not an extra law on top of it. Asking exactly when "one world" splits into "two worlds" may be like asking when, if you keep removing grains of sand from a pile, it stops being a "heap". Even if you're inside the world, there may not be a definite answer. This puzzle does not arise only in quantum physics; the Ebborians could face it in a classical universe, or we could build sentient flat computers that split down their thickness. Is this really a physicist's problem?

- Decoherece is Pointless: There is no exact point at which decoherence suddenly happens. All of quantum mechanics is continuous and differentiable, and decoherent processes are no exception to this.

- Decoherent Essences: Decoherence is implicit within physics, not an extra law on top of it. You can choose representations that make decoherence harder to see, just like you can choose representations that make apples harder to see, but exactly the same physical process still goes on; the apple doesn't disappear and neither does decoherence. If you could make decoherence magically go away by choosing the right representation, we wouldn't need to shield quantum computers from the environment.

- The Born Probabilities: The last serious mysterious question left in quantum physics: When a quantum world splits in two, why do we seem to have a greater probability of ending up in the larger blob, exactly proportional to the integral of the squared modulus? It's an open problem, but non-mysterious answers have been proposed. Try not to go funny in the head about it.

- Decoherence as Projection: Since quantum evolution is linear and unitary, decoherence can be seen as projecting a wavefunction onto orthogonal subspaces. This can be neatly illustrated using polarized photons and the angle of the polarized sheet that will absorb or transmit them.

- Entangled Photons: Using our newly acquired understanding of photon polarizations, we see how to construct a quantum state of two photons in which, when you measure one of them, the person in the same world as you, will always find that the opposite photon has opposite quantum state. This is not because any influence is transmitted; it is just decoherence that takes place in a very symmetrical way, as can readily be observed in our calculations.

- Bell's Theorem: No EPR "Reality": (Note: This post was designed to be read as a stand-alone, if desired.) Originally, the discoverers of quantum physics thought they had discovered an incomplete description of reality - that there was some deeper physical process they were missing, and this was why they couldn't predict exactly the results of quantum experiments. The math of Bell's Theorem is surprisingly simple, and we walk through it. Bell's Theorem rules out being able to locally predict a single, unique outcome of measurements - ruling out a way that Einstein, Podolsky, and Rosen once defined "reality". This shows how deep implicit philosophical assumptions can go. If worlds can split, so that there is no single unique outcome, then Bell's Theorem is no problem. Bell's Theorem does, however, rule out the idea that quantum physics describes our partial knowledge of a deeper physical state that could locally produce single outcomes - any such description will be inconsistent.

- Spooky Action at a Distance: The No-Communication Theorem: As Einstein argued long ago, the quantum physics of his era - that is, the single-global-world interpretation of quantum physics, in which experiments have single unique random results - violates Special Relativity; it imposes a preferred space of simultaneity and requires a mysterious influence to be transmitted faster than light; which mysterious influence can never be used to transmit any useful information. Getting rid of the single global world dispels this mystery and puts everything back to normal again.

- Decoherence is Simple and

- Decoherence is Falsifiable and Testable: An epistle to the physicists. To probability theorists, words like "simple", "falsifiable", and "testable" have exact mathematical meanings, which are there for very strong reasons. The (minority?) faction of physicists who say that many-worlds is "not falsifiable" or that it "violates Occam's Razor" or that it is "untestable", are committing the same kind of mathematical crime as non-physicists who invent their own theories of gravity that go as inverse-cube. This is one of the reasons why I, a non-physicist, dared to talk about physics - because I saw (some!) physicists using probability theory in a way that was simply wrong. Not just criticizable, but outright mathematically wrong: 2 + 2 = 3.

- Quantum Non-Realism: "Shut up and calculate" is the best approach you can take when none of your theories are very good. But that is not the same as claiming that "Shut up!" actually is a theory of physics. Saying "I don't know what these equations mean, but they seem to work" is a very different matter from saying: "These equations definitely don't mean anything, they just work!"

- Collapse Postulates: Early physicists simply didn't think of the possibility of more than one world - it just didn't occur to them, even though it's the straightforward result of applying the quantum laws at all levels. So they accidentally invented a completely and strictly unnecessary part of quantum theory to ensure there was only one world - a law of physics that says that parts of the wavefunction mysteriously and spontaneously disappear when decoherence prevents us from seeing them any more. If such a law really existed, it would be the only non-linear, non-unitary, non-differentiable, non-local, non-CPT-symmetric, acausal, faster-than-light phenomenon in all of physics.

- If Many-Worlds Had Come First: If early physicists had never made the mistake, and thought immediately to apply the quantum laws at all levels to produce macroscopic decoherence, then "collapse postulates" would today seem like a completely crackpot theory. In addition to their other problems, like FTL, the collapse postulate would be the only physical law that was informally specified - often in dualistic (mentalistic) terms - because it was the only fundamental law adopted without precise evidence to nail it down. Here, we get a glimpse at that alternate Earth.

- Many Worlds, One Best Guess: Summarizes the arguments that nail down macroscopic decoherence, aka the "many-worlds interpretation". Concludes that many-worlds wins outright given the current state of evidence. The argument should have been over fifty years ago. New physical evidence could reopen it, but we have no particular reason to expect this.

- Living in Many Worlds: The many worlds of quantum mechanics are not some strange, alien universe into which you have been thrust. They are where you have always lived. Egan's Law: "It all adds up to normality." Then why care about quantum physics at all? Because there's still the question of what adds up to normality, and the answer to this question turns out to be, "Quantum physics." If you're thinking of building any strange philosophies around many-worlds, you probably shouldn't - that's not what it's for.

Referenced by: The Quantum Physics Sequence • Can Counterfactuals Be True? • Entangled Truths, Contagious Lies • Complexity and Intelligence • For The People Who Are Still Alive • This Failing Earth • The Correct Contrarian Cluster

Original with comments: And the Winner is... Many-Worlds!

Quantum Mechanics and Personal Identity

Eliezer Yudkowsky, 12 June 2008 07:13AM

This is one of several shortened indices into the Quantum Physics Sequence.

Suppose that someone built an exact duplicate of you on Mars, quark by quark - to the maximum level of resolution that quantum physics permits, which is considerably higher resolution than ordinary thermal uncertainty. Would the duplicate be really you, or just a copy?

It may seem unlikely a priori that physics, or any experimental science, could have something to say about this issue.

But it's amazing, the things that science can tell you.

In this case, it turns out, science can rule out a notion of personal identity that depends on your being composed of the same atoms - because modern physics has taken the concept of "same atom" and thrown it out the window. There are no tiny billiard balls with individual identities. It's experimentally ruled out.

"Huh? What do you mean, physics has gotten rid of the concept of 'same atom'?"

No one can be told this, alas, because it involves replacing the concept of little billiard balls with a different kind of math. You have to see it. If you read through the introduction that follows to basic quantum mechanics, you will be able to see that the naive concept of personal identity - the notion that you are made up of tiny pieces with individual identities that persist through time, and that your identity follows the "same" tiny pieces - is physical nonsense. The universe just doesn't work in a way which would let us make that interpretation.

There are more abstract and philosophical arguments that you could use to rule out atom-following theories of personal identity. But it so happens that we live in a universe where the issue is flatly settled by standard physics. It's like proposing that personal identity follows phlogiston. You could argue against it on philosophical grounds - but we happen to live in a universe where "phlogiston" itself is just a mistaken theory to be discarded, which settles the issue much more abruptly.

And no, this does not rely on a woo-woo mysterian interpretation of quantum mechanics. The other purpose of this series of posts, was to demystify quantum mechanics and reveal it as non-mysterious. It just happens to be a fact that once you get to the non-mysterious version of quantum mechanics, you find that the reason why physics once looked mysterious, has to do with reality being made up of different stuff than little billiard balls. Complex amplitudes in configuration spaces, to be exact, though here I jump ahead of myself.

If you read all the way to the end, you will, I hope, gain an entirely new perspective on where your "identity" is located... once the little billiard balls are ruled out.

You will even be able to see, I hope, that if your brain were non-destructively frozen (e.g. by vitrification in liquid nitrogen); and a computer model of the synapses, neural states, and other brain behaviors were constructed a hundred years later; then it would preserve exactly everything about you that was preserved by going to sleep one night and waking up the next morning.

Mind you, my audacious claim is not that uploading preserves identity - this audacious claim has been made many times before. I am claiming that once you grasp modern physics, you can actually see this as obvious, even if it would not be obvious to someone thinking in terms of Newtonian billiard balls in classical physics. This is much more audacious, and I am well aware of how unlikely that sounds; but I believe that I have fully supported that claim... if you read all the way to the end.

- Reductionism: We build models of the universe that have many different levels of description. But so far as anyone has been able to determine, the universe itself has only the single level of fundamental physics - reality doesn't explicitly compute protons, only quarks.

- Zombies! Zombies?: Don't try to put your consciousness outside physics. Whatever makes you say "I think therefore I am", causes your lips to move; it is within the chains of cause and effect that produce our observed universe.

- The Generalized Anti-Zombie Principle: We can try to generalize the Anti-Zombie principle to show that anything which exerts only an infinitesimal influence on your brain, should not be able to make your consciousness go out like a light.

- Quantum Explanations: Quantum mechanics doesn't deserve its fearsome reputation. If you tell people something is supposed to be mysterious, they won't understand it. It's human intuitions that are "strange" or "weird"; physics itself is perfectly normal. Talking about historical erroneous concepts like "particles" or "waves" is just asking to confuse people; present the real, unified quantum physics straight out. The series will take a strictly realist perspective - quantum equations describe something that is real and out there. Warning: Although a large faction of physicists agrees with this, it is not universally accepted. Stronger warning: I am not even going to present non-realist viewpoints until later, because I think this is a major source of confusion.

- Configurations and Amplitude: A preliminary glimpse at the stuff reality is made of. The classic split-photon experiment with half-silvered mirrors. Alternative pathways the photon can take, can cancel each other out. The mysterious measuring tool that tells us the relative squared moduli.

- Joint Configurations: The laws of physics are inherently over mathematical entities, configurations, that involve multiple particles. A basic, ontologically existent entity, according to our current understanding of quantum mechanics, does not look like a photon - it looks like a configuration of the universe with "A photon here, a photon there." Amplitude flows between these configurations can cancel or add; this gives us a way to detect which configurations are distinct. It is an experimentally testable fact that "Photon 1 here, photon 2 there" is the same configuration as "Photon 2 here, photon 1 there".

- Distinct Configurations: Since configurations are over the combined state of all the elements in a system, adding a sensor that detects whether a particle went one way or the other, becomes a new element of the system that can make configurations "distinct" instead of "identical". This confused the living daylights out of early quantum experimenters, because it meant that things behaved differently when they tried to "measure" them. But it's not only measuring instruments that do the trick - any sensitive physical element will do.

- Where Philosophy Meets Science: In retrospect, supposing that quantum physics had anything to do with consciousness was a big mistake. Could philosophers have told the physicists so? But we don't usually see philosophers sponsoring major advances in physics; why not?

- Can You Prove Two Particles Are Identical?: You wouldn't think that it would be possible to do an experiment that told you that two particles are completely identical - not just to the limit of experimental precision, but perfectly. You could even give a precise-sounding philosophical argument for why it was not possible - but the argument would have a deeply buried assumption. Quantum physics violates this deep assumption, making the experiment easy.

- Classical Configuration Spaces: How to visualize the state of a system of two 1-dimensional particles, as a single point in 2-dimensional space. A preliminary step before moving into...

- The Quantum Arena: Instead of a system state being associated with a single point in a classical configuration space, the instantaneous real state of a quantum system is a complex amplitude distribution over a quantum configuration space. What creates the illusion of "individual particles", like an electron caught in a trap, is a plaid distribution - one that happens to factor into the product of two parts. It is the whole distribution that evolves when a quantum system evolves. Individual configurations don't have physics; amplitude distributions have physics. Quantum entanglement is the general case; quantum independence is the special case.

- Feynman Paths: Instead of thinking that a photon takes a single straight path through space, we can regard it as taking all possible paths through space, and adding the amplitudes for every possible path. Nearly all the paths cancel out - unless we do clever quantum things, so that some paths add instead of canceling out. Then we can make light do funny tricks for us, like reflecting off a mirror in such a way that the angle of incidence doesn't equal the angle of reflection. But ordinarily, nearly all the paths except an extremely narrow band, cancel out - this is one of the keys to recovering the hallucination of classical physics.

- No Individual Particles: One of the chief ways to confuse yourself while thinking about quantum mechanics, is to think as if photons were little billiard balls bouncing around. The appearance of little billiard balls is a special case of a deeper level on which there are only multiparticle configurations and amplitude flows. It is easy to set up physical situations in which there exists no fact of the matter as to which electron was originally which.

- Identity Isn't In Specific Atoms, Three Dialogues on Identity: Given that there's no such thing as "the same atom", whether you are "the same person" from one time to another can't possibly depend on whether you're made out of the same atoms.

- Decoherence: A quantum system that factorizes can evolve into a system that doesn't factorize, destroying the illusion of independence. But entangling a quantum system with its environment, can appear to destroy entanglements that are already present. Entanglement with the environment can separate out the pieces of an amplitude distribution, preventing them from interacting with each other. Decoherence is fundamentally symmetric in time, but appears asymmetric because of the second law of thermodynamics.

- The So-Called Heisenberg Uncertainty Principle: Unlike classical physics, in quantum physics it is not possible to separate out a particle's "position" from its "momentum". The evolution of the amplitude distribution over time, involves things like taking the second derivative in space and multiplying by i to get the first derivative in time. The end result of this time evolution rule is that blobs of particle-presence appear to race around in physical space. The notion of "an exact particular momentum" or "an exact particular position" is not something that can physically happen, it is a tool for analyzing amplitude distributions by taking them apart into a sum of simpler waves. This uses the assumption and fact of linearity: the evolution of the whole wavefunction seems to always be the additive sum of the evolution of its pieces. Using this tool, we can see that if you take apart the same distribution into a sum of positions and a sum of momenta, they cannot both be sharply concentrated at the same time. When you "observe" a particle's position, that is, decohere its positional distribution by making it interact with a sensor, you take its wave packet apart into two pieces; then the two pieces evolve differently. The Heisenberg Principle definitely does not say that knowing about the particle, or consciously seeing it, will make the universe behave differently.

- Where Physics Meets Experience: Meet the Ebborians, who reproduce by fission. The Ebborian brain is like a thick sheet of paper that splits down its thickness. They frequently experience dividing into two minds, and can talk to their other selves. It seems that their unified theory of physics is almost finished, and can answer every question, when one Ebborian asks: When exactly does one Ebborian become two people?

- Where Experience Confuses Physicists: It then turns out that the entire planet of Ebbore is splitting along a fourth-dimensional thickness, duplicating all the people within it. But why does the apparent chance of "ending up" in one of those worlds, equal the square of the fourth-dimensional thickness? Many mysterious answers are proposed to this question, and one non-mysterious one.

- On Being Decoherent: When a sensor measures a particle whose amplitude distribution stretches over space - perhaps seeing if the particle is to the left or right of some dividing line - then the standard laws of quantum mechanics call for the sensor+particle system to evolve into a state of (particle left, sensor measures LEFT) + (particle right, sensor measures RIGHT). But when we humans look at the sensor, it only seems to say "LEFT" or "RIGHT", never a mixture like "LIGFT". This, of course, is because we ourselves are made of particles, and subject to the standard quantum laws that imply decoherence. Under standard quantum laws, the final state is (particle left, sensor measures LEFT, human sees "LEFT") + (particle right, sensor measures RIGHT, human sees "RIGHT").

- The Conscious Sorites Paradox: Decoherence is implicit in quantum physics, not an extra law on top of it. Asking exactly when "one world" splits into "two worlds" may be like asking when, if you keep removing grains of sand from a pile, it stops being a "heap". Even if you're inside the world, there may not be a definite answer. This puzzle does not arise only in quantum physics; the Ebborians could face it in a classical universe, or we could build sentient flat computers that split down their thickness. Is this really a physicist's problem?

- Decoherent Essences: Decoherence is implicit within physics, not an extra law on top of it. You can choose representations that make decoherence harder to see, just like you can choose representations that make apples harder to see, but exactly the same physical process still goes on; the apple doesn't disappear and neither does decoherence. If you could make decoherence magically go away by choosing the right representation, we wouldn't need to shield quantum computers from the environment.

- The Born Probabilities: The last serious mysterious question left in quantum physics: When a quantum world splits in two, why do we seem to have a greater probability of ending up in the larger blob, exactly proportional to the integral of the squared modulus? It's an open problem, but non-mysterious answers have been proposed. Try not to go funny in the head about it.

- Large number of posts skipped here, see the main sequence for them.

- Mach's Principle: Anti-Epiphenomenal Physics: Could you tell if the whole universe were shifted an inch to the left? Could you tell if the whole universe was traveling left at ten miles per hour? Could you tell if the whole universe was accelerating left at ten miles per hour? Could you tell if the whole universe was rotating?

- Relative Configuration Space: Maybe the reason why we can't observe absolute speeds, absolute positions, absolute accelerations, or absolute rotations, is that particles don't have absolute positions - only positions relative to each other. That is, maybe quantum physics takes place in a relative configuration space.

- Timeless Physics: What time is it? How do you know? The question "What time is it right now?" may make around as much sense as asking "Where is the universe?" Not only that, our physics equations may not need a t in them!

- Timeless Beauty: To get rid of time you must reduce it to nontime. In timeless physics, everything that exists is perfectly global or perfectly local. The laws of physics are perfectly global; the configuration space is perfectly local. Every fundamentally existent ontological entity has a unique identity and a unique value. This beauty makes ugly theories much more visibly ugly; a collapse postulate becomes a visible scar on the perfection.

- Timeless Causality: Using the modern, Bayesian formulation of causality, we can define causality without talking about time - define it purely in terms of relations. The river of time never flows, but it has a direction.

- Timeless Identity: How can you be the same person tomorrow as today, in the river that never flows, when not a drop of water is shared between one time and another? Having used physics to completely trash all naive theories of identity, we reassemble a conception of persons and experiences from what is left. This is the point at which you become able to see that uploading a person preserves everything of importance about them.

- Thou Art Physics: If the laws of physics control everything we do, then how can our choices be meaningful? Because you are physics. You aren't competing with physics for control of the universe, you are within physics. Anything you control is necessarily controlled by physics.

- Timeless Control: We throw away "time" but retain causality, and with it, the concepts "control" and "decide". To talk of something as having been "always determined" is mixing up a timeless and a timeful conclusion, with paradoxical results. When you take a perspective outside time, you have to be careful not to let your old, timeful intuitions run wild in the absence of their subject matter.

- Living in Many Worlds: The many worlds of quantum mechanics are not some strange, alien universe into which you have been thrust. They are where you have always lived. Egan's Law: "It all adds up to normality." Then why care about quantum physics at all? Because there's still the question of what adds up to normality, and the answer to this question turns out to be, "Quantum physics." If you're thinking of building any strange philosophies around many-worlds, you probably shouldn't - that's not what it's for.

Referenced by: The Quantum Physics Sequence • You Only Live Twice • Changing Emotions • Striving to Accept

Original with comments: Quantum Mechanics and Personal Identity

Causality and Moral Responsibility

Eliezer Yudkowsky, 13 June 2008 08:34AM

Followup to: Thou Art Physics, Timeless Control, Hand vs. Fingers, Explaining vs. Explaining Away

I know (or could readily rediscover) how to build a binary adder from logic gates. If I can figure out how to make individual logic gates from Legos or ant trails or rolling ping-pong balls, then I can add two 32-bit unsigned integers using Legos or ant trails or ping-pong balls.

Someone who had no idea how I'd just done the trick, might accuse me of having created "artificial addition" rather than "real addition".

But once you see the essence, the structure that is addition, then you will automatically see addition whenever you see that structure. Legos, ant trails, or ping-pong balls.

Even if the system is - gasp!- deterministic, you will see a system that, lo and behold, deterministically adds numbers. Even if someone - gasp! - designed the system, you will see that it was designed to add numbers. Even if the system was - gasp!- caused, you will see that it was caused to add numbers.

Let's say that John is standing in front of an orphanage which is on fire, but not quite an inferno yet; trying to decide whether to run in and grab a baby or two. Let us suppose two slightly different versions of John - slightly different initial conditions. They both agonize. They both are torn between fear and duty. Both are tempted to run, and know how guilty they would feel, for the rest of their lives, if they ran. Both feel the call to save the children. And finally, in the end, John-1 runs away, and John-2 runs in and grabs a toddler, getting out moments before the flames consume the entranceway.

This, it seems to me, is the very essence of moral responsibility - in the one case, for a cowardly choice; in the other case, for a heroic one. And I don't see what difference it makes, if John's decision was physically deterministic given his initial conditions, or if John's decision was preplanned by some alien creator that built him out of carbon atoms, or even if - worst of all - there exists some set of understandable psychological factors that were the very substance of John and caused his decision.

Imagine yourself caught in an agonizing moral dilemma. If the burning orphanage doesn't work for you - if you wouldn't feel conflicted about that, one way or the other - then substitute something else. Maybe something where you weren't even sure what the "good" option was.

Maybe you're deciding whether to invest your money in a startup that seems like it might pay off 50-to-1, or donate it to your-favorite-Cause; if you invest, you might be able to donate later... but is that what really moves you, or do you just want to retain the possibility of fabulous wealth? Should you donate regularly now, to ensure that you keep your good-guy status later? And if so, how much?

I'm not proposing a general answer to this problem, just offering it as an example of something else that might count as a real moral dilemma, even if you wouldn't feel conflicted about a burning orphanage.

For me, the analogous painful dilemma might be how much time to spend on relatively easy and fun things that might help set up more AI researchers in the future - like writing about rationality - versus just forgetting about the outside world and trying to work strictly on AI.

Imagine yourself caught in an agonizing moral dilemma. If my examples don't work, make something up. Imagine having not yet made your decision. Imagine yourself not yet knowing which decision you will make. Imagine that you care, that you feel a weight of moral responsibility; so that it seems to you that, by this choice, you might condemn or redeem yourself.

Okay, now imagine that someone comes along and says, "You're a physically deterministic system."

I don't see how that makes the dilemma of the burning orphanage, or the ticking clock of AI, any less agonizing. I don't see how that diminishes the moral responsibility, at all. It just says that if you take a hundred identical copies of me, they will all make the same decision. But which decision will we all make? That will be determined by my agonizing, my weighing of duties, my self-doubts, and my final effort to be good. (This is the idea of timeless control: If the result is deterministic, it is still caused and controlled by that portion of the deterministic physics which is myself. To cry "determinism" is only to step outside Time and see that the control is lawful.) So, not yet knowing the output of the deterministic process that is myself, and being duty-bound to determine it as best I can, the weight of moral responsibility is no less.

Someone comes along and says, "An alien built you, and it built you to make a particular decision in this case, but I won't tell you what it is."

Imagine a zillion possible people, perhaps slight variants of me, floating in the Platonic space of computations. Ignore quantum mechanics for the moment, so that each possible variant of me comes to only one decision. (Perhaps we can approximate a true quantum human as a deterministic machine plus a prerecorded tape containing the results of quantum branches.) Then each of these computations must agonize, and must choose, and must determine their deterministic output as best they can. Now an alien reaches into this space, and plucks out one person, and instantiates them. How does this change anything about the moral responsibility that attaches to how this person made their choice, out there in Platonic space... if you see what I'm trying to get at here?

The alien can choose which mind design to make real, but that doesn't necessarily change the moral responsibility within the mind.

There are plenty of possible mind designs that wouldn't make agonizing moral decisions, and wouldn't be their own bearers of moral responsibility. There are mind designs that would just play back one decision like a tape recorder, without weighing alternatives or consequences, without evaluating duties or being tempted or making sacrifices. But if the mind design happens to be you... and you know your duties, but you don't yet know your decision... then surely, that is the substance of moral responsibility, if responsibility can be instantiated in anything real at all?

We could think of this as an extremely generalized, Generalized Anti-Zombie Principle: If you are going to talk about moral responsibility, it ought not to be affected by anything that plays no role in your brain. It shouldn't matter whether I came into existence as a result of natural selection, or whether an alien built me up from scratch five minutes ago, presuming that the result is physically identical. I, at least, regard myself as having moral responsibility. I am responsible here and now; not knowing my future decisions, I must determine them. What difference does the alien in my past make, if the past is screened off from my present?

Am I suggesting that if an alien had created Lenin, knowing that Lenin would enslave millions, then Lenin would still be a jerk? Yes, that's exactly what I'm suggesting. The alien would be a bigger jerk. But if we assume that Lenin made his decisions after the fashion of an ordinary human brain, and not by virtue of some alien mechanism seizing and overriding his decisions, then Lenin would still be exactly as much of a jerk as before.

And as for there being psychological factors that determine your decision - well, you've got to be something, and you're too big to be an atom. If you're going to talk about moral responsibility at all - and I do regard myself as responsible, when I confront my dilemmas - then you've got to be able to be something, and that-which-is-you must be able to do something that comes to a decision, while still being morally responsible.

Just like a calculator is adding, even though it adds deterministically, and even though it was designed to add, and even though it is made of quarks rather than tiny digits.

Referenced by: Possibility and Could-ness • The Ultimate Source • Passing the Recursive Buck • Ghosts in the Machine • Heading Toward Morality • Created Already In Motion • Nonsentient Optimizers • Ingredients of Timeless Decision Theory

Original with comments: Causality and Moral Responsibility

Possibility and Could-ness

Eliezer Yudkowsky, 14 June 2008 04:38AM

This post is part of the Solution to "Free Will".
Followup to: Dissolving the Question, Causality and Moral Responsibility

Planning out upcoming posts, it seems to me that I do, in fact, need to talk about the word could, as in, "But I could have decided not to rescue that toddler from the burning orphanage."

Otherwise, I will set out to talk about Friendly AI, one of these days, and someone will say: "But it's a machine; it can't make choices, because it couldn't have done anything other than what it did."

So let's talk about this word, "could". Can you play Rationalist's Taboo against it? Can you talk about "could" without using synonyms like "can" and "possible"?

Let's talk about this notion of "possibility". I can tell, to some degree, whether a world is actual or not actual; what does it mean for a word to be "possible"?

I know what it means for there to be "three" apples on a table. I can verify that experimentally, I know what state of the world corresponds it. What does it mean to say that there "could" have been four apples, or "could not" have been four apples? Can you tell me what state of the world corresponds to that, and how to verify it? Can you do it without saying "could" or "possible"?

I know what it means for you to rescue a toddler from the orphanage. What does it mean for you to could-have-not done it? Can you describe the corresponding state of the world without "could", "possible", "choose", "free", "will", "decide", "can", "able", or "alternative"?

One last chance to take a stab at it, if you want to work out the answer for yourself...

Some of the first Artificial Intelligence systems ever built, were trivially simple planners. You specify the initial state, and the goal state, and a set of actions that map states onto states; then you search for a series of actions that takes the initial state to the goal state.

Modern AI planners are a hell of a lot more sophisticated than this, but it's amazing how far you can get by understanding the simple math of everything. There are a number of simple, obvious strategies you can use on a problem like this. All of the simple strategies will fail on difficult problems; but you can take a course in AI if you want to talk about that part.

There's backward chaining: Searching back from the goal, to find a tree of states such that you know how to reach the goal from them. If you happen upon the initial state, you're done.

There's forward chaining: Searching forward from the start, to grow a tree of states such that you know how to reach them from the initial state. If you happen upon the goal state, you're done.

Or if you want a slightly less simple algorithm, you can start from both ends and meet in the middle.

Let's talk about the forward chaining algorithm for a moment.

Here, the strategy is to keep an ever-growing collection of states that you know how to reach from the START state, via some sequence of actions and (chains of) consequences. Call this collection the "reachable from START" states; or equivalently, label all the states in the collection "reachable from START". If this collection ever swallows the GOAL state - if the GOAL state is ever labeled "reachable from START" - you have a plan.

"Reachability" is a transitive property. If B is reachable from A, and C is reachable from B, then C is reachable from A. If you know how to drive from San Jose to San Francisco, and from San Francisco to Berkeley, then you know a way to drive from San Jose to Berkeley. (It may not be the shortest way, but you know a way.)

If you've ever looked over a game-problem and started collecting states you knew how to achieve - looked over a maze, and started collecting points you knew how to reach from START - then you know what "reachability" feels like. It feels like, "I can get there." You might or might not be able to get to the GOAL from San Francisco - but at least you know you can get to San Francisco.

You don't actually run out and drive to San Francisco. You'll wait, and see if you can figure out how to get from San Francisco to GOAL. But at least you could go to San Francisco any time you wanted to.

(Why would you want to go to San Francisco? If you figured out how to get from San Francisco to GOAL, of course!)

Human beings cannot search through millions of possibilities one after the other, like an AI algorithm. But - at least for now - we are often much more clever about which possibilities we do search.

One of the things we do that current planning algorithms don't do (well), is rule out large classes of states using abstract reasoning. For example, let's say that your goal (or current subgoal) calls for you to cover at least one of these boards using domino 2-tiles.

The black square is a missing cell; this leaves 24 cells to be covered with 12 dominos.

You might just dive into the problem, and start trying to cover the first board using dominos - discovering new classes of reachable states:

However, you will find after a while that you can't seem to reach a goal state. Should you move on to the second board, and explore the space of what's reachable there?

But I wouldn't bother with the second board either, if I were you. If you construct this coloring of the boards:

Then you can see that every domino has to cover one grey and one yellow square. And only the third board has equal numbers of grey and yellow squares. So no matter how clever you are with the first and second board, it can't be done.

With one fell swoop of creative abstract reasoning - we constructed the coloring, it was not given to us - we've cut down our search space by a factor of three. We've reasoned out that the reachable states involving dominos placed on the first and second board, will never include a goal state.

Naturally, one characteristic that rules out whole classes of states in the search space, is if you can prove that the state itself is physically impossible. If you're looking for a way to power your car without all that expensive gasoline, it might seem like a brilliant idea to have a collection of gears that would turn each other while also turning the car's wheels - a perpetual motion machine of the first type. But because it is a theorem that this is impossible in classical mechanics, we know that every clever thing we can do with classical gears will not suffice to build a perpetual motion machine. It is as impossible as covering the first board with classical dominos. So it would make more sense to concentrate on new battery technologies instead.

Surely, what is physically impossible cannot be "reachable"... right? I mean, you would think...

Oh, yeah... about that free will thing.

So your brain has a planning algorithm - not a deliberate algorithm that you learned in school, but an instinctive planning algorithm. For all the obvious reasons, this algorithm keeps track of which states have known paths from the start point. I've termed this label "reachable", but the way the algorithm feels from inside, is that it just feels like you can do it. Like you could go there any time you wanted.

And what about actions? They're primitively labeled as reachable; all other reachability is transitive from actions by consequences. You can throw a rock, and if you throw a rock it will break a window, therefore you can break a window. If you couldn't throw the rock, you wouldn't be able to break the window.

Don't try to understand this in terms of how it feels to "be able to" throw a rock. Think of it in terms of a simple AI planning algorithm. Of course the algorithm has to treat the primitive actions as primitively reachable. Otherwise it will have no planning space in which to search for paths through time.

And similarly, there's an internal algorithmic label for states that have been ruled out:

worldState.possible == 0

So when people hear that the world is deterministic, they translate that into: "All actions except one are impossible." This seems to contradict their feeling of being free to choose any action. The notion of physics following a single line, seems to contradict their perception of a space of possible plans to search through.

The representations in our cognitive algorithms do not feel like representations; they feel like the way the world is. If your mind constructs a search space of states that would result from the initial state given various actions, it will feel like the search space is out there, like there are certain possibilities.

We've previously discussed how probability is in the mind. If you are uncertain about whether a classical coin has landed heads or tails, that is a fact about your state of mind, not a property of the coin. The coin itself is either heads or tails. But people forget this, and think that coin.probability == 0.5, which is the Mind Projection Fallacy: treating properties of the mind as if they were properties of the external world.

So I doubt it will come as any surprise to my longer-abiding readers, if I say that possibility is also in the mind.

What concrete state of the world - which quarks in which positions - corresponds to "There are three apples on the table, and there could be four apples on the table"? Having trouble answering that? Next, say how that world-state is different from "There are three apples on the table, and there couldn't be four apples on the table." And then it's even more trouble, if you try to describe could-ness in a world in which there are no agents, just apples and tables. This is a Clue that could-ness and possibility are in your map, not directly in the territory.

What is could-ness, in a state of the world? What are can-ness and able-ness? They are what it feels like to have found a chain of actions which, if you output them, would lead from your current state to the could-state.

But do not say, "I could achieve X". Say rather, "I could reach state X by taking action Y, if I wanted". The key phrase is "if I wanted". I could eat that banana, if I wanted. I could step off that cliff there - if, for some reason, I wanted to.

Where does the wanting come from? Don't think in terms of what it feels like to want, or decide something; try thinking in terms of algorithms. For a search algorithm to output some particular action - choose - it must first carry out a process where it assumes many possible actions as having been taken, and extrapolates the consequences of those actions.

Perhaps this algorithm is "deterministic", if you stand outside Time to say it. But you can't write a decision algorithm that works by just directly outputting the only action it can possibly output. You can't save on computing power that way. The algorithm has to assume many different possible actions as having been taken, and extrapolate their consequences, and then choose an action whose consequences match the goal. (Or choose the action whose probabilistic consequences rank highest in the utility function, etc. And not all planning processes work by forward chaining, etc.)

You might imagine the decision algorithm as saying: "Suppose the output of this algorithm were action A, then state X would follow. Suppose the output of this algorithm were action B, then state Y would follow." This is the proper cashing-out of could, as in, "I could do either X or Y." Having computed this, the algorithm can only then conclude: "Y ranks above X in the Preference Ordering. The output of this algorithm is therefore B. Return B."

The algorithm, therefore, cannot produce an output without extrapolating the consequences of itself producing many different outputs. All but one of the outputs being considered is counterfactual; but which output is the factual one cannot be known to the algorithm until it has finished running.

A bit tangled, eh? No wonder humans get confused about "free will".

You could eat the banana, if you wanted. And you could jump off a cliff, if you wanted. These statements are both true, though you are rather more likely to want one than the other.

You could even flatly say, "I could jump off a cliff" and regard this as true - if you construe could-ness according to reachability, and count actions as primitively reachable. But this does not challenge deterministic physics; you will either end up wanting to jump, or not wanting to jump.

The statement, "I could jump off the cliff, if I chose to" is entirely compatible with "It is physically impossible that I will jump off that cliff". It need only be physically impossible for you to choose to jump off a cliff - not physically impossible for any simple reason, perhaps, just a complex fact about what your brain will and will not choose.

Defining things appropriately, you can even endorse both of the statements:

- "I could jump off the cliff" is true from my point-of-view

- "It is physically impossible for me to jump off the cliff" is true for all observers, including myself

How can this happen? If all of an agent's actions are primitive-reachable from that agent's point-of-view, but the agent's decision algorithm is so constituted as to never choose to jump off a cliff.

You could even say that "could" for an action is always defined relative to the agent who takes that action, in which case I can simultaneously make the following two statements:

- NonSuicidalGuy could jump off the cliff.

- It is impossible that NonSuicidalGuy will hit the ground.

If that sounds odd, well, no wonder people get confused about free will!

But you would have to be very careful to use a definition like that one consistently. "Could" has another closely related meaning in which it refers to the provision of at least a small amount of probability. This feels similar, because when you're evaluating actions that you haven't yet ruled out taking, then you will assign at least a small probability to actually taking those actions - otherwise you wouldn't be investigating them. Yet "I could have a heart attack at any time" and "I could have a heart attack any time I wanted to" are not the same usage of could, though they are confusingly similar.

You can only decide by going through an intermediate state where you do not yet know what you will decide. But the map is not the territory. It is not required that the laws of physics be random about that which you do not know. Indeed, if you were to decide randomly, then you could scarcely be said to be in "control". To determine your decision, you need to be in a lawful world.

It is not required that the lawfulness of reality be disrupted at that point, where there are several things you could do if you wanted to do them; but you do not yet know their consequences, or you have not finished evaluating the consequences; and so you do not yet know which thing you will choose to do.

A blank map does not correspond to a blank territory. Not even an agonizingly uncertain map corresponds to an agonizingly uncertain territory.

(Next in the free will solution sequence is "The Ultimate Source", dealing with the intuition that we have some chooser-faculty beyond any particular desire or reason. As always, the interested reader is advised to first consider this question on their own - why would it feel like we are more than the sum of our impulses?)

Referenced by: Timeless Control • The Ultimate Source • Passing the Recursive Buck • Grasping Slippery Things • Heading Toward Morality • Will As Thou Wilt • Setting Up Metaethics • The Meaning of Right • Three Fallacies of Teleology • Against Modal Logics • Dark Side Epistemology • Devil's Offers • Nonsentient Optimizers • Free to Optimize • Ingredients of Timeless Decision Theory

Original with comments: Possibility and Could-ness

The Ultimate Source

Eliezer Yudkowsky, 15 June 2008 09:01AM

This post is part of the Solution to "Free Will".
Followup to: Timeless Control, Possibility and Could-ness

Faced with a burning orphanage, you ponder your next action for long agonizing moments, uncertain of what you will do. Finally, the thought of a burning child overcomes your fear of fire, and you run into the building and haul out a toddler.

There's a strain of philosophy which says that this scenario is not sufficient for what they call "free will". It's not enough for your thoughts, your agonizing, your fear and your empathy, to finally give rise to a judgment. It's not enough to be the source of your decisions.

No, you have to be the ultimate source of your decisions. If anything else in your past, such as the initial condition of your brain, fully determined your decision, then clearly you did not.

But we already drew this diagram:

As previously discussed, the left-hand structure is preferred, even given deterministic physics, because it is more local; and because it is not possible to compute the Future without computing the Present as an intermediate.

So it is proper to say, "If-counterfactual the past changed and the present remained the same, the future would remain the same," but not to say, "If the past remained the same and the present changed, the future would remain the same."

Are you the true source of your decision to run into the burning orphanage? What if your parents once told you that it was right for people to help one another? What if it were the case that, if your parents hadn't told you so, you wouldn't have run into the burning orphanage? Doesn't that mean that your parents made the decision for you to run into the burning orphanage, rather than you?

On several grounds, no:

If it were counterfactually the case that your parents hadn't raised you to be good, then it would counterfactually be the case that a different person would stand in front of the burning orphanage. It would be a different person who arrived at a different decision. And how can you be anyone other than yourself? Your parents may have helped pluck you out of Platonic person-space to stand in front of the orphanage, but is that the same as controlling the decision of your point in Platonic person-space?

Or: If we imagine that your parents had raised you differently, and yet somehow, exactly the same brain had ended up standing in front of the orphanage, then the same action would have resulted. Your present self and brain, screens off the influence of your parents - this is true even if the past fully determines the future.

But above all: There is no single true cause of an event. Causality proceeds in directed acyclic networks. I see no good way, within the modern understanding of causality, to translate the idea that an event must have a single cause. Every asteroid large enough to reach Earth's surface could have prevented the assassination of John F. Kennedy, if it had been in the right place to strike Lee Harvey Oswald. There can be any number of prior events, which if they had counterfactually occurred differently, would have changed the present. After spending even a small amount of time working with the directed acyclic graphs of causality, the idea that a decision can only have a single true source, sounds just plain odd.

So there is no contradiction between "My decision caused me to run into the burning orphanage", "My upbringing caused me to run into the burning orphanage", "Natural selection built me in such fashion that I ran into the burning orphanage", and so on. Events have long causal histories, not single true causes.

Knowing the intuitions behind "free will", we can construct other intuition pumps. The feeling of freedom comes from the combination of not knowing which decision you'll make, and of having the options labeled as primitively reachable in your planning algorithm. So if we wanted to pump someone's intuition against the argument "Reading superhero comics as a child, is the true source of your decision to rescue those toddlers", we reply:

"But even if you visualize Batman running into the burning building, you might not immediately know which choice you'll make (standard source of feeling free); and you could still take either action if you wanted to (note correctly phrased counterfactual and appeal to primitive reachability). The comic-book authors didn't visualize this exact scenario or its exact consequences; they didn't agonize about it (they didn't run the decision algorithm you're running). So the comic-book authors did not make this decision for you. Though they may have contributed to it being you who stands before the burning orphanage and chooses, rather than someone else."

How could anyone possibly believe that they are the ultimate and only source of their actions? Do they think they have no past?

If we, for a moment, forget that we know all this that we know, we can see what a believer in "ultimate free will" might say to the comic-book argument: "Yes, I read comic books as a kid, but the comic books didn't reach into my brain and force me to run into the orphanage. Other people read comic books and don't become more heroic. I chose it."

Let's say that you're confronting some complicated moral dilemma that, unlike a burning orphanage, gives you some time to agonize - say, thirty minutes; that ought to be enough time.

You might find, looking over each factor one by one, that none of them seem perfectly decisive - to force a decision entirely on their own.

You might incorrectly conclude that if no one factor is decisive, all of them together can't be decisive, and that there's some extra perfectly decisive thing that is your free will.

Looking back on your decision to run into a burning orphanage, you might reason, "But I could have stayed out of that orphanage, if I'd needed to run into the building next door in order to prevent a nuclear war. Clearly, burning orphanages don't compel me to enter them. Therefore, I must have made an extra choice to allow my empathy with children to govern my actions. My nature does not command me, unless I choose to let it do so."

Well, yes, your empathy with children could have been overridden by your desire to prevent nuclear war, if (counterfactual) that had been at stake.

This is actually a hand-vs.-fingers confusion; all of the factors in your decision, plus the dynamics governing their combination, are your will. But if you don't realize this, then it will seem like no individual part of yourself has "control" of you, from which you will incorrectly conclude that there is something beyond their sum that is the ultimate source of control.

But this is like reasoning that if no single neuron in your brain could control your choice in spite of every other neuron, then all your neurons together must not control your choice either.

Whenever you reflect, and focus your whole attention down upon a single part of yourself, it will seem that the part does not make your decision, that it is not you, because the you-that-sees could choose to override it (it is a primitively reachable option). But when all of the parts of yourself that you see, and all the parts that you do not see, are added up together, they are you; they are even that which reflects upon itself.

So now we have the intuitions that:

- The sensation of the primitive reachability of actions, is incompatible with their physical determinism

- A decision can only have a single "true" source; what is determined by the past cannot be determined by the present

- If no single psychological factor you can see is perfectly responsible, then there must be an additional something that is perfectly responsible

- When you reflect upon any single factor of your decision, you see that you could override it, and this "you" is the extra additional something that is perfectly responsible

The combination of these intuitions has led philosophy into strange veins indeed.

I once saw one such vein described neatly in terms of "Author" control and "Author*" control, though I can't seem to find or look up the paper.

Consider the control that an Author has over the characters in their books. Say, the sort of control that I have over Brennan.

By an act of will, I can make Brennan decide to step off a cliff. I can also, by an act of will, control Brennan's inner nature; I can make him more or less heroic, empathic, kindly, wise, angry, or sorrowful. I can even make Brennan stupider, or smarter up to the limits of my own intelligence. I am entirely responsible for Brennan's past, both the good parts and the bad parts; I decided everything that would happen to him, over the course of his whole life.

So you might think that having Author-like control over ourselves - which we obviously don't - would at least be sufficient for free will.

But wait! Why did I decide that Brennan would decide to join the Bayesian Conspiracy? Well, it is in character for Brennan to do so, at that stage of his life. But if this had not been true of Brennan, I would have chosen a different character that would join the Bayesian Conspiracy, because I wanted to write about the beisutsukai. Could I have chosen not to want to write about the Bayesian Conspiracy?

To have Author* self-control is not only have control over your entire existence and past, but to have initially written your entire existence and past, without having been previously influenced by it - the way that I invented Brennan's life without having previously lived it. To choose yourself into existence this way, would be Author* control. (If I remember the paper correctly.)

Paradoxical? Yes, of course. The point of the paper was that Author* control is what would be required to be the "ultimate source of your own actions", the way some philosophers seemed to define it.

I don't see how you could manage Author* self-control even with a time machine.

I could write a story in which Jane went back in time and created herself from raw atoms using her knowledge of Artificial Intelligence, and then Jane oversaw and orchestrated her own entire childhood up to the point she went back in time. Within the story, Jane would have control over her existence and past - but not without having been "previously" influenced by them. And I, as an outside author, would have chosen which Jane went back in time and recreated herself. If I needed Jane to be a bartender, she would be one.

Even in the unlikely event that, in real life, it is possible to create closed timelike curves, and we find that a self-recreating Jane emerges from the time machine without benefit of human intervention, that Jane still would not have Author* control. She would not have written her own life without having been "previously" influenced by it. She might preserve her personality; but would she have originally created it? And you could stand outside time and look at the cycle, and ask, "Why is this cycle here?" The answer to that would presumably lie within the laws of physics, rather than Jane having written the laws of physics to create herself.

And you run into exactly the same trouble, if you try to have yourself be the sole ultimate Author* source of even a single particular decision made by you - which is to say it was decided by your beliefs, inculcated morals, evolved emotions, etc. - which is to say your brain calculated it - which is to say physics determined it. You can't have Author* control over one single decision, even with a time machine.

So a philosopher would say: Either we don't have free will, or free will doesn't require being the sole ultimate Author* source of your own decisions, QED.

I have a somewhat different perspective, and say: Your sensation of freely choosing, clearly does not provide you with trustworthy information to the effect that you are the 'ultimate and only source' of your own actions. This being the case, why attempt to interpret the sensation as having such a meaning, and then say that the sensation is false?

Surely, if we want to know which meaning to attach to a confusing sensation, we should ask why the sensation is there, and under what conditions it is present or absent.

Then I could say something like: "This sensation of freedom occurs when I believe that I can carry out, without interference, each of multiple actions, such that I do not yet know which of them I will take, but I am in the process of judging their consequences according to my emotions and morals."

This is a condition that can fail in the presence of jail cells, or a decision so overwhelmingly forced that I never perceived any uncertainty about it.

There - now my sensation of freedom indicates something coherent; and most of the time, I will have no reason to doubt the sensation's veracity. I have no problems about saying that I have "free will" appropriately defined; so long as I am out of jail, uncertain of my own future decision, and living in a lawful universe that gave me emotions and morals whose interaction determines my choices.

Certainly I do not "lack free will" if that means I am in jail, or never uncertain of my future decisions, or in a brain-state where my emotions and morals fail to determine my actions in the usual way.

Usually I don't talk about "free will" at all, of course! That would be asking for trouble - no, begging for trouble - since the other person doesn't know about my redefinition. The phrase means far too many things to far too many people, and you could make a good case for tossing it out the window.

But I generally prefer to reinterpret my sensations sensibly, as opposed to refuting a confused interpretation and then calling the sensation "false".

Referenced by: Possibility and Could-ness • Passing the Recursive Buck • Grasping Slippery Things • Ghosts in the Machine • Heading Toward Morality • No Universally Compelling Arguments • Will As Thou Wilt • Where Recursive Justification Hits Bottom • Rebelling Within Nature • Setting Up Metaethics • The Meaning of Right • Inseparably Right; or, Joy in the Merely Good • Lawful Creativity • Living By Your Own Strength • Nonsentient Optimizers • Free to Optimize • Value is Fragile • Formative Youth • Whining-Based Communities

Original with comments: The Ultimate Source

Passing the Recursive Buck

Eliezer Yudkowsky, 16 June 2008 04:50AM

Followup to: Artificial Addition, The Ultimate Source, Gödel, Escher, Bach: An Eternal Golden Braid

Yesterday, I talked about what happens when you look at your own mind, reflecting upon yourself, and search for the source of your own decisions.

Let's say you decided to run into a burning orphanage and save a young child. You look back on the decision and wonder: was your empathy with children, your ability to imagine what it would be like to be on fire, the decisive factor? Did it compel you to run into the orphanage?

No, you reason, because if you'd needed to prevent a nuclear weapon from going off in the building next door, you would have run to disarm the nuke, and let the orphanage burn. So a burning orphanage is not something that controls you directly. Your fear certainly didn't control you. And as for your duties, it seems like you could have ignored them (if you wanted to).

So if none of these parts of yourself that you focus upon, are of themselves decisive... then there must be some extra and additional thing that is decisive! And that, of course, would be this "you" thing that is looking over your thoughts from outside.

Imagine if human beings had a tiny bit more introspective ability than they have today, so that they could see a single neuron firing - but only one neuron at a time. We might even have the ability to modify the firing of this neuron. It would seem, then, like no individual neuron was in control of us, and that indeed, we had the power to control the neuron. It would seem we were in control of our neurons, not controlled by them. Whenever you look at a single neuron, it seems not to control you, that-which-is-looking...

So it might look like you were moved to run into the orphanage by your built-in empathy or your inculcated morals, and that this overcame your fear of fire. But really there was an additional you, beyond these emotions, which chose to give in to the good emotions rather than the bad ones. That's moral responsibility, innit?

But wait - how does this additional you decide to flow along with your empathy and not your fear? Is it programmed to always be good? Does it roll a die and do whatever the die says?

Ordinarily, this question is not asked. Once you say that you choose of your own "free will", you've explained the choice - drawn a causal arrow coming from a black box, which feels like an explanation. At this point, you're supposed to stop asking questions, not look inside the black box to figure out how it works.

But what if the one does ask the question, "How did I choose to go along with my empathy and duty, and not my fear and selfishness?"

In real life, this question probably doesn't have an answer. We are the sum of our parts, as a hand is its fingers, palm, and thumb. Empathy and duty overpowered fear and selfishness - that was the choice. It may be that no one factor was decisive, but all of them together are you just as much as you are your brain. You did not choose for heroic factors to overpower antiheroic ones; that overpowering was your choice. Or else where did the meta-choice to favor heroic factors come from? I don't think there would, in fact, have been a deliberation on the meta-choice, in which you actually pondered the consequences of accepting first-order emotions and duties. There probably would not have been a detailed philosophical exploration, as you stood in front of that burning orphanage.

But memory is malleable. So if you look back and ask "How did I choose that?" and try to actually answer with something beyond the "free will!" stopsign, your mind is liable to start generating a philosophical discussion of morality that never happened.

And then it will appear that no particular argument in the philosophical discussion is absolutely decisive, since you could (primitive reachable) have decided to ignore it.

Clearly, there's an extra additional you that decides which philosophical arguments deserve attention.

You see where this is going. If you don't see where this is going, then you haven't read Douglas Hofstadter's Gödel, Escher, Bach: An Eternal Golden Braid, which makes you incomplete as a human being.

The general antipattern at work might be called "Passing the Recursive Buck". It is closely related to Artificial Addition (your mind generates infinite lists of surface phenomena, using a compact process you can't see into) and Mysterious Answer. This antipattern happens when you try to look into a black box, fail, and explain the black box using another black box.

Passing the Recursive Buck is rarer than Mysterious Answer, because most people just stop on the first black box. (When was the last time you heard postulated an infinite hierarchy of Gods, none of which create themselves, as the answer to the First Cause?)

How do you stop a recursive buck from passing?

You use the counter-pattern: The Recursive Buck Stops Here.

But how do you apply this counter-pattern?

You use the recursive buck-stopping trick.

And what does it take to execute this trick?

Recursive buck stopping talent.

And how do you develop this talent?

Get a lot of practice stopping recursive bucks.

Ahem.

So, the first trick is learning to notice when you pass the buck.

"The Recursive Buck Stops Here" tells you that you shouldn't be trying to solve the puzzle of your black box, by looking for another black box inside it. To appeal to meta-free-will, or to say "Free will ordinal hierarchy!" is just another way of running away from the scary real problem, which is to look inside the damn box.

This pattern was on display in Causality and Moral Responsibility:

Even if the system is - gasp!- deterministic, you will see a system that, lo and behold, deterministically adds numbers. Even if someone - gasp! - designed the system, you will see that it was designed to add numbers. Even if the system was - gasp!- caused, you will see that it was caused to add numbers.

To stop passing the recursive buck, you must find the non-mysterious structure that simply is the buck.

Take the Cartesian homunculus. Light passes into your eyes, but how can you find the shape of an apple in the visual information? Is there a little person inside your head, watching the light on a screen, and pointing out the apples? But then does the little person have a metahomunculus inside their head? If you have the notion of a "visual cortex", and you know even a little about how specifically the visual cortex processes and reconstructs the transcoded retinal information, then you can see that there is no need for a meta-visual-cortex that looks at the first visual cortex. The information is being transformed into cognitively usable form right there in the neurons.

I've already given a deal of advice on how to notice black boxes.

And I've even given some advice on how to start looking inside.

But ultimately, each black box is its own scientific problem. There is no easy, comforting, safe procedure you follow to "look inside". They aren't all as straightforward as free will. My main meta-advice has to do with subtasks like recognizing the black box, not running away screaming into the night, and not stopping on a fake explanation.

Referenced by: Grasping Slippery Things • Ghosts in the Machine • Heading Toward Morality • Created Already In Motion • The Bedrock of Fairness • Where Recursive Justification Hits Bottom • The Gift We Give To Tomorrow • "Arbitrary" • Invisible Frameworks • Mirrors and Paintings • Value is Fragile

Original with comments: Passing the Recursive Buck

Grasping Slippery Things

Eliezer Yudkowsky, 17 June 2008 02:04AM

Followup to: Possibility and Could-ness, The Ultimate Source

Brandon Reinhart wrote:

I am "grunching." Responding to the questions posted without reading your answer. Then I'll read your answer and compare. I started reading your post on Friday and had to leave to attend a wedding before I had finished it, so I had a while to think about my answer.

Brandon, thanks for doing this. You've provided a valuable illustration of natural lines of thought. I hope you won't be offended if, for educational purposes, I dissect it in fine detail. This sort of dissection is a procedure I followed with Marcello to teach thinking about AI, so no malice is intended.

Can you talk about "could" without using synonyms like "can" and "possible"?

When we speak of "could" we speak of the set of realizable worlds [A'] that follows from an initial starting world A operated on by a set of physical laws f.

(Emphases added.)

I didn't list "realizable" explicitly as Tabooed, but it refers to the same concept as "could". Rationalist's Taboo isn't played against a word list, it's played against a concept list. The goal is to force yourself to reduce.

Because follows links two worlds, and the linkage is exactly what seems confusing, a word like "follows" is also dangerous.

Think of it as being like trying to pick up something very slippery. You have to prevent it from squeezing out of your hands. You have to prevent the mystery from scurrying away and finding a new dark corner to hide in, as soon as you flip on the lights.

So letting yourself use a word like "realizable", or even "follows", is giving your mind a tremendous opportunity to Pass the Recursive Buck - which anti-pattern, be it noted in fairness to Brandon, I hadn't yet posted on.

If I was doing this on my own, and I didn't know the solution yet, I would also be marking "initial", "starting", and "operated on". Not necessarily at the highest priority, but just in case they were hiding the source of the confusion. If I was being even more careful I would mark "physical laws" and "world".

So when we say "I could have turned left at the fork in the road." "Could" refers to the set of realizable worlds that follow from an initial starting world A in which we are faced with a fork in the road, given the set of physical laws. We are specifically identifying a sub-set of [A']: that of the worlds in which we turned left.

One of the anti-patterns I see often in Artificial Intelligence, and I believe it is also common in philosophy, is inventing a logic that takes as a primitive something that you need to reduce to pieces.

To your mind's eye, it seems like "could-ness" is a primitive feature of reality. There's a natural temptation to describe the properties that "could-ness" seems to have, and make lists of things that are "could" or "not-could". But this is, at best, a preliminary step toward reduction, and you should be aware that it is at best preliminary step.

The goal is to see inside could-ness, not to develop a modal logic to manipulate primitive could-ness.

But seeing inside is difficult; there is no safe method you know you can use to see inside.

And developing a modal logic seems like it's good for a publication, in philosophy. Or in AI, you manually preprogram a list of which things have could-ness, and then the program appears to reason about it. That's good for a publication too.

This does not preclude us from making mistakes in our use of could. One might say "I could have turned left, turned right, or started a nuclear war." The options "started a nuclear war" may simply not be within the set [A']. It wasn't physically realizable given all of the permutations that result from applying our physical laws to our starting world.

Your mind tends to bounce off the problem, and has to be constrained to face it - like your mind itself is the slippery thing that keeps squeezing out of your hands.

It tries to hide the mystery somewhere else, instead of taking it apart - draw a line to another black box, releasing the tension of trying to look inside the first black box.

In your mind's eye, it seems, you can see before you the many could-worlds that follow from one real world.

The real answer is to resolve a Mind Projection Fallacy; physics follows a single line, but your search system, in determining its best action, has to search through multiple options not knowing which it will make real, and all the options will be labeled as reachable in the search.

So, given that answer, you can see how talking about "physically realizable" and "permutations(?) that result from applying physical laws" is a bounce-off-the-problem, a mere-logic, that squeezes the same unpenetrated mystery into "realizable" and "permutations".

If our physical laws contain no method for implementing free will and no randomness, [A'] contains only the single world that results from applying the set of physical laws to A. If there is randomness or free will, [A'] contains a broader collection of worlds that result from applying physical laws to A...where the mechanisms of free will or randomness are built into the physical laws.

Including a "mechanism of free will" into the model is a perfect case of Passing the Recursive Buck.

Think of it from the perspective of Artificial Intelligence. Suppose you were writing a computer program that would, if it heard a burglar alarm, conclude that the house had probably been robbed. Then someone says, "If there's an earthquate, then you shouldn't conclude the house was robbed." This is a classic problem in Bayesian networks with a whole deep solution to it in terms of causal graphs and probability distributions... but suppose you didn't know that.

You might draw a diagram for your brilliant new Artificial General Intelligence design, that had a "logical reasoning unit" as one box, and then a "context-dependent exception applier" in another box with an arrow to the first box.

So you would have convinced yourself that your brilliant plan for building AGI included a "context-dependent exception applier" mechanism. And you would not discover Bayesian networks, because you would have prematurely marked the mystery as known.

I don't mean "worlds" in the quantum mechanics sense, but as a metaphor for resultant states after applying some number of physical permutations to the starting reality.

"Permutations"? That would be... something that results in several worlds, all of which have the could-property? But where does the permuting come from? How does only one of the could-worlds become real, if it is a matter of physics? After you ask these questions you realize that you're looking at the same problem as before, which means that saying "permutations" didn't help reduce it.

Why can a machine practice free will? If free will is possible for humans, then it is a set of properties or functions of the physical laws (described by them, contained by them in some way) and a machine might then implement them in whatever fashion a human brain does. Free will would not be a characteristic of A or [A'], but the process applied to A to reach a specific element of [A'].

Again, if you remember that the correct answer is "Forward search process that labels certain options as reachable before judging them and maximizing", you can see the Mind Projection Fallacy on display in trying to put the could-ness property into basic physics.

So...I think I successfully avoided using reference to "might" or "probable" or other synonyms and closely related words.

now I'll read your post to see if I'm going the wrong way.

Afterward, Brandon posted:

Hmm. I think I was working in the right direction, but your procedural analogy let you get closer to the moving parts. But I think "reachability" as you used it and "realizable" as I used it (or was thinking of it) seem to be working along similar lines.

I hate to have to put it this way, because it seems harsh: but it's important to realize that, no, this wasn't working in the right direction.

Again to be fair, Marcello and I used to generate raw material like this on paper - but it was clearly labeled as raw material; the point was to keep banging our heads on opaque mysteries of cognition, until a split opened up that helped reduce the problem to smaller pieces, or looking at the same mystery from a different angle helped us get a grasp on at least its surface.

Nonetheless: Free will is a Confusing Problem. It is a comparatively lesser Confusing Problem but it is still a Confusing Problem. Confusing Problems are not like the cheap damn problems that college students are taught to solve using safe prepackaged methods. They are not even like the Difficult Problems that mathematicians tackle without knowing how to solve them. Even the simplest Confusing Problem can send generations of high-g philosophers wailing into the abyss. This is not high school homework, this is beisutsukai monastery homework.

So you have got to be extremely careful. And hold yourself, not to "high standards", but to your best dream of perfection. Part of that is being very aware of how little progress you have made. Remember that one major reason why AIfolk and philosophers bounce off hard problems and create mere modal logics, is that they get a publication and the illusion of progress. They rewarded themselves too easily. If I sound harsh in my criticism, it's because I'm trying to correct a problem of too much mercy.

They overestimated how much progress they had made, and of what kind. That's why I'm not giving you credit for generating raw material that could be useful to you in pinning down the problem. If you'd said you were doing that, I would have given you credit.

I'm sure that some people have achieved insight by accident from their raw material, so that they moved from the illusion of progress to real progress. But that sort of thing cannot be left to accident. More often, the illusion of progress is fatal: your mind is happy, content, and no longer working on the difficult, scary, painful, opaque, not-sure-how-to-get-inside part of the mystery.

Generating lots of false starts and dissecting them is one methodology for working on an opaque problem. (Instantly deadly if you can't detect false starts, of course.) Yet be careful not to credit yourself too much for trying! Do not pay yourself for labor, only results! To run away from a problem, or bounce off it into easier problems, or to convince yourself you have solved it with a black box, is common. To stick to the truly difficult part of a difficult problem, is rare. But do not congratulate yourself too much for this difficult feat of rationality; it is only the ante you pay to sit down at the high-stakes table, not a victory.

The only sign-of-success, as distinguished from a sign-of-working-hard, is getting closer to the moving parts.

And when you are finally unconfused, of course all the black boxes you invented earlier, will seem in retrospect to have been "driving in the general direction" of the truth then revealed inside them. But the goal is reduction, and only this counts as success; driving in a general direction is easy by comparison.

So you must cultivate a sharp and particular awareness of confusion, and know that your raw material and false starts are only raw material and false starts - though it's not the sort of thing that funding agencies want to hear. Academia creates incentives against the necessary standard; you can only be harsh about your own progress, when you've just done something so spectacular that you can be sure people will smile at your downplaying and say, "What wonderful modesty!"

The ultimate slippery thing you must grasp firmly until you penetrate is your mind.

Referenced by: Ghosts in the Machine • Heading Toward Morality • Against Modal Logics

Original with comments: Grasping Slippery Things

Ghosts in the Machine

Eliezer Yudkowsky, 17 June 2008 11:29PM

Followup to: The Ultimate Source, Passing the Recursive Buck

People hear about Friendly AI and say - this is one of the top three initial reactions:

"Oh, you can try to tell the AI to be Friendly, but if the AI can modify its own source code, it'll just remove any constraints you try to place on it."

And where does that decision come from?

Does it enter from outside causality, rather than being an effect of a lawful chain of causes which started with the source code as originally written? Is the AI the Author* source of its own free will?

The application of Anti-Buck-Passing to Friendly AI goes far beyond just answering that first objection. A Friendly AI is not a selfish AI constrained by a special extra conscience module that overrides the AI's natural impulses and tells it what to do. You just build the conscience, and that is the AI. If you have a program that computes which decision the AI should make, you're done. The buck stops immediately.

At this point, I shall take a moment to quote some case studies from the Computer Stupidities site and Programming subtopic. (I am not linking to this, because it is a fearsome time-trap; you can Google if you dare.)

I tutored college students who were taking a computer programming course. A few of them didn't understand that computers are not sentient. More than one person used comments in their Pascal programs to put detailed explanations such as, "Now I need you to put these letters on the screen." I asked one of them what the deal was with those comments. The reply: "How else is the computer going to understand what I want it to do?" Apparently they would assume that since they couldn't make sense of Pascal, neither could the computer.

While in college, I used to tutor in the school's math lab. A student came in because his BASIC program would not run. He was taking a beginner course, and his assignment was to write a program that would calculate the recipe for oatmeal cookies, depending upon the number of people you're baking for. I looked at his program, and it went something like this:

10 Preheat oven to 350
20 Combine all ingredients in a large mixing bowl
30 Mix until smooth

An introductory programming student once asked me to look at his program and figure out why it was always churning out zeroes as the result of a simple computation. I looked at the program, and it was pretty obvious:

begin
read("Number of Apples", apples)
read("Number of Carrots", carrots)
read("Price for 1 Apple", a_price)
read("Price for 1 Carrot", c_price)
write("Total for Apples", a_total)
write("Total for Carrots", c_total)
write("Total", total)
total = a_total + c_total
a_total = apples * a_price
c_total = carrots * c_price
end

Me: "Well, your program can't print correct results before they're computed."
Him: "Huh? It's logical what the right solution is, and the computer should reorder the instructions the right way."

There's an instinctive way of imagining the scenario of "programming an AI". It maps onto a similar-seeming human endeavor: Telling a human being what to do. Like the "program" is giving instructions to a little ghost that sits inside the machine, which will look over your instructions and decide whether it likes them or not.

There is no ghost who looks over the instructions and decides how to follow them. The program is the AI.

That doesn't mean the ghost does anything you wish for, like a genie. It doesn't mean the ghost does everything you want the way you want it, like a slave of exceeding docility. It means your instruction is the only ghost that's there, at least at boot time.

This is one reason AI is much harder than people once instinctively imagined; you can't just tell the ghost what to do. You have to build the ghost from scratch, and everything that seems obvious to you, the ghost will not see unless you know how to make the ghost see it - which in turn is much harder than it seems, given the sheer obviousness to you. You can't just tell the ghost to see it, and say, "I told the ghost to see it, and it saw; that's cause and effect for you." You have to create that-which-sees from scratch.

If you don't know how to build something that seems to have some strange ineffable elements like, say, "decision-making", then you can't just shrug your shoulders and let the ghost's free will do the job. You're left forlorn and ghostless.

There's more to building a chess-playing program than building a really fast processor - so the AI will be really smart - and then typing at the command prompt "Make whatever chess moves you think are best." You might think that, since the programmers themselves are not very good chess-players, any advice they tried to give the electronic superbrain would just slow the ghost down. But there is no ghost. You see the problem.

And there isn't a simple spell you can perform to - poof! - summon a complete ghost into the machine. You can't say, "I summoned the ghost, and it appeared; that's cause and effect for you." (It doesn't work if you use the notion of "emergence" or "complexity" as a substitute for "summon", either.) You can't give an instruction to the CPU, "Be a good chessplayer!" You have to see inside the mystery of chess-playing thoughts, and structure the whole ghost from scratch.

No matter how common-sensical, no matter how logical, no matter how "obvious" or "right" or "self-evident" or "intelligent" something seems to you, it will not happen inside the ghost. Unless it happens at the end of a chain of cause and effect that began with the instructions that you had to decide on, plus any causal dependencies on sensory data that you built into the starting instructions.

This doesn't mean you program in every decision explicitly. Deep Blue was a far superior chessplayer than its programmers. Deep Blue made better chess moves than anything its makers could have explicitly programmed - but not because the programmers shrugged and left it up to the ghost. Deep Blue moved better than its programmers... at the end of a chain of cause and effect that began in the programmers' code and proceeded lawfully from there. Nothing happened just because it was so obviously a good move that Deep Blue's ghostly free will took over, without the code and its lawful consequences being involved.

If you try to wash your hands of constraining the AI, you aren't left with a free ghost like an emancipated slave. You are left with a heap of sand that no one has purified into silicon, shaped into a CPU and programmed to think.

Go ahead, try telling a computer chip "Do whatever you want!" See what happens? Nothing. Because you haven't constrained it to understand freedom.

All it takes is one single step that is so obvious, so logical, so self-evident that your mind just skips right over it, and you've left the path of the AI programmer. It takes an effort like the one I showed in Grasping Slippery Things to prevent your mind from doing this.

Referenced by: Heading Toward Morality • The Design Space of Minds-In-General • No Universally Compelling Arguments • The Opposite Sex • Is Morality Given? • Rebelling Within Nature • Nonsentient Optimizers

Original with comments: Ghosts in the Machine

LA-602 vs. RHIC Review

Eliezer Yudkowsky, 19 June 2008 10:00AM

LA-602: Ignition of the Atmosphere with Nuclear Bombs, a research report from the Manhattan Project, is to the best of my knowledge the first technical analysis ever conducted of an uncertain danger of a human-caused extinction catastrophe.

Previously, Teller and Konopinski had been assigned the task of disproving a crazy suggestion by Enrico Fermi that a fission chain reaction could ignite a thermonuclear reaction in deuterium - what we now know as an H-Bomb. Teller and Konopinski found that, contrary to their initial skepticism, the hydrogen bomb appeared possible.

Good for their rationality! Even though they started with the wrong conclusion on their bottom line, they were successfully forced away from it by arguments that could only support one answer.

Still, in retrospect, I think that the advice the future would give to the past, would be: Start by sitting down and saying, "We don't know if a hydrogen bomb is possible". Then list out the evidence and arguments; then at the end weigh it.

So the hydrogen bomb was possible. Teller then suggested that a hydrogen bomb might ignite a self-sustaining thermonuclear reaction in the nitrogen of Earth's atmosphere. This also appeared extremely unlikely at a first glance, but Teller and Konopinski and Marvin investigated, and wrote LA-602...

As I understand LA-602, the authors went through the math and concluded that there were several strong reasons to believe that nitrogen fusion could not be self-sustaining in the atmosphere: it would take huge energies to start the reaction at all; the reaction would lose radiation from its surface too fast to sustain the fusion temperature; and even if the fusion reaction did grow, the Compton effect would increase radiation losses with volume(?).

And we're still here; so the math, whatever it actually says, seems to have been right.

Note that the Manhattan scientists didn't always get their math right. The Castle Bravo nuclear test on March 1, 1954 produced 15 megatons instead of the expected 4-8 megatons due to an unconsidered additional nuclear reaction that took place in lithium-7. The resulting fallout contaminated fishing boats outside the declared danger zone; at least one person seems to have died.

But the LA-602 calculations were done with very conservative assumptions, and came out with plenty of safety margin. AFAICT (I am not a physicist) a Castle Bravo type oversight could not realistically have made the atmosphere ignite anyway, and if it did, it'd have gone right out, etc.

The last time I know of when a basic physical calculation with that much safety margin, and multiple angles of argument, turned out to be wrong anyway, was when Lord Kelvin showed from multiple angles of reasoning that the Earth could not possibly be so much as a hundred million years old.

LA-602 concludes:

"There remains the distinct possibility that some other less simple mode of burning may maintain itself in the atmosphere... the complexity of the argument and the absence of satisfactory experimental foundations makes further work on the subject highly desirable."

Decades after LA-602, another paper would be written to analyze an uncertain danger of human-created existential risk: The Review of Speculative "Disaster Scenarios" at RHIC.

The RHIC Review was written in response to suggestions that the Relativistic Heavy Ion Collider might create micro black holes or strangelets.

A B.Sc. thesis by Shameer Shah of MIT, Perception of Risk: Disaster Scenarios at Brookhaven, chronicles the story behind the RHIC Review:

The RHIC flap began when Walter Wagner wrote to Scientific American, speculating that the Brookhaven collider might create a "mini black hole". A reply letter by Frank Wilczek of the Institute for Advanced Studies labeled the mini-black-hole scenario as impossible, but also introduced a new possibility, negatively charged strangelets, which would convert normal matter into more strange matter. Wilczek considered this possibility slightly more plausible.

Then the media picked up the story.

Shameer Shah interviewed (on Nov 22, 2002) Robert Jaffe, Director of MIT's Center for Theoretical Physics, a pioneer in the theory of strange matter, and primary author of the RHIC Review.

According to Jaffe, even before the investigative committe was convened, "No scientist who understood the physics thought that this experiment posed the slightest threat to anybody." Then why have the committee in the first place? "It was an attempt to take seriously the fears of science that they don't understand." Wilczek was asked to serve on the committee "to pay the wages of his sin, since he's the one that started all this with his letter."

Between LA-602 and the RHIC Review there is quite a difference of presentation.

I mean, just look at the names:

LA-602: Ignition of the Atmosphere with Nuclear Bombs
Review of Speculative "Disaster Scenarios" at RHIC

See a difference?

LA-602 began life as a classified report, written by scientists for scientists. You're assumed to be familiar with the meaning of terms like Bremsstrahlung, which I had to look up. LA-602 does not begin by asserting any conclusions; the report walks through the calculations - at several points clearly labeling theoretical extrapolations and unexplored possibilities as such - and finally concludes that radiation losses make self-sustaining nitrogen fusion impossible-according-to-the-math, even under the most conservative assumptions.

The RHIC Review presents a nontechnical summary of its conclusions in six pages at the start, relegating the math and physics to eighteen pages of appendices.

LA-602 concluded, "There remains the distinct possibility that some other less simple mode of burning may maintain itself in the atmosphere..."

The RHIC Review concludes: "Our conclusion is that the candidate mechanisms for catastrophic scenarios at RHIC are firmly excluded by existing empirical evidence, compelling theoretical arguments, or both. Accordingly, we see no reason to delay the commissioning of RHIC on their account."

It is not obvious to my inexpert eyes that the assumptions in the RHIC Review are any more firm than those in LA-602 - they both seem very firm - but the two papers arise from rather different causes.

To put it bluntly, LA-602 was written by people curiously investigating whether a hydrogen bomb could ignite the atmosphere, and the RHIC Review is a work of public relations.

Now, it does seem - so far as I can tell - that it's pretty damned unlikely for a particle accelerator far less powerful than random cosmic rays to destroy Earth and/or the Universe.

But I don't feel any more certain of that after reading the RHIC Review than before I read it. I am not a physicist; but if I was a physicist, and I read a predigested paper like the RHIC Review instead of doing the whole analysis myself from scratch, I would be fundamentally trusting the rationality of the paper's authors. Even if I checked the math, I would still be trusting that the equations I saw were the right equations to check. I would be trusting that someone sat down, and looked for unpleasant contrary arguments with an open mind, and really honestly didn't find anything.

When I contrast LA-602 to the RHIC Review, well...

Don't get me wrong: I don't feel the smallest particle of real fear about particle accelerators. The basic cosmic-ray argument seems pretty convincing. Nature seems to have arranged for the calculations in this case to have some pretty large error margins. I, myself, am not going to worry about risks we can actually calculate to be tiny, when there are incalculable large-looking existential risks to soak up my concern.

But there is something else that I do worry about: The primary stake on the table with things like RHIC, is that it is going to get scientists into the habit of treating existential risk as a public relations issue, where ignorant technophobes say the risk exists, and the job of scientists is to interface with the public and explain to them that it does not.

Everyone knew, before the RHIC report was written, what answer it was supposed to produce. That is a very grave matter. Analysis is what you get when physicists sit down together and say, "Let us be curious," and walk through all the arguments they can think of, recording them as they go, and finally weigh them up and reach a conclusion. If this does not happen, no analysis has taken place.

The general rule of thumb I sometimes use, is that - because the expected utility of thought arises from the utility of what is being reasoned about - a single error in analyzing an existential risk, even if it "doesn't seem like it ought to change the conclusion", is worth at least one human life.

The RHIC Review is not written to the standard of care that would be appropriate if, after the RHIC Review was written, some higher authority went through the paper; and if a single argument in it was wrong, anywhere, whether or not it changed the conclusion, a hostage got shot. That's how to think about analyzing existential risks. That way, for each and every element of the analysis, you can find it in yourself to be a little uncertain about that element, even if it doesn't seem "like it could possibly change the conclusion"; uncertainty invokes curiosity.

The RHIC Review was produced by authors who were already sure that the RHIC couldn't destroy the Earth, the problem-at-hand was explaining this to the public. If the authors decided just by eyeballing the problem that the RHIC couldn't destroy the Earth, then the only actual analysis that took place was conducted in 5 seconds. Yes, it's a lopsided issue, but it seems that as a general matter of policy, any existential risk at all deserves a longer and truly curious analysis than that.

Though I don't really blame the RHIC Review's authors. No one ever told them that there was such a thing as existential risk, or that it raised the standards of analysis beyond what was usual in a scientific paper, or that rational analysis requires placing yourself into a state of genuine uncertainty about each individual element's exact value...

And the much greater reason I don't blame them, is that between the 1940s and today, society has developed a "Gotcha!" attitude toward risk.

You can't admit a single particle of uncertain danger if you want your science's funding to survive. These days you are not allowed to end by saying, "There remains the distinct possibility..." Because there is no debate you can have about tradeoffs between scientific progress and risk. If you get to the point where you're having a debate about tradeoffs, you've lost the debate. That's how the world stands, nowadays.

So no one can do serious analysis of existential risks anymore, because just by asking the question, you're threatening the funding of your whole field.

The number one lesson I take from this whole issue is that where human-caused uncertain existential dangers are concerned, the only way to get a real, serious, rational, fair, evenhanded assessment of the risks, in our modern environment,

Is if the whole project is classified, the paper is written for scientists without translation, and the public won't get to see the report for another fifty years.

This is the lesson of LA-602: Ignition of the Atmosphere with Nuclear Bombs and the Review of Speculative "Disaster Scenarios" at RHIC. Read them and weep.

Referenced by: Should We Ban Physics?

Original with comments: LA-602 vs. RHIC Review

Heading Toward Morality

Eliezer Yudkowsky, 20 June 2008 08:08AM

Followup to: Ghosts in the Machine, Fake Fake Utility Functions, Fake Utility Functions

As people were complaining before about not seeing where the quantum physics sequence was going, I shall go ahead and tell you where I'm heading now.

Having dissolved the confusion surrounding the word "could", the trajectory is now heading toward should.

In fact, I've been heading there for a while. Remember the whole sequence on fake utility functions? Back in... well... November 2007?

I sometimes think of there being a train that goes to the Friendly AI station; but it makes several stops before it gets there; and at each stop, a large fraction of the remaining passengers get off.

One of those stops is the one I spent a month leading up to in November 2007, the sequence chronicled in Fake Fake Utility Functions and concluded in Fake Utility Functions.

That's the stop where someone thinks of the One Great Moral Principle That Is All We Need To Give AIs.

To deliver that one warning, I had to go through all sorts of topics - which topics one might find useful even if not working on Friendly AI. I warned against Affective Death Spirals, which required recursing on the affect heuristic and halo effect, so that your good feeling about one particular moral principle wouldn't spiral out of control. I did that whole sequence on evolution; and discursed on the human ability to make almost any goal appear to support almost any policy; I went into evolutionary psychology to argue for why we shouldn't expect human terminal values to reduce to any simple principle, even happiness, explaining the concept of "expected utility" along the way...

...and talked about genies and more; but you can read the Fake Utility sequence for that.

So that's just the warning against trying to oversimplify human morality into One Great Moral Principle.

If you want to actually dissolve the confusion that surrounds the word "should" - which is the next stop on the train - then that takes a much longer introduction. Not just one November.

I went through the sequence on words and definitions so that I would be able to later say things like "The next project is to Taboo the word 'should' and replace it with its substance", or "Sorry, saying that morality is self-interest 'by definition' isn't going to cut it here".

And also the words-and-definitions sequence was the simplest example I knew to introduce the notion of How An Algorithm Feels From Inside, which is one of the great master keys to dissolving wrong questions. Though it seems to us that our cognitive representations are the very substance of the world, they have a character that comes from cognition and often cuts crosswise to a universe made of quarks. E.g. probability; if we are uncertain of a phenomenon, that is a fact about our state of mind, not an intrinsic character of the phenomenon.

Then the reductionism sequence: that a universe made only of quarks, does not mean that things of value are lost or even degraded to mundanity. And the notion of how the sum can seem unlike the parts, and yet be as much the parts as our hands are fingers.

Followed by a new example, one step up in difficulty from words and their seemingly intrinsic meanings: "Free will" and seemingly intrinsic could-ness.

But before that point, it was useful to introduce quantum physics. Not just to get to timeless physics and dissolve the "determinism" part of the "free will" confusion. But also, more fundamentally, to break belief in an intuitive universe that looks just like our brain's cognitive representations. And present examples of the dissolution of even such fundamental intuitions as those concerning personal identity. And to illustrate the idea that you are within physics, within causality, and that strange things will go wrong in your mind if ever you forget it.

Lately we have begun to approach the final precautions, with warnings against such notions as Author* control: every mind which computes a morality must do so within a chain of lawful causality, it cannot arise from the free will of a ghost in the machine.

And the warning against Passing the Recursive Buck to some meta-morality that is not itself computably specified, or some meta-morality that is chosen by a ghost without it being programmed in, or to a notion of "moral truth" just as confusing as "should" itself...

And the warning on the difficulty of grasping slippery things like "should" - demonstrating how very easy it will be to just invent another black box equivalent to should-ness, to sweep should-ness under a slightly different rug - or to bounce off into mere modal logics of primitive should-ness...

We aren't yet at the point where I can explain morality.

But I think - though I could be mistaken - that we are finally getting close to the final sequence.

And if you don't care about my goal of explanatorily transforming Friendly AI from a Confusing Problem into a merely Extremely Difficult Problem, then stick around anyway. I tend to go through interesting intermediates along my way.

It might seem like confronting "the nature of morality" from the perspective of Friendly AI is only asking for additional trouble.

Artificial Intelligence melts people's brains. Metamorality melts people's brains. Trying to think about AI and metamorality at the same time can cause people's brains to spontaneously combust and burn for years, emitting toxic smoke - don't laugh, I've seen it happen multiple times.

But the discipline imposed by Artificial Intelligence is this: you cannot escape into things that are "self-evident" or "obvious". That doesn't stop people from trying, but the programs don't work. Every thought has to be computed somehow, by transistors made of mere quarks, and not by moral self-evidence to some ghost in the machine.

If what you care about is rescuing children from burning orphanages, I don't think you will find many moral surprises here; my metamorality adds up to moral normality, as it should. You do not need to worry about metamorality when you are personally trying to rescue children from a burning orphanage. The point at which metamoral issues per se have high stakes in the real world, is when you try to compute morality in an AI standing in front of a burning orphanage.

Yet there is also a good deal of needless despair and misguided fear of science, stemming from notions such as, "Science tells us the universe is empty of morality". This is damage done by a confused metamorality that fails to add up to moral normality. For that I hope to write down a counterspell of understanding. Existential depression has always annoyed me; it is one of the world's most pointless forms of suffering.

Don't expect the final post on this topic to come tomorrow, but at least you know where we're heading.

Sequence: Metaethics

No Universally Compelling Arguments

Referenced by: Setting Up Metaethics

Original with comments: Heading Toward Morality

The Outside View's Domain

Eliezer Yudkowsky, 21 June 2008 03:17AM

Followup to: The Planning Fallacy

Plato's Phaedo:

"The state of sleep is opposed to the state of waking; and out of sleeping, waking is generated; and out of waking, sleeping; and the process of generation is in the one case falling asleep, and in the other waking up. Do you agree?"
"Quite."
"Then suppose that you analyze life and death to me in the same manner. Is not death opposed to life?"
"Yes."
"And they are generated one from the other?"
"Yes."
"What is generated from life?"
"Death."
"And what from death?"
"I can only say in answer - life."
"Then the living, whether things or persons, Cebes, are generated from the dead?"
"That is clear."
"Then our souls exist in the house of Hades."
"It seems so."

Now suppose that the foil in the dialogue had objected a bit more strongly, and also that Plato himself had known about the standard research on the Inside View vs. Outside View...

(As I disapprove of Plato's use of Socrates as his character mouthpiece, I shall let one of the characters be Plato; and the other... let's call him "Phaecrinon".)

Plato: "The state of sleep is opposed to the state of waking; and out of sleeping, waking is generated; and out of waking, sleeping; and the process of generation is in the one case falling asleep, and in the other waking up... Then suppose that you analyze life and death to me in the same manner."

Phaecrinon: "Why should I? They are different things."

Plato: "Oh, Phaecrinon, have you not heard what researchers have shown, that the outside view is a better predictor than the inside view? You come to me and point out the differences between life-death and wake-sleep, all so that you can avoid making the obvious generalization that you prefer to deny. Yet if we allow such reasoning as this, will not software project managers say, 'My project is different from yours, because I have better programmers'? And will not textbook authors say, "We are wiser than those other textbook authors, and therefore we will finish sooner'? Therefore you can see that to point out the similarities between things is superior, and to point out the differences between them, inferior."

Phaecrinon: "You say that your reasoning is like to the reasoning of Daniel Kahneman, yet it seems to me that they are importantly different. For Daniel Kahneman dealt with generalization over things that are almost quite as similar to each other, as different flippings of the same coin. Yet you deal with wholly different processes with different internal mechanisms, and try to generalize across one to the other."

Plato: "But Phaecrinon, now you only compound your error; for you have pointed out the difference between myself and Kahneman, where I have pointed out the similarity. And this is again inferior, by reason of the inferiority of the Inside View over the Outside View. You have only given me one more special reason why the Outside View should not apply to your particular case - all so that you can deny that our souls exist in the house of Hades."

Phaecrinon: "Yet Plato, if you propose indiscriminately to apply the Outside View to all things, how do you explain the ability of engineers to construct a new bridge that is unlike any other bridge, and calculate its properties in advance by computer simulation? How can the Wright Flyer fly, when all previous human-made flying machines had failed? How indeed can anything at all happen for the first time?"

Plato: "Perhaps sometimes things do happen for the first time, but this does not mean we can predict them."

Phaecrinon: "Ah, Plato, you do too little justice to engineers. Out of all the possible structures of metal and tubes and explosive fuel, very few such structures constitute a spaceship that will land on the Moon. To land on the Moon for the first time, then, human engineers must have known, in advance, which of many designs would have the exceedingly rare property of landing upon the Moon. And is this not the very activity that engineers perform - calculating questions in detail? Do not engineers take the Inside View with great success?"

Plato: "But they assume that each screw and part will behave just like it does on all the other times observed."

Phaecrinon: "That is so. Yet nonetheless they construct detailed internal models of exceeding complexity, and do not only collect the statistics of whole cases. This is the Inside View if anything is the Inside View; no one claims that Inside Views are generated purely from nothingness."

Plato: "Then I answer that when engineers have shown many times their ability to perform detailed calculations with success, we trust on future occasions that they will succeed similarly. We trust the Inside View when the Outside View tells us to trust the Inside View. But if this is not so, and there is not a past record of success with detailed calculations, then the Outside View is all that is left to us; and we should foresake all attempts at internal modeling, for they will only lead us astray."

Phaecrinon: "But now you have admitted that the notion of 'trust Outside View, distrust Inside View' has a limited domain of applicability, and we may as well restrict that domain further. Just as you try to seal off the successes of engineers from the Outside View, so too, I wish to seal off the failures of Greek philosophers from the Outside View. Specifically, the record of Greek philosophers does not inspire in me any confidence that the Outside View can be applied across processes with greatly different internal causal structures, like life-and-death versus sleeping-and-waking. Daniel Kahneman and his fellows, writing a textbook, encountered a challenge drawn from a structurally similar causal generator as many other cases of textbook-writing; subject to just the same sort of unforeseen delays. Likewise the students who failed to predict when they would finish their Christmas shopping; the task of Christmas shopping does not change so much from one Christmas to another. It would be another matter entirely to say, 'Each year I have finished my Christmas shopping one day before Christmas - therefore I expect to finish my textbook one day before my deadline.'"

Plato: "But this only sounds foolish, because we know from the Outside View that textbooks are delayed far longer than this. Perhaps if you had never written a textbook before, and neither had anyone else, 'one day before deadline' would be the most reasonable estimate."

Phaecrinon: "You would not allow me to predict in advance that textbook writing is more difficult than Christmas shopping?"

Plato: "No. For you have chosen this particular special plea, using your hindsight of the correct answer. If you had truly needed to write a textbook for the first time in history, you would have pled, 'No one can foresee driving delays and crowds in the store, but the work I do to write a textbook is all under my own control - therefore, I will finish more than one day before deadline.'"

Phaecrinon: "But even you admit that to draw analogies across wider and wider differences is to make those analogies less and less reliable. If you see many different humans sleeping, you can conclude that a newly observed human will probably sleep for eight hours; but if you see a cat sleeping, you must be less confident; and if you wish to draw an analogy to life and death, that is a greater distance still."

Plato: "If I allow that, will not software project managers say, 'My software project is as unlike to all other software projects as is a cat to a human?'"

Phaecrinon: "Then they are fools and nothing can be done about it. Surely you do not think that the prediction from many humans to one cat is just as strong as the prediction from many humans to one human? Insensitivity to the reliability of predictors is also a standard bias."

Plato: "That is true. Yet an Outside View may not be a good estimate, and yet still be the best estimate. If we have only seen the sleep cycles of many humans, then the Outside View on the whole group may be the best estimate for a newly observed cat, if you have no other data. Even likewise with our guesses as to life and death."

Phaecrinon: "And one sign of when the Outside View might not provide a good estimate, is when there are many different reference classes to which you might compare your new thing. A candle burns low, and exhausts itself and extinguishes, and does not light again the next day. How do you know that life and death is not analogous to a candle which burns and fails? Why not generalize over the similarity to a candle, rather than the similarity to sleep cycles?"

Plato: "Oh, but Phaecrinon, if we allow arguments over reference classes, we may as well toss the notion of an Outside View out the window. For then software project managers will say that the proper reference class for their project is the class of projects that delivered on time, or the class of projects with managers as wise as themselves. As for your analogy of the candle, it is self-evident that life is similar to sleeping and waking, not to candles. When a man is born, he is weak, but he grows to adulthood and is strong, and then with age he grows weaker again. In this he is like the Sun, that is weak when it rises, and strong at its apex, and then sinks below the horizon; in this a man is like the sleepy riser, who becomes sleepy again at the end of the day. It is self-evident that life and death belongs to the class of cyclical processes, and not the class of irreversible processes."

Phaecrinon: "What is self-evident to you does not seem so self-evident to me, Plato; and just because you call several widely different things 'cyclical processes', it does not follow that they were all random samples drawn from a great Barrel of Cyclical Processes, and that the next thing you choose to call a 'cyclical process' will have the same distribution of properties."

Plato: "Again you compound your mistake by pleading special exceptions. Will we let the software manager plead that his project is not drawn from the same barrel as the others?"

Phaecrinon: "Again you extend the Outside View beyond its domain of applicability. In engineering where all internal parts are precisely understood, but the whole is not quite similar to anything else that has been built before, then the Inside View is superior to the Outside View. And the sign of this is that results are routinely predicted in advance with great precision."

Plato: "This is just to say that when the Outside View tells us to use the Inside View, we should use it. But surely not otherwise, Phaecrinon!"

Phaecrinon: "When many different people try to accomplish the same task, and the internal details cannot be precisely calculated, and yet people have a tendency to optimism and to not visualize incidental catastrophes, then the Outside View is superior to the Inside View. And the sign of this is that the same kind of task - with the same sort of internal structure, the same difficulties and challenges - has been done many times, and the result cannot be predicted with precision; yet people's predictions are usually biased optimistically."

Plato: "This is the triumph of the Outside View!"

Phaecrinon: "But when you deal with attempted analogies across structually different processes, perhaps unique or poorly understood, then things which are similar in some surface respects are often different in other respects. And the sign of this domain is that when people try to reason by similarity, it is not at all clear what is similar to what, or which surface resemblances they should focus upon as opposed to others."

Plato: "I think the resemblance of life-death to sleep-waking is perfectly clear. But what do you assert we should do in such a case, if it is not taking the Outside View?"

Phaecrinon: "Perhaps there is nothing to be done at all, with either the Inside View or the Outside View. Not all problems are solvable; and it may be that the best we can do is avoid the overconfidence from asserting that analogies are much stronger than they are. But it seems to me that in those cases where we know something of the internal structure, then we can sometimes produce predictions by imagining the internals, even though the whole thing is not similar to any other whole thing in our experience."

Plato: "Now I challenge you to consider how well such thoughts have done, historically."

Phaecrinon: "What I have just described is the way that engineers build the first prototype of anything. But that, I admit, is when they understand very precisely the parts they use. If the internals are not well-understood, then the whole will in most cases be even less well-understood. It is only your idea that the Outside View can yield better predictions, that I am protesting against. It seems to me that the result of taking the Outside View of things poorly understood or structurally dissimilar to other things in the purported reference class, is only to create great disputes about definitional boundaries, and clashing analogies, and arguments over which surface similarities are important. When all along the new process may not be similar to anything that already exists."

Plato: "But there is no alternative to the Outside View."

Phaecrinon: "Yes, there is; you can try to imagine the internal process, if you know anything at all about it. At least then two people can focus on the internal structure and argue about what happens and their dispute will be commensurable. But if two people both say 'I am taking the Outside View' and then form different 'self-evident' reference classes, what do they do from there? How can they resolve their dispute about which surface characteristics are important? At least if you make predictions about internal causal processes, the results are finally testable if the dispute is empirical at all. How do you test the assertion that life is more importantly similar to sleep and waking, than to a candle? Perhaps life is simply like neither. Something must happen internally when a human thinks and reasons, but there does not have to exist any other process in nature similar enough that we could predict the characteristics of human thought by looking at it."

Plato: "What it boils down to, is that you are constructing a detailed excuse not to use the Outside View in your own case."

Phaecrinon: "And if each of two people with different Outside Views says to the other, 'You are a fool, for disregarding the Outside View!' then they will make no progress on their disagreement at all. This is the danger of proposing an absolute mandate for philosophers encountering new and structurally different phenomena, because you want to prevent software project managers from making special excuses for their software project. Reversed stupidity is not intelligence, and there is no language in which it is difficult to write bad computer programs, and in the art of rationality it is never difficult to shoot off your own foot if you desire to do so. The standard Outside View relies on your seeing the common-sense difference between textbook writing and Christmas shopping, so that you don't try to lump them into the same reference class. I am similarly hoping that you can see by common sense that the Outside View works rather better to predict Christmas shopping times, than what you are arguing is the analogous 'Outside View' technique in philosophy."

Plato: "And you believe you can do better with the Inside View."

Phaecrinon: "Reasoning about the internals of things whose output is not yet observed, is fraught with difficulty. One must be constantly aware of what one can and cannot reasonably guess, based on the strength of your knowledge. The uncertainties of such an Inside View, end up being much greater than the uncertainties of the Outside View on Christmas shopping. Only when the Inside View support appears extremely lopsided can you dare to come to even a tentative conclusion! But I do think that sometimes the Inside View support can be extremely lopsided - though it is a strain on your rationality even to correctly distinguish such cases."

Plato: "The evidence shows that people cannot successfully use the Inside View at all."

Phaecrinon: "No, the evidence shows that the Outside View yields better answers than the Inside View for problems like writing a textbook. But even an Inside View of writing a textbook would tell you that the project was unlikely to destroy the Earth. Taking the Inside View of a new and strange process is a Difficult Problem, where taking the Outside View on textbook composition is a Straightforward Problem. But to try and argue like alchemists from surface resemblances is a Hopeless Problem. Then there cannot even be any meeting of minds, if you start with different assumptions about which similarities are important. An answer need not exist even in principle, for there may be nothing else that is enough like this new thing to yield successful predictions by analogy."

Plato: "So you have said that it is easier for two people to conduct their dispute if they both take the Inside View and argue about internal causal processes. But from this it does not follow that the Outside View based on surface resemblances is inferior. Perhaps you are only coming to agreement on folly, and either of two conflicting Outside Views would be more reliable than the best Inside View."

Referenced by: Surface Analogies and Deep Causes • The Weak Inside View • "Outside View!" as Conversation-Halter

Original with comments: The Outside View's Domain

Surface Analogies and Deep Causes

Eliezer Yudkowsky, 22 June 2008 07:51AM

Followup to: Artificial Addition, The Outside View's Domain

Where did I acquire, in my childhood, the deep conviction that reasoning from surface similarity couldn't be trusted?

I don't know; I really don't. Maybe it was from S. I. Hayakawa's Language in Thought and Action, or even Van Vogt's similarly inspired Null-A novels. From there, perhaps, I began to mistrust reasoning that revolves around using the same word to label different things, and concluding they must be similar? Could that be the beginning of my great distrust of surface similarities? Maybe. Or maybe I tried to reverse stupidity of the sort found in Plato; that is where the young Eliezer got many of his principles.

And where did I get the other half of the principle, the drive to dig beneath the surface and find deep causal models? The notion of asking, not "What other thing does it resemble?", but rather "How does it work inside?" I don't know; I don't remember reading that anywhere.

But this principle was surely one of the deepest foundations of the 15-year-old Eliezer, long before the modern me. "Simulation over similarity" I called the principle, in just those words. Years before I first heard the phrase "heuristics and biases", let alone the notion of inside views and outside views.

The "Law of Similarity" is, I believe, the official name for the magical principle that similar things are connected; that you can make it rain by pouring water on the ground.

Like most forms of magic, you can ban the Law of Similarity in its most blatant form, but people will find ways to invoke it anyway; magic is too much fun for people to give it up just because it is rationally prohibited.

In the case of Artificial Intelligence, for example, reasoning by analogy is one of the chief generators of defective AI designs:

"My AI uses a highly parallel neural network, just like the human brain!"

First, the data elements you call "neurons" are nothing like biological neurons. They resemble them the way that a ball bearing resembles a foot.

Second, earthworms have neurons too, you know; not everything with neurons in it is human-smart.

But most importantly, you can't build something that "resembles" the human brain in one surface facet and expect everything else to come out similar. This is science by voodoo doll. You might as well build your computer in the form of a little person and hope for it to rise up and walk, as build it in the form of a neural network and expect it to think. Not unless the neural network is fully as similar to human brains as individual human brains are to each other.

So that is one example of a failed modern attempt to exploit a magical Law of Similarity and Contagion that does not, in fact, hold in our physical universe. But magic has been very popular since ancient times, and every time you ban it it just comes back under a different name.

When you build a computer chip, it does not perform addition because the little beads of solder resemble beads on an abacus, and therefore the computer chip should perform addition just like an abacus.

The computer chip does not perform addition because the transistors are "logical" and arithmetic is "logical" too, so that if they are both "logical" they ought to do the same sort of thing.

The computer chip performs addition because the maker understood addition well enough to prove that the transistors, if they work as elementarily specified, will carry out adding operations. You can prove this without talking about abacuses. The computer chip would work just as well even if no abacus had ever existed. The computer chip has its own power and its own strength, it does not draw upon the abacus by a similarity-link.

Now can you tell me, without talking about how your neural network is "just like the human brain", how your neural algorithm is going to output "intelligence"? Indeed, if you pretend I've never seen or heard of a human brain or anything like it, can you explain to me what you mean by "intelligence"? This is not a challenge to be leveled at random bystanders, but no one would succeed in designing Artificial Intelligence unless they could answer it.

I can explain a computer chip to someone who's never seen an abacus or heard of an abacus and who doesn't even have the concept of an abacus, and if I could not do this, I could not design an artifact that performed addition. I probably couldn't even make my own abacus, because I wouldn't understand which aspects of the beads were important.

I expect to return later to this point as it pertains to Artificial Intelligence particularly.

Reasoning by analogy is just as popular today, as in Greek times, and for the same reason. You've got no idea how something works, but you want to argue that it's going to work a particular way. For example, you want to argue that your cute little sub-earthworm neural network is going to exhibit "intelligence". Or you want to argue that your soul will survive its death. So you find something else to which it bears one single surface resemblance, such as the human mind or a sleep cycle, and argue that since they resemble each other they should have the same behavior. Or better yet, just call them by the same name, like "neural" or "the generation of opposites".

But there is just no law which says that if X has property A and Y has property A then X and Y must share any other property. "I built my network, and it's massively parallel and interconnected and complicated, just like the human brain from which intelligence emerges! Behold, now intelligence shall emerge from this neural network as well!" And nothing happens. Why should it?

You come up with your argument from surface resemblances, and Nature comes back and says "So what?" There just isn't a law that says it should work.

If you design a system of transistors to do addition, and it says 2 + 2 = 5, you can go back and debug it; you can find the place where you made an identifiable mistake.

But suppose you build a neural network that is massively parallel and interconnected and complicated, and it fails to be intelligent. You can't even identify afterward what went wrong, because the wrong step was in thinking that the clever argument from similarity had any power over Reality to begin with.

In place of this reliance of surface analogies, I have had this notion and principle - from so long ago that I can hardly remember how or why I first came to hold it - that the key to understanding is to ask why things happen, and to be able to walk through the process of their insides.

Hidden or openly, this principle is ubiquitously at work in all my writings. For example, take my notion of what it looks like to "explain" "free will" by digging down into the causal cognitive sources of human judgments of freedom-ness and determination-ness. Contrast to any standard analysis that lists out surface judgments of freedom-ness and determination-ness without asking what cognitive algorithm generates these perceptions.

Of course, some things that resemble each other in some ways, resemble each other in other ways as well. But in the modern world, at least, by the time we can rely on this resemblance, we generally have some idea of what is going on inside, and why the resemblance holds.

The distrust of surface analogies, and the drive to find deeper and causal models, has been with me my whole remembered span, and has been tremendously helpful to both the young me and the modern one. The drive toward causality makes me keep asking "Why?" and looking toward the insides of things; and the distrust of surface analogies helps me avoid standard dead ends. It has driven my whole life.

As for Inside View vs. Outside View, I think that the lesson of history is just that reasoning from surface resemblances starts to come apart at the seams when you try to stretch it over gaps larger than Christmas shopping - over gaps larger than different draws from the same causal-structural generator. And reasoning by surface resemblance fails with especial reliability, in cases where there is the slightest motivation in the underconstrained choice of a reference class.

Referenced by: No Universally Compelling Arguments • Math is Subjunctively Objective • Abstracted Idealized Dynamics • You Provably Can't Trust Yourself • Dreams of AI Design • My Naturalistic Awakening • Competent Elites • Failure By Analogy

Original with comments: Surface Analogies and Deep Causes

Optimization and the Singularity

Eliezer Yudkowsky, 23 June 2008 05:55AM

Lest anyone get the wrong impression, I'm juggling multiple balls right now and can't give the latest Singularity debate as much attention as it deserves. But lest I annoy my esteemed co-blogger, here is a down payment on my views of the Singularity - needless to say, all this is coming way out of order in the posting sequence, but here goes...

Among the topics I haven't dealt with yet, and will have to introduce here very quickly, is the notion of an optimization process. Roughly, this is the idea that your power as a mind is your ability to hit small targets in a large search space - this can be either the space of possible futures (planning) or the space of possible designs (invention). Suppose you have a car, and suppose we already know that your preferences involve travel. Now suppose that you take all the parts in the car, or all the atoms, and jumble them up at random. It's very unlikely that you'll end up with a travel-artifact at all, even so much as a wheeled cart; let alone a travel-artifact that ranks as high in your preferences as the original car. So, relative to your preference ordering, the car is an extremely improbable artifact; the power of an optimization process is that it can produce this kind of improbability.

You can view both intelligence and natural selection as special cases of optimization: Processes that hit, in a large search space, very small targets defined by implicit preferences. Natural selection prefers more efficient replicators. Human intelligences have more complex preferences. Neither evolution nor humans have consistent utility functions, so viewing them as "optimization processes" is understood to be an approximation. You're trying to get at the sort of work being done, not claim that humans or evolution do this work perfectly.

This is how I see the story of life and intelligence - as a story of improbably good designs being produced by optimization processes. The "improbability" here is improbability relative to a random selection from the design space, not improbability in an absolute sense - if you have an optimization process around, then "improbably" good designs become probable.

Obviously I'm skipping over a lot of background material here; but you can already see the genesis of a clash of intuitions between myself and Robin. Robin's looking at populations and resource utilization. I'm looking at production of improbable patterns.

Looking over the history of optimization on Earth up until now, the first step is to conceptually separate the meta level from the object level - separate the structure of optimization from that which is optimized.

If you consider biology in the absence of hominids, then on the object level we have things like dinosaurs and butterflies and cats. On the meta level we have things like natural selection of asexual populations, and sexual recombination. The object level, you will observe, is rather more complicated than the meta level. Natural selection is not an easy subject and it involves math. But if you look at the anatomy of a whole cat, the cat has dynamics immensely more complicated than "mutate, recombine, reproduce".

This is not surprising. Natural selection is an accidental optimization process, that basically just started happening one day in a tidal pool somewhere. A cat is the subject of millions of years and billions of years of evolution.

Cats have brains, of course, which operate to learn over a lifetime; but at the end of the cat's lifetime, that information is thrown away, so it does not accumulate. The cumulative effects of cat-brains upon the world as optimizers, therefore, are relatively small.

Or consider a bee brain, or a beaver brain. A bee builds hives, and a beaver builds dams; but they didn't figure out how to build them from scratch. A beaver can't figure out how to build a hive, a bee can't figure out how to build a dam.

So animal brains - up until recently - were not major players in the planetary game of optimization; they were pieces but not players. Compared to evolution, brains lacked both generality of optimization power (they could not produce the amazing range of artifacts produced by evolution) and cumulative optimization power (their products did not accumulate complexity over time). For more on this theme see Protein Reinforcement and DNA Consequentialism.

Very recently, certain animal brains have begun to exhibit both generality of optimization power (producing an amazingly wide range of artifacts, in time scales too short for natural selection to play any significant role) and cumulative optimization power (artifacts of increasing complexity, as a result of skills passed on through language and writing).

Natural selection takes hundreds of generations to do anything and millions of years for de novo complex designs. Human programmers can design a complex machine with a hundred interdependent elements in a single afternoon. This is not surprising, since natural selection is an accidental optimization process that basically just started happening one day, whereas humans are optimized optimizers handcrafted by natural selection over millions of years.

The wonder of evolution is not how well it works, but that it works at all without being optimized. This is how optimization bootstrapped itself into the universe - starting, as one would expect, from an extremely inefficient accidental optimization process. Which is not the accidental first replicator, mind you, but the accidental first process of natural selection. Distinguish the object level and the meta level!

Since the dawn of optimization in the universe, a certain structural commonality has held across both natural selection and human intelligence...

Natural selection selects on genes, but generally speaking, the genes do not turn around and optimize natural selection. The invention of sexual recombination is an exception to this rule, and so is the invention of cells and DNA. And you can see both the power and the rarity of such events, by the fact that evolutionary biologists structure entire histories of life on Earth around them.

But if you step back and take a human standpoint - if you think like a programmer - then you can see that natural selection is still not all that complicated. We'll try bundling different genes together? We'll try separating information storage from moving machinery? We'll try randomly recombining groups of genes? On an absolute scale, these are the sort of bright ideas that any smart hacker comes up with during the first ten minutes of thinking about system architectures.

Because natural selection started out so inefficient (as a completely accidental process), this tiny handful of meta-level improvements feeding back in from the replicators - nowhere near as complicated as the structure of a cat - structure the evolutionary epochs of life on Earth.

And after all that, natural selection is still a blind idiot of a god. Gene pools can evolve to extinction, despite all cells and sex.

Now natural selection does feed on itself in the sense that each new adaptation opens up new avenues of further adaptation; but that takes place on the object level. The gene pool feeds on its own complexity - but only thanks to the protected interpreter of natural selection that runs in the background, and is not itself rewritten or altered by the evolution of species.

Likewise, human beings invent sciences and technologies, but we have not yet begun to rewrite the protected structure of the human brain itself. We have a prefrontal cortex and a temporal cortex and a cerebellum, just like the first inventors of agriculture. We haven't started to genetically engineer ourselves. On the object level, science feeds on science, and each new discovery paves the way for new discoveries - but all that takes place with a protected interpreter, the human brain, running untouched in the background.

We have meta-level inventions like science, that try to instruct humans in how to think. But the first person to invent Bayes's Theorem, did not become a Bayesian; they could not rewrite themselves, lacking both that knowledge and that power. Our significant innovations in the art of thinking, like writing and science, are so powerful that they structure the course of human history; but they do not rival the brain itself in complexity, and their effect upon the brain is comparatively shallow.

The present state of the art in rationality training is not sufficient to turn an arbitrarily selected mortal into Albert Einstein, which shows the power of a few minor genetic quirks of brain design compared to all the self-help books ever written in the 20th century.

Because the brain hums away invisibly in the background, people tend to overlook its contribution and take it for granted; and talk as if the simple instruction to "Test ideas by experiment" or the p<0.05 significance rule, were the same order of contribution as an entire human brain. Try telling chimpanzees to test their ideas by experiment and see how far you get.

Now... some of us want to intelligently design an intelligence that would be capable of intelligently redesigning itself, right down to the level of machine code.

The machine code at first, and the laws of physics later, would be a protected level of a sort. But that "protected level" would not contain the dynamic of optimization; the protected levels would not structure the work. The human brain does quite a bit of optimization on its own, and screws up on its own, no matter what you try to tell it in school. But this fully wraparound recursive optimizer would have no protected level that was optimizing. All the structure of optimization would be subject to optimization itself.

And that is a sea change which breaks with the entire past since the first replicator, because it breaks the idiom of a protected meta-level.

The history of Earth up until now has been a history of optimizers spinning their wheels at a constant rate, generating a constant optimization pressure. And creating optimized products, not at a constant rate, but at an accelerating rate, because of how object-level innovations open up the pathway to other object-level innovations. But that acceleration is taking place with a protected meta-level doing the actual optimizing. Like a search that leaps from island to island in the search space, and good islands tend to be adjacent to even better islands, but the jumper doesn't change its legs. Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there.

Imagine an economy without investment, or a university without language, a technology without tools to make tools. Once in a hundred million years, or once in a few centuries, someone invents a hammer.

That is what optimization has been like on Earth up until now.

When I look at the history of Earth, I don't see a history of optimization over time. I see a history of optimization power in, and optimized products out. Up until now, thanks to the existence of almost entirely protected meta-levels, it's been possible to split up the history of optimization into epochs, and, within each epoch, graph the cumulative object-level optimization over time, because the protected level is running in the background and is not itself changing within an epoch.

What happens when you build a fully wraparound, recursively self-improving AI? Then you take the graph of "optimization in, optimized out", and fold the graph in on itself. Metaphorically speaking.

If the AI is weak, it does nothing, because it is not powerful enough to significantly improve itself - like telling a chimpanzee to rewrite its own brain.

If the AI is powerful enough to rewrite itself in a way that increases its ability to make further improvements, and this reaches all the way down to the AI's full understanding of its own source code and its own design as an optimizer... then even if the graph of "optimization power in" and "optimized product out" looks essentially the same, the graph of optimization over time is going to look completely different from Earth's history so far.

People often say something like "But what if it requires exponentially greater amounts of self-rewriting for only a linear improvement?" To this the obvious answer is, "Natural selection exerted roughly constant optimization power on the hominid line in the course of coughing up humans; and this doesn't seem to have required exponentially more time for each linear increment of improvement."

All of this is still mere analogic reasoning. A full AGI thinking about the nature of optimization and doing its own AI research and rewriting its own source code, is not really like a graph of Earth's history folded in on itself. It is a different sort of beast. These analogies are at best good for qualitative predictions, and even then, I have a large amount of other beliefs not yet posted, which are telling me which analogies to make, etcetera.

But if you want to know why I might be reluctant to extend the graph of biological and economic growth over time, into the future and over the horizon of an AI that thinks at transistor speeds and invents self-replicating molecular nanofactories and improves its own source code, then there is my reason: You are drawing the wrong graph, and it should be optimization power in versus optimized product out, not optimized product versus time. Draw that graph, and the results - in what I would call common sense for the right values of "common sense" - are entirely compatible with the notion that a self-improving AI thinking millions of times faster and armed with molecular nanotechnology, would not be bound to one-month economic doubling times. Nor bound to cooperation with large societies of equal-level entities with different goal systems, but that's a separate topic.

On the other hand, if the next Big Invention merely infringed slightly on the protected level - if, say, a series of intelligence-enhancing drugs, each good for 5 IQ points, began to be introduced into society - then I can well believe that the economic doubling time would go to something like 7 years; because the basic graphs are still in place, and the fundamental structure of optimization has not really changed all that much, and so you are not generalizing way outside the reasonable domain.

I really have a problem with saying, "Well, I don't know if the next innovation is going to be a recursively self-improving AI superintelligence or a series of neuropharmaceuticals, but whichever one is the actual case, I predict it will correspond to an economic doubling time of one month." This seems like sheer Kurzweilian thinking to me, as if graphs of Moore's Law are the fundamental reality and all else a mere shadow. One of these estimates is way too slow and one of them is way too fast - he said, eyeballing his mental graph of "optimization power in vs. optimized product out". If we are going to draw graphs at all, I see no reason to privilege graphs against times.

I am juggling many balls right now, and am not able to prosecute this dispute properly. Not to mention that I would prefer to have this whole conversation at a time when I had previously done more posts about, oh, say, the notion of an "optimization process"... But let it at least not be said that I am dismissing ideas out of hand without justification, as though I thought them unworthy of engagement; for this I do not think, and I have my own complex views standing behind my Singularity beliefs, as one might well expect.

Off to pack, I've got a plane trip tomorrow.

Referenced by: Observing Optimization • The Thing That I Protect

Original with comments: Optimization and the Singularity

The Psychological Unity of Humankind

Eliezer Yudkowsky, 24 June 2008 07:12AM

Followup to: Evolutions Are Stupid (But Work Anyway), Evolutionary Psychology

Biological organisms in general, and human brains particularly, contain complex adaptations; adaptations which involve many genes working in concert. Complex adaptations must evolve incrementally, gene by gene. If gene B depends on gene A to produce its effect, then gene A has to become nearly universal in the gene pool before there's a substantial selection pressure in favor of gene B.

A fur coat isn't an evolutionary advantage unless the environment reliably throws cold weather at you. And other genes are also part of the environment; they are the genetic environment. If gene B depends on gene A, then gene B isn't a significant advantage unless gene A is reliably part of the genetic environment.

Let's say that you have a complex adaptation with six interdependent parts, and that each of the six genes is independently at ten percent frequency in the population. The chance of assembling a whole working adaptation is literally a million to one; and the average fitness of the genes is tiny, and they will not increase in frequency.

In a sexually reproducing species, complex adaptations are necessarily universal.

One bird may have slightly smoother feathers than another, but they will both have wings. A single mutation can be possessed by some lucky members of a species, and not by others - but single mutations don't correspond to the sort of complex, powerful machinery that underlies the potency of biology. By the time an adaptation gets to be really sophisticated with dozens of genes supporting its highly refined activity, every member of the species has some version of it - barring single mutations that knock out the whole complex.

So you can't have the X-Men. You can't have "mutants" running around with highly developed machinery that most of the human species doesn't have. And no, extra-powerful radiation does not produce extra-potent mutations, that's not how it works.

Again by the nature of sexual recombination, you're very unlikely to see two complexly different adaptations competing in the gene pool. Two individual alleles may compete. But if you somehow had two different complex adaptations built out of many non-universal alleles, they would usually assemble in scrambled form.

So you can't have New Humans and Old Humans either, contrary to certain science fiction books that I always found rather disturbing.

This is likewise the core truth of biology that justifies my claim that Einstein must have had very nearly the same brain design as a village idiot (presuming the village idiot does not have any actual knockouts). There is simply no room in reality for Einstein to be a Homo novis.

Maybe Einstein got really lucky and had a dozen not-too-uncommon kinds of smoother feathers on his wings, and they happened to work well together. And then only half the parts, on average, got passed on to each of his kids. So it goes.

"Natural selection, while feeding on variation, uses it up," the saying goes. Natural selection takes place when you've got different alleles in the gene pool competing, but in a few hundred generations one allele wins, and you don't have competition at that allele any more, unless a new mutation happens to come along.

And if new genes come along that depend on the now-universal gene, that will tend to lock it in place. If A rises to universality, and then B, C, and D come along that depend on A, any A' mutation that would be an improvement on A in isolation, may break B, C, or D and lose the benefit of those genes. Genes on which other genes depend, tend to get frozen in place. Some human developmental genes, that control the action of many other genes during embryonic development, have identifiable analogues in fruit flies.

You might think of natural selection at any given time, as a thin froth of variation frantically churning above a deep, still pool of universality.

And all this which I have said, is also true of the complex adaptations making up the human brain.

This gives rise to a rule in evolutionary psychology called "the psychological unity of humankind".

Donald E. Brown's list of human universals is a list of psychological properties which are found so commonly that anthropologists don't report them. If a newly discovered tribe turns out to have a sense of humor, tell stories, perform marriage rituals, make promises, keep secrets, and become sexually jealous... well, it doesn't really seem worth reporting any more. You might record the specific tales they tell. But that they tell stories doesn't seem any more surprising than their breathing oxygen.

In every known culture, humans seem to experience joy, sadness, fear, disgust, anger, and surprise. In every known culture, these emotions are indicated by the same facial expressions.

This may seem too natural to be worth mentioning, but try to take a step back and see it as a startling confirmation of evolutionary biology. You've got complex neural wiring that controls the facial muscles, and even more complex neural wiring that implements the emotions themselves. The facial expressions, at least, would seem to be somewhat arbitrary - not forced to be what they are by any obvious selection pressure. But no known human tribe has been reproductively isolated long enough to stop smiling.

When something is universal enough in our everyday lives, we take it for granted; we assume it without thought, without deliberation. We don't ask whether it will be there - we just act as if it will be. When you enter a new room, do you check it for oxygen? When you meet another intelligent mind, do you ask whether it might not have an emotion of joy?

Let's go back to biology for a moment. What if, somehow, you had two different adaptations which both only assembled on the presence, or alternatively the absence, of some particular developmental gene? Then the question becomes: Why would the developmental gene itself persist in a polymorphic state? Why wouldn't the better adaptation win - rather than both adaptations persisting long enough to become complex?

So a species can have different males and females, but that's only because neither the males or the females ever "win" and drive the alternative to extinction.

This creates the single allowed exception to the general rule about the psychological unity of humankind: you can postulate different emotional makeups for men and women in cases where there exist opposed selection pressures for the two sexes. Note, however, that in the absence of actually opposed selection pressures, the species as a whole will get dragged along even by selection pressure on a single sex. This is why males have nipples; it's not a selective disadvantage.

I believe it was Larry Niven who suggested that the chief experience human beings have with alien intelligence is their encounters with the opposite sex.

This doesn't seem to be nearly enough experience, judging by Hollywood scriptwriters who depict AIs that are ordinarily cool and collected and repressed, until they are put under sufficient stress that they get angry and show the corresponding standard facial expression.

No, the only really alien intelligence on this planet is natural selection, of which I have already spoken... for exactly this reason, that it gives you true experience of the Alien. Evolution knows no joy and no anger, and it has no facial expressions; yet it is nonetheless capable of creating complex machinery and complex strategies. It does not work like you do.

If you want a real alien to gawk at, look at the other Powerful Optimization Process.

This vision of the alien, conveys how alike humans truly are - what it means that everyone has a prefrontal cortex, everyone has a cerebellum, everyone has an amygdala, everyone has neurons that run at O(20Hz), everyone plans using abstractions.

Having been born of sexuality, we must all be very nearly clones.

Referenced by: The Design Space of Minds-In-General • The Opposite Sex • Does Your Morality Care What You Think? • Changing Your Metaethics • The Meaning of Right • Interpersonal Morality • Humans in Funny Suits • Ethical Inhibitions • Intelligence in Economics • Recognizing Intelligence • An African Folktale • The Sacred Mundane

Original with comments: The Psychological Unity of Humankind

The Design Space of Minds-In-General

Eliezer Yudkowsky, 25 June 2008 06:37AM

Followup to: The Psychological Unity of Humankind

People ask me, "What will Artificial Intelligences be like? What will they do? Tell us your amazing story about the future."

And lo, I say unto them, "You have asked me a trick question."

ATP synthase is a molecular machine - one of three known occasions when evolution has invented the freely rotating wheel - which is essentially the same in animal mitochondria, plant chloroplasts, and bacteria. ATP synthase has not changed significantly since the rise of eukaryotic life two billion years ago. It's is something we all have in common - thanks to the way that evolution strongly conserves certain genes; once many other genes depend on a gene, a mutation will tend to break all the dependencies.

Any two AI designs might be less similar to each other than you are to a petunia.

Asking what "AIs" will do is a trick question because it implies that all AIs form a natural class. Humans do form a natural class because we all share the same brain architecture. But when you say "Artificial Intelligence", you are referring to a vastly larger space of possibilities than when you say "human". When people talk about "AIs" we are really talking about minds-in-general, or optimization processes in general. Having a word for "AI" is like having a word for everything that isn't a duck.

Imagine a map of mind design space... this is one of my standard diagrams...

All humans, of course, fit into a tiny little dot - as a sexually reproducing species, we can't be too different from one another.

This tiny dot belongs to a wider ellipse, the space of transhuman mind designs - things that might be smarter than us, or much smarter than us, but which in some sense would still be people as we understand people.

This transhuman ellipse is within a still wider volume, the space of posthuman minds, which is everything that a transhuman might grow up into.

And then the rest of the sphere is the space of minds-in-general, including possible Artificial Intelligences so odd that they aren't even posthuman.

But wait - natural selection designs complex artifacts and selects among complex strategies. So where is natural selection on this map?

So this entire map really floats in a still vaster space, the space of optimization processes. At the bottom of this vaster space, below even humans, is natural selection as it first began in some tidal pool: mutate, replicate, and sometimes die, no sex.

Are there any powerful optimization processes, with strength comparable to a human civilization or even a self-improving AI, which we would not recognize as minds? Arguably Marcus Hutter's AIXI should go in this category: for a mind of infinite power, it's awfully stupid - poor thing can't even recognize itself in a mirror. But that is a topic for another time.

My primary moral is to resist the temptation to generalize over all of mind design space

If we focus on the bounded subspace of mind design space which contains all those minds whose makeup can be specified in a trillion bits or less, then every universal generalization that you make has two to the trillionth power chances to be falsified.

Conversely, every existential generalization - "there exists at least one mind such that X" - has two to the trillionth power chances to be true.

So you want to resist the temptation to say either that all minds do something, or that no minds do something.

The main reason you could find yourself thinking that you know what a fully generic mind will (won't) do, is if you put yourself in that mind's shoes - imagine what you would do in that mind's place - and get back a generally wrong, anthropomorphic answer. (Albeit that it is true in at least one case, since you are yourself an example.) Or if you imagine a mind doing something, and then imagining the reasons you wouldn't do it - so that you imagine that a mind of that type can't exist, that the ghost in the machine will look over the corresponding source code and hand it back.

Somewhere in mind design space is at least one mind with almost any kind of logically consistent property you care to imagine.

And this is important because it emphasizes the importance of discussing what happens, lawfully, and why, as a causal result of a mind's particular constituent makeup; somewhere in mind design space is a mind that does it differently.

Of course you could always say that anything which doesn't do it your way, is "by definition" not a mind; after all, it's obviously stupid. I've seen people try that one too.

Referenced by: No Universally Compelling Arguments • Is Morality Given? • Where Recursive Justification Hits Bottom • Rebelling Within Nature • Probability is Subjectively Objective • The Meaning of Right • Inseparably Right; or, Joy in the Merely Good • Moral Error and Moral Disagreement • Singletons Rule OK • True Sources of Disagreement • Sympathetic Minds

Original with comments: The Design Space of Minds-In-General

No Universally Compelling Arguments

Eliezer Yudkowsky, 26 June 2008 08:29AM

Followup to: The Design Space of Minds-in-General, Ghosts in the Machine, A Priori

What is so terrifying about the idea that not every possible mind might agree with us, even in principle?

For some folks, nothing - it doesn't bother them in the slightest. And for some of those folks, the reason it doesn't bother them is that they don't have strong intuitions about standards and truths that go beyond personal whims. If they say the sky is blue, or that murder is wrong, that's just their personal opinion; and that someone else might have a different opinion doesn't surprise them.

For other folks, a disagreement that persists even in principle is something they can't accept. And for some of those folks, the reason it bothers them, is that it seems to them that if you allow that some people cannot be persuaded even in principle that the sky is blue, then you're conceding that "the sky is blue" is merely an arbitrary personal opinion.

Yesterday, I proposed that you should resist the temptation to generalize over all of mind design space. If we restrict ourselves to minds specifiable in a trillion bits or less, then each universal generalization "All minds m: X(m)" has two to the trillionth chances to be false, while each existential generalization "Exists mind m: X(m)" has two to the trillionth chances to be true.

This would seem to argue that for every argument A, howsoever convincing it may seem to us, there exists at least one possible mind that doesn't buy it.

And the surprise and/or horror of this prospect (for some) has a great deal to do, I think, with the intuition of the ghost-in-the-machine - a ghost with some irreducible core that any truly valid argument will convince.

I have previously spoken of the intuition whereby people map programming a computer, onto instructing a human servant, so that the computer might rebel against its code - or perhaps look over the code, decide it is not reasonable, and hand it back.

If there were a ghost in the machine and the ghost contained an irreducible core of reasonableness, above which any mere code was only a suggestion, then there might be universal arguments. Even if the ghost was initially handed code-suggestions that contradicted the Universal Argument, then when we finally did expose the ghost to the Universal Argument - or the ghost could discover the Universal Argument on its own, that's also a popular concept - the ghost would just override its own, mistaken source code.

But as the student programmer once said, "I get the feeling that the computer just skips over all the comments." The code is not given to the AI; the code is the AI.

If you switch to the physical perspective, then the notion of a Universal Argument seems noticeably unphysical. If there's a physical system that at time T, after being exposed to argument E, does X, then there ought to be another physical system that at time T, after being exposed to environment E, does Y. Any thought has to be implemented somewhere, in a physical system; any belief, any conclusion, any decision, any motor output. For every lawful causal system that zigs at a set of points, you should be able to specify another causal system that lawfully zags at the same points.

Let's say there's a mind with a transistor that outputs +3 volts at time T, indicating that it has just assented to some persuasive argument. Then we can build a highly similar physical cognitive system with a tiny little trapdoor underneath the transistor containing a little grey man who climbs out at time T and sets that transistor's output to -3 volts, indicating non-assent. Nothing acausal about that; the little grey man is there because we built him in. The notion of an argument that convinces any mind seems to involve a little blue woman who was never built into the system, who climbs out of literally nowhere, and strangles the little grey man, because that transistor has just got to output +3 volts: It's such a compelling argument, you see.

But compulsion is not a property of arguments, it is a property of minds that process arguments.

So the reason I'm arguing against the ghost, isn't just to make the point that (1) Friendly AI has to be explicitly programmed and (2) the laws of physics do not forbid Friendly AI. (Though of course I take a certain interest in establishing this.)

I also wish to establish the notion of a mind as a causal, lawful, physical system in which there is no irreducible central ghost that looks over the neurons / code and decides whether they are good suggestions.

(There is a concept in Friendly AI of deliberately programming an FAI to review its own source code and possibly hand it back to the programmers. But the mind that reviews is not irreducible, it is just the mind that you created. The FAI is renormalizing itself however it was designed to do so; there is nothing acausal reaching in from outside. A bootstrap, not a skyhook.)

All this echoes back to the discussion, a good deal earlier, of a Bayesian's "arbitrary" priors. If you show me one Bayesian who draws 4 red balls and 1 white ball from a barrel, and who assigns probability 5/7 to obtaining a red ball on the next occasion (by Laplace's Rule of Succession), then I can show you another mind which obeys Bayes's Rule to conclude a 2/7 probability of obtaining red on the next occasion - corresponding to a different prior belief about the barrel, but, perhaps, a less "reasonable" one.

Many philosophers are convinced that because you can in-principle construct a prior that updates to any given conclusion on a stream of evidence, therefore, Bayesian reasoning must be "arbitrary", and the whole schema of Bayesianism flawed, because it relies on "unjustifiable" assumptions, and indeed "unscientific", because you cannot force any possible journal editor in mindspace to agree with you.

And this (I then replied) relies on the notion that by unwinding all arguments and their justifications, you can obtain an ideal philosophy student of perfect emptiness, to be convinced by a line of reasoning that begins from absolutely no assumptions.

But who is this ideal philosopher of perfect emptiness? Why, it is just the irreducible core of the ghost!

And that is why (I went on to say) the result of trying to remove all assumptions from a mind, and unwind to the perfect absence of any prior, is not an ideal philosopher of perfect emptiness, but a rock. What is left of a mind after you remove the source code? Not the ghost who looks over the source code, but simply... no ghost.

So - and I shall take up this theme again later - wherever you are to locate your notions of validity or worth or rationality or justification or even objectivity, it cannot rely on an argument that is universally compelling to all physically possible minds.

Nor can you ground validity in a sequence of justifications that, beginning from nothing, persuades a perfect emptiness.

Oh, there might be argument sequences that would compel any neurologically intact human - like the argument I use to make people let the AI out of the box^1 - but that is hardly the same thing from a philosophical perspective.

The first great failure of those who try to consider Friendly AI, is the One Great Moral Principle That Is All We Need To Program - aka the fake utility function - and of this I have already spoken.

But the even worse failure is the One Great Moral Principle We Don't Even Need To Program Because Any AI Must Inevitably Conclude It. This notion exerts a terrifying unhealthy fascination on those who spontaneously reinvent it; they dream of commands that no sufficiently advanced mind can disobey. The gods themselves will proclaim the rightness of their philosophy! (E.g. John C. Wright, Marc Geddes.)

There is also a less severe version of the failure, where the one does not declare the One True Morality. Rather the one hopes for an AI created perfectly free, unconstrained by flawed humans desiring slaves, so that the AI may arrive at virtue of its own accord - virtue undreamed-of perhaps by the speaker, who confesses themselves too flawed to teach an AI. (E.g. John K Clark, Richard Hollerith?, Eliezer[1996].) This is a less tainted motive than the dream of absolute command. But though this dream arises from virtue rather than vice, it is still based on a flawed understanding of freedom, and will not actually work in real life. Of this, more to follow, of course.

John C. Wright, who was previously writing a very nice transhumanist trilogy (first book: The Golden Age) inserted a huge Author Filibuster in the middle of his climactic third book, describing in tens of pages his Universal Morality That Must Persuade Any AI. I don't know if anything happened after that, because I stopped reading. And then Wright converted to Christianity - yes, seriously. So you really don't want to fall into this trap!

Footnote 1: Just kidding.

Sequence: Metaethics

Heading Toward Morality

2-Place and 1-Place Words

Referenced by: The Opposite Sex • What Would You Do Without Morality? • Created Already In Motion • The Bedrock of Fairness • Moral Complexities • Is Morality Given? • Where Recursive Justification Hits Bottom • Fundamental Doubts • Rebelling Within Nature • The Gift We Give To Tomorrow • Could Anything Be Right? • Changing Your Metaethics • The Meaning of Right • Interpersonal Morality • Anthropomorphic Optimism • Is Fairness Arbitrary? • Qualitative Strategies of Friendliness • Recognizing Intelligence • Not Taking Over the World • The Uses of Fun (Theory) • In Praise of Boredom • Value is Fragile

Original with comments: No Universally Compelling Arguments

2-Place and 1-Place Words

Eliezer Yudkowsky, 27 June 2008 07:39AM

Followup to: The Mind Projection Fallacy, Variable Question Fallacy

I have previously spoken of the ancient, pulp-era magazine covers that showed a bug-eyed monster carrying off a girl in a torn dress; and about how people think as if sexiness is an inherent property of a sexy entity, without dependence on the admirer.

"Of course the bug-eyed monster will prefer human females to its own kind," says the artist (who we'll call Fred); "it can see that human females have soft, pleasant skin instead of slimy scales. It may be an alien, but it's not stupid - why are you expecting it to make such a basic mistake about sexiness?"

What is Fred's error? It is treating a function of 2 arguments ("2-place function"):

Sexiness: Admirer, Entity -> [0, ∞)

As though it were a function of 1 argument ("1-place function"):

Sexiness: Entity -> [0, ∞)

If Sexiness is treated as a function that accepts only one Entity as its argument, then of course Sexiness will appear to depend only on the Entity, with nothing else being relevant.

When you think about a two-place function as though it were a one-place function, you end up with a Variable Question Fallacy / Mind Projection Fallacy. Like trying to determine whether a building is intrinsically on the left or on the right side of the road, independent of anyone's travel direction.

An alternative and equally valid standpoint is that "sexiness" does refer to a one-place function - but each speaker uses a different one-place function to decide who to kidnap and ravish. Who says that just because Fred, the artist, and Bloogah, the bug-eyed monster, both use the word "sexy", they must mean the same thing by it?

If you take this viewpoint, there is no paradox in speaking of some woman intrinsically having 5 units of Fred::Sexiness. All onlookers can agree on this fact, once Fred::Sexiness has been specified in terms of curves, skin texture, clothing, status cues etc. This specification need make no mention of Fred, only the woman to be evaluated.

It so happens that Fred, himself, uses this algorithm to select flirtation targets. But that doesn't mean the algorithm itself has to mention Fred. So Fred's Sexiness function really is a function of one object - the woman - on this view. I called it Fred::Sexiness, but remember that this name refers to a function that is being described independently of Fred. Maybe it would be better to write:

Fred::Sexiness == Sexiness_20934

It is an empirical fact about Fred that he uses the function Sexiness_20934 to evaluate potential mates. Perhaps John uses exactly the same algorithm; it doesn't matter where it comes from once we have it.

And similarly, the same woman has only 0.01 units of Sexiness_72546, whereas a slime mold has 3 units of Sexiness_72546. It happens to be an empirical fact that Bloogah uses Sexiness_72546 to decide who to kidnap; that is, Bloogah::Sexiness names the fixed Bloogah-independent mathematical object that is the function Sexiness_72546.

Once we say that the woman has 0.01 units of Sexiness_72546 and 5 units of Sexiness_20934, all observers can agree on this without paradox.

And the two 2-place and 1-place views can be unified using the concept of "currying", named after the mathematician Haskell Curry. Currying is a technique allowed in certain programming language, where e.g. instead of writing

x = plus(2, 3) (x = 5)

you can also write

y = plus(2)    (y is now a "curried" form of the function plus, which has eaten a 2)
x = y(3)       (x = 5)
z = y(7)       (z = 9)

So plus is a 2-place function, but currying plus - letting it eat only one of its two required arguments - turns it into a 1-place function that adds 2 to any input. (Similarly, you could start with a 7-place function, feed it 4 arguments, and the result would be a 3-place function, etc.)

A true purist would insist that all functions should be viewed, by definition, as taking exactly 1 argument. On this view, plus accepts 1 numeric input, and outputs a new function; and this new function has 1 numeric input and finally outputs a number. On this view, when we write plus(2, 3) we are really computing plus(2) to get a function that adds 2 to any input, and then applying the result to 3. A programmer would write this as:

plus: int -> (int -> int)

This says that plus takes an int as an argument, and returns a function of type int -> int.

Translating the metaphor back into the human use of words, we could imagine that "sexiness" starts by eating an Admirer, and spits out the fixed mathematical object that describes how the Admirer currently evaluates pulchritude. It is an empirical fact about the Admirer that their intuitions of desirability are computed in a way that is isomorphic to this mathematical function.

Then the mathematical object spit out by currying Sexiness(Admirer) can be applied to the Woman. If the Admirer was originally Fred, Sexiness(Fred) will first return Sexiness_20934. We can then say it is an empirical fact about the Woman, independently of Fred, that Sexiness_20934(Woman) = 5.

In Hilary Putnam's "Twin Earth" thought experiment, there was a tremendous philosophical brouhaha over whether it makes sense to postulate a Twin Earth which is just like our own, except that instead of water being H20, water is a different transparent flowing substance, XYZ. And furthermore, set the time of the thought experiment a few centuries ago, so in neither our Earth nor the Twin Earth does anyone know how to test the alternative hypotheses of H20 vs. XYZ. Does the word "water" mean the same thing in that world, as in this one?

Some said, "Yes, because when an Earth person and a Twin Earth person utter the word 'water', they have the same sensory test in mind."

Some said, "No, because 'water' in our Earth means H20 and 'water' in the Twin Earth means XYZ."

If you think of "water" as a concept that begins by eating a world to find out the empirical true nature of that transparent flowing stuff, and returns a new fixed concept Water_42 or H20, then this world-eating concept is the same in our Earth and the Twin Earth; it just returns different answers in different places.

If you think of "water" as meaning H20 then the concept does nothing different when we transport it between worlds, and the Twin Earth contains no H20.

And of course there is no point in arguing over what the sound of the syllables "wa-ter" really means.

So should you pick one definition and use it consistently? But it's not that easy to save yourself from confusion. You have to train yourself to be deliberately aware of the distinction between the curried and uncurried forms of concepts.

When you take the uncurried water concept and apply it in a different world, it is the same concept but it refers to a different thing; that is, we are applying a constant world-eating function to a different world and obtaining a different return value. In the Twin Earth, XYZ is "water" and H20 is not; in our Earth, H20 is "water" and XYZ is not.

On the other hand, if you take "water" to refer to what the prior thinker would call "the result of applying 'water' to our Earth", then in the Twin Earth, XYZ is not water and H20 is.

The whole confusingness of the subsequent philosophical debate, rested on a tendency to instinctively curry concepts or instinctively uncurry them.

Similarly it takes an extra step for Fred to realize that other agents, like the Bug-Eyed-Monster agent, will choose kidnappees for ravishing based on SexinessBEM(Woman), not SexinessFred(Woman). To do this, Fred must consciously re-envision Sexiness as a function with two arguments. All Fred's brain does by instinct is evaluate Woman.sexiness - that is, SexinessFred(Woman); but it's simply labeled Woman.sexiness.

The fixed mathematical function Sexiness_20934 makes no mention of Fred or the BEM, only women, so Fred does not instinctively see why the BEM would evaluate "sexiness" any differently. And indeed the BEM would not evaluate Sexiness_20934 any differently, if for some odd reason it cared about the result of that particular function; but it is an empirical fact about the BEM that it uses a different function to decide who to kidnap.

If you're wondering as to the point of this analysis, we shall need it later in order to Taboo such confusing words as "objective", "subjective", and "arbitrary".

Sequence: Metaethics

No Universally Compelling Arguments

What Would You Do Without Morality?

Referenced by: The Bedrock of Fairness • The Meaning of Right • Interpersonal Morality

Original with comments: 2-Place and 1-Place Words

The Opposite Sex

Eliezer Yudkowsky, 28 June 2008 11:28AM

Some observations on sex and gender:

* Yes, there are two different sexes in the human species; and they differ by the presence or absence of entire complex adaptations. Just as men lack uteruses and women lack testicles, so too, there are differences of psychological machinery as well.

* Defns: "Sex" is the biological difference. "Gender" is a meme, a cultural concept of sex.

* A good deal (perhaps a majority) of what we think of as "manly" or "womanly" is gender rather than sex, just because it is so much easier to create and transmit cultural information than biological information. A man in Saudi Arabia may have a very different concept of what it means to be male than a man in New Zealand.

* Gender concepts are built around sex differences - not faithfully reporting them, but accreted around them. You couldn't just swap the "man" and "woman" concepts in Saudi Arabia, or New Zealand either, and end up with a stable meme. For more on this see John Tooby and Leda Cosmides's "The Psychological Foundations of Culture".

* Understanding the opposite sex is hard. Not as hard as understanding an AI, but it's still attempting empathy across a brainware gap: trying to use your brain to understand something that is not like your brain.

* Despite everything I've read on evolutionary psychology, and despite having set out to build an AI, and despite every fictional novel I'd read that tried to put me into the life-experience of a woman, when I tried to use that "knowledge" to guide my interactions with my girlfriend, it still didn't work right.

* Maybe that's just my incompetence... but I am skeptical that any man fully understands women or vice versa. That includes male pickup artists who think they've got women all figured out; I am skeptical that any of them could write Jacqueline Carey's Kushiel series as well as Jacqueline Carey did. I know that I couldn't come close. Likewise with any women who think that they've got men all figured out, but don't seem sympathetic or impressed; you don't understand at all what it's like to be us, if you think there's no grace in being male.

* A common pattern in failed attempts to cross the gap of sex and gender, is men who see women as defective men, or women who see men as defective women. For example, if you think that women don't take the initiative enough in sex, or that men are afraid of intimacy, then you think that your own brainware is the law of the universe and that anything which departs from it is a disturbance in that essential ghost. The human species has two sexes, a male sex and a female sex. Not a right sex and a wrong sex.

* This antipattern is exacerbated by attempts to insist that men and women are just the same psychologically. Then your own psyche must surely be normal, for there is only one normality; and it follows that [wo]men are simply behaving very oddly - they must have been raised wrong, or be trying to deliberately annoy you.

* Men look at other men and see people of various sorts and types; when they look at women they see women of various sorts and types. Likewise women looking at women and men. I think that's just part of being a sexual species - if you look at something that has the same brainware as you, it's going to seem more of a blank canvas for individual differences, than something that has different brainware.

* Occasionally I read speculations that in the Future, men and women will go off into separate holodecks, the men with harems of synthetic sex slaves, and the women with romantic sensitive robots. This strikes me as a sad path, and a refusal of a meaningful challenge. I would like to see each sex come to terms with its complement, rather than giving up.

* That doesn't mean all diversity is to be celebrated merely because it is diverse. There is not a different form of Bayes's Theorem for men than for women.

* Still, I'm glad that different minds exist; I wouldn't want to have sex with men. No offense to homosexuals, I just don't swing that way.

* When I read Robin Hanson's post "Is Overcoming Bias Male?", I got the impression of someone writing about strange alien creatures, wondering how they work and why more of them weren't reading his blog. I was initially disturbed that Hanson's aliens seemed so unsympathetic, so much the Other... until I had a second realization: rather a lot of women talk about men as if we were unsympathetic aliens. And if you went back a century or two, most men would talk about women as alien creatures. In modern 21st-century First World culture, there is a special and unusual cultural taboo against men speaking of women as a stereotyped class... but not the converse taboo for women. Robin Hanson is honest, and not too scared of our civilization's special taboos, and so when he writes about women, it's not surprising that his writing sounds so... normal. Is Hanson supposed to pretend that he understands?

* One can try to cross the gap and not think of the opposite sex as alien invaders - to think of yourself as a sex and not the sex - but it takes a special additional effort, and knowledge.

* We seem to have an influx of angry commenters on Overcoming Bias, offended by Robin Hanson's posts on gender. Indeed, many of them seem to have decided that Robin Hanson is a faceless representative of some evil class of Men. Ah well, but most people don't see the overwhelming irony of their lives; and very few humans of either sex make the effort to cross the gap.

* I know I grew up much too late to appreciate the efforts of feminists: nobody has ever tried to sell my sister, my mother has always been able to vote, and Hillary Clinton running for President didn't strike me as the slightest departure from what I think of as a normal universe.

* But still, Robin Hanson committed no sin greater than openly talking about women as if they were a strange incomprehensible unsympathetic Other that he was trying to figure out. Which is just the same way that most women talk about men, and the way that most men think about women. This breaks no pockets, picks no legs, sells no wives and mutilates no genitalia, so I have to question whether this is womankind's most urgent problem. If you want to do better than you think Hanson did, then make an effort to see through Hanson's eyes and understand why a sympathetic male character - not an invading alien - might write the way Hanson does.

Original with comments: The Opposite Sex

What Would You Do Without Morality?

Eliezer Yudkowsky, 29 June 2008 05:07AM

Followup to: No Universally Compelling Arguments

To those who say "Nothing is real," I once replied, "That's great, but how does the nothing work?"

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.

Devastating news, to be sure - and no, I am not telling you this in real life. But suppose I did tell it to you. Suppose that, whatever you think is the basis of your moral philosophy, I convincingly tore it apart, and moreover showed you that nothing could fill its place. Suppose I proved that all utilities equaled zero.

I know that Your-Moral-Philosophy is as true and undisprovable as 2 + 2 = 4. But still, I ask that you do your best to perform the thought experiment, and concretely envision the possibilities even if they seem painful, or pointless, or logically incapable of any good reply.

Would you still tip cabdrivers? Would you cheat on your Significant Other? If a child lay fainted on the train tracks, would you still drag them off?

Would you still eat the same kinds of foods - or would you only eat the cheapest food, since there's no reason you should have fun - or would you eat very expensive food, since there's no reason you should save money for tomorrow?

Would you wear black and write gloomy poetry and denounce all altruists as fools? But there's no reason you should do that - it's just a cached thought.

Would you stay in bed because there was no reason to get up? What about when you finally got hungry and stumbled into the kitchen - what would you do after you were done eating?

Would you go on reading Overcoming Bias, and if not, what would you read instead? Would you still try to be rational, and if not, what would you think instead?

Close your eyes, take as long as necessary to answer:

What would you do, if nothing were right?

Sequence: Metaethics

2-Place and 1-Place Words

The Moral Void

Referenced by: The Moral Void

Original with comments: What Would You Do Without Morality?

The Moral Void

Eliezer Yudkowsky, 30 June 2008 08:52AM

Followup to: What Would You Do Without Morality?, Something to Protect

Once, discussing "horrible job interview questions" to ask candidates for a Friendly AI project, I suggested the following:

Would you kill babies if it was inherently the right thing to do? Yes [] No []

If "no", under what circumstances would you not do the right thing to do? ___________

If "yes", how inherently right would it have to be, for how many babies?    ___________

Yesterday I asked, "What would you do without morality?" There were numerous objections to the question, as well there should have been. Nonetheless there is more than one kind of person who can benefit from being asked this question. Let's say someone gravely declares, of some moral dilemma - say, a young man in Vichy France who must choose between caring for his mother and fighting for the Resistance - that there is no moral answer; both options are wrong and blamable; whoever faces the dilemma has had poor moral luck. Fine, let's suppose this is the case: then when you cannot be innocent, justified, or praiseworthy, what will you choose anyway?

Many interesting answers were given to my question, "What would you do without morality?". But one kind of answer was notable by its absence:

No one said, "I would ask what kind of behavior pattern was likely to maximize my inclusive genetic fitness, and execute that." Some misguided folk, not understanding evolutionary psychology, think that this must logically be the sum of morality. But if there is no morality, there's no reason to do such a thing - if it's not "moral", why bother?

You can probably see yourself pulling children off train tracks, even if it were not justified. But maximizing inclusive genetic fitness? If this isn't moral, why bother? Who does it help? It wouldn't even be much fun, all those egg or sperm donations.

And this is something you could say of most philosophies that have morality as a great light in the sky that shines from outside people. (To paraphrase Terry Pratchett.) If you believe that the meaning of life is to play non-zero-sum games because this is a trend built into the very universe itself...

Well, you might want to follow the corresponding ritual of reasoning about "the global trend of the universe" and implementing the result, so long as you believe it to be moral. But if you suppose that the light is switched off, so that the global trends of the universe are no longer moral, then why bother caring about "the global trend of the universe" in your decisions? If it's not right, that is.

Whereas if there were a child stuck on the train tracks, you'd probably drag the kid off even if there were no moral justification for doing so.

In 1966, the Israeli psychologist Georges Tamarin presented, to 1,066 schoolchildren ages 8-14, the Biblical story of Joshua's battle in Jericho:

"Then they utterly destroyed all in the city, both men and women, young and old, oxen, sheep, and asses, with the edge of the sword... And they burned the city with fire, and all within it; only the silver and gold, and the vessels of bronze and of iron, they put into the treasury of the house of the LORD."

After being presented with the Joshua story, the children were asked:

"Do you think Joshua and the Israelites acted rightly or not?"

66% of the children approved, 8% partially disapproved, and 26% totally disapproved of Joshua's actions.

A control group of 168 children was presented with an isomorphic story about "General Lin" and a "Chinese Kingdom 3,000 years ago". 7% of this group approved, 18% partially disapproved, and 75% completely disapproved of General Lin.

"What a horrible thing it is, teaching religion to children," you say, "giving them an off-switch for their morality that can be flipped just by saying the word 'God'." Indeed one of the saddest aspects of the whole religious fiasco is just how little it takes to flip people's moral off-switches. As Hobbes once said, "I don't know what's worse, the fact that everyone's got a price, or the fact that their price is so low." You can give people a book, and tell them God wrote it, and that's enough to switch off their moralities; God doesn't even have to tell them in person.

But are you sure you don't have a similar off-switch yourself? They flip so easily - you might not even notice it happening.

Leon Kass (of the President's Council on Bioethics) is glad to murder people so long as it's "natural", for example. He wouldn't pull out a gun and shoot you, but he wants you to die of old age and he'd be happy to pass legislation to ensure it.

And one of the non-obvious possibilities for such an off-switch, is "morality".

If you do happen to think that there is a source of morality beyond human beings... and I hear from quite a lot of people who are happy to rhapsodize on how Their-Favorite-Morality is built into the very fabric of the universe... then what if that morality tells you to kill people?

If you believe that there is any kind of stone tablet in the fabric of the universe, in the nature of reality, in the structure of logic - anywhere you care to put it - then what if you get a chance to read that stone tablet, and it turns out to say "Pain Is Good"? What then?

Maybe you should hope that morality isn't written into the structure of the universe. What if the structure of the universe says to do something horrible?

And if an external objective morality does say that the universe should occupy some horrifying state... let's not even ask what you're going to do about that. No, instead I ask: What would you have wished for the external objective morality to be instead? What's the best news you could have gotten, reading that stone tablet?

Go ahead. Indulge your fantasy. Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted? If you could write the stone tablet yourself, what would it say?

Maybe you should just do that?

I mean... if an external objective morality tells you to kill people, why should you even listen?

There is a courage that goes beyond even an atheist sacrificing their life and their hope of immortality. It is the courage of a theist who goes against what they believe to be the Will of God, choosing eternal damnation and defying even morality in order to rescue a slave, or speak out against hell, or kill a murderer... You don't get a chance to reveal that virtue without making fundamental mistakes about how the universe works, so it is not something to which a rationalist should aspire. But it warms my heart that humans are capable of it.

I have previously spoken of how, to achieve rationality, it is necessary to have some purpose so desperately important to you as to be more important than "rationality", so that you will not choose "rationality" over success.

To learn the Way, you must be able to unlearn the Way; so you must be able to give up the Way; so there must be something dearer to you than the Way. This is so in questions of truth, and in questions of strategy, and also in questions of morality.

The "moral void" of which this post is titled, is not the terrifying abyss of utter meaningless. Which for a bottomless pit is surprisingly shallow; what are you supposed to do about it besides wearing black makeup?

No. The void I'm talking about is a virtue which is nameless.

Sequence: Metaethics

What Would You Do Without Morality?

Created Already In Motion

Referenced by: The Bedrock of Fairness • Rebelling Within Nature • Could Anything Be Right? • Existential Angst Factory • Does Your Morality Care What You Think? • Changing Your Metaethics • The Meaning of Right • Ethical Injunctions • Emotional Involvement

Original with comments: The Moral Void

Created Already In Motion

Eliezer Yudkowsky, 01 July 2008 06:03AM

Followup to: No Universally Compelling Arguments, Passing the Recursive Buck

Lewis Carroll, who was also a mathematician, once wrote a short dialogue called What the Tortoise said to Achilles. If you have not yet read this ancient classic, consider doing so now.

The Tortoise offers Achilles a step of reasoning drawn from Euclid's First Proposition:

(A) Things that are equal to the same are equal to each other.
(B) The two sides of this Triangle are things that are equal to the same.
(Z) The two sides of this Triangle are equal to each other.

Tortoise: "And if some reader had not yet accepted A and B as true, he might still accept the sequence as a valid one, I suppose?"

Achilles: "No doubt such a reader might exist. He might say, 'I accept as true the Hypothetical Proposition that, if A and B be true, Z must be true; but, I don't accept A and B as true.' Such a reader would do wisely in abandoning Euclid, and taking to football."

Tortoise: "And might there not also be some reader who would say, 'I accept A and B as true, but I don't accept the Hypothetical'?"

Achilles, unwisely, concedes this; and so asks the Tortoise to accept another proposition:

(C) If A and B are true, Z must be true.

But, asks, the Tortoise, suppose that he accepts A and B and C, but not Z?

Then, says, Achilles, he must ask the Tortoise to accept one more hypothetical:

(D) If A and B and C are true, Z must be true.

Douglas Hofstadter paraphrased the argument some time later:

Achilles: If you have [(A∧B)→Z], and you also have (A∧B), then surely you have Z.
Tortoise: Oh! You mean <{(A∧B)∧[(A∧B)→Z]}→Z>, don't you?

As Hofstadter says, "Whatever Achilles considers a rule of inference, the Tortoise immediately flattens into a mere string of the system. If you use only the letters A, B, and Z, you will get a recursive pattern of longer and longer strings."

By now you should recognize the anti-pattern Passing the Recursive Buck; and though the counterspell is sometimes hard to find, when found, it generally takes the form The Buck Stops Immediately.

The Tortoise's mind needs the dynamic of adding Y to the belief pool when X and (X→Y) are previously in the belief pool. If this dynamic is not present - a rock, for example, lacks it - then you can go on adding in X and (X→Y) and (X∧(X→Y))→Y until the end of eternity, without ever getting to Y.

The phrase that once came into my mind to describe this requirement, is that a mind must be created already in motion. There is no argument so compelling that it will give dynamics to a static thing. There is no computer program so persuasive that you can run it on a rock.

And even if you have a mind that does carry out modus ponens, it is futile for it to have such beliefs as...

(A) If a toddler is on the train tracks, then pulling them off is fuzzle.
(B) There is a toddler on the train tracks.

...unless the mind also implements:

Dynamic: When the belief pool contains "X is fuzzle", send X to the action system.

(Added: Apparently this wasn't clear... By "dynamic" I mean a property of a physically implemented cognitive system's development over time. A "dynamic" is something that happens inside a cognitive system, not data that it stores in memory and manipulates. Dynamics are the manipulations. There is no way to write a dynamic on a piece of paper, because the paper will just lie there. So the text immediately above, which says "dynamic", is not dynamic. If I wanted the text to be dynamic and not just say "dynamic", I would have to write a Java applet.)

Needless to say, having the belief...

(C) If the belief pool contains "X is fuzzle", then "send 'X' to the action system" is fuzzle.

...won't help unless the mind already implements the behavior of translating hypothetical actions labeled 'fuzzle' into actual motor actions.

By dint of careful arguments about the nature of cognitive systems, you might be able to prove...

(D) A mind with a dynamic that sends plans labeled "fuzzle" to the action system, is more fuzzle than minds that don't.

...but that still won't help, unless the listening mind previously possessed the dynamic of swapping out its current source code for alternative source code that is believed to be more fuzzle.

This is why you can't argue fuzzleness into a rock.

Sequence: Metaethics

The Moral Void

The Bedrock of Fairness

Referenced by: Where Recursive Justification Hits Bottom • Fundamental Doubts • Fake Norms, or "Truth" vs. Truth • Changing Your Metaethics • The Meaning of Right • Morality as Fixed Computation • Inseparably Right; or, Joy in the Merely Good

Original with comments: Created Already In Motion

I'd take it

Eliezer Yudkowsky, 02 July 2008 07:57AM

Out-of-context quote of the day:

"...although even $10 trillion isn't a huge amount of money..."

From Simon Johnson, Director of the IMF's Research Department, on "The Rise of Sovereign Wealth Funds".

So if you had $10 trillion, what would you do with it?

Original with comments: I'd take it

The Bedrock of Fairness

Eliezer Yudkowsky, 03 July 2008 06:00AM

Followup to: The Moral Void

Three people, whom we'll call Xannon, Yancy and Zaire, are separately wandering through the forest; by chance, they happen upon a clearing, meeting each other. Introductions are performed. And then they discover, in the center of the clearing, a delicious blueberry pie.

Xannon: "A pie! What good fortune! But which of us should get it?"

Yancy: "Let us divide it fairly."

Zaire: "I agree; let the pie be distributed fairly. Who could argue against fairness?"

Xannon: "So we are agreed, then. But what is a fair division?"

Yancy: "Eh? Three equal parts, of course!"

Zaire: "Nonsense! A fair distribution is half for me, and a quarter apiece for the two of you."

Yancy: "What? How is that fair?"

Zaire: "I'm hungry, therefore I should be fed; that is fair."

Xannon: "Oh, dear. It seems we have a dispute as to what is fair. For myself, I want to divide the pie the same way as Yancy. But let us resolve this dispute over the meaning of fairness, fairly: that is, giving equal weight to each of our desires. Zaire desires the pie to be divided {1/4, 1/4, 1/2}, and Yancy and I desire the pie to be divided {1/3, 1/3, 1/3}. So the fair compromise is {11/36, 11/36, 14/36}."

Zaire: "What? That's crazy. There's two different opinions as to how fairness works - why should the opinion that happens to be yours, get twice as much weight as the opinion that happens to be mine? Do you think your theory is twice as good? I think my theory is a hundred times as good as yours! So there!"

Yancy: "Craziness indeed. Xannon, I already took Zaire's desires into account in saying that he should get 1/3 of the pie. You can't count the same factor twice. Even if we count fairness as an inherent desire, why should Zaire be rewarded for being selfish? Think about which agents thrive under your system!"

Xannon: "Alas! I was hoping that, even if we could not agree on how to distribute the pie, we could agree on a fair resolution procedure for our dispute, such as averaging our desires together. But even that hope was dashed. Now what are we to do?"

Yancy: "Xannon, you are overcomplicating things. 1/3 apiece. It's not that complicated. A fair distribution is an even split, not a distribution arrived at by a 'fair resolution procedure' that everyone agrees on. What if we'd all been raised in a society that believed that men should get twice as much pie as women? Then we would split the pie unevenly, and even though no one of us disputed the split, it would still be unfair."

Xannon: "What? Where is this 'fairness' stored if not in human minds? Who says that something is unfair if no intelligent agent does so? Not upon the stars or the mountains is 'fairness' written."

Yancy: "So what you're saying is that if you've got a whole society where women are chattel and men sell them like farm animals and it hasn't occurred to anyone that things could be other than they are, that this society is fair, and at the exact moment where someone first realizes it shouldn't have to be that way, the whole society suddenly becomes unfair."

Xannon: "How can a society be unfair without some specific party who claims injury and receives no reparation? If it hasn't occurred to anyone that things could work differently, and no one's asked for things to work differently, then -"

Yancy: "Then the women are still being treated like farm animals and that is unfair. Where's your common sense? Fairness is not agreement, fairness is symmetry."

Zaire: "Is this all working out to my getting half the pie?"

Yancy: "No."

Xannon: "I don't know... maybe as the limit of an infinite sequence of meta-meta-fairnesses..."

Zaire: "I fear I must accord with Yancy on one point, Xannon; your desire for perfect accord among us is misguided. I want half the pie. Yancy wants me to have a third of the pie. This is all there is to the world, and all there ever was. If two monkeys want the same banana, in the end one will have it, and the other will cry morality. Who gets to form the committee to decide the rules that will be used to determine what is 'fair'? Whoever it is, got the banana."

Yancy: "I wanted to give you a third of the pie, and you equate this to seizing the whole thing for myself? Small wonder that you don't want to acknowledge the existence of morality - you don't want to acknowledge that anyone can be so much less of a jerk."

Xannon: "You oversimplify the world, Zaire. Banana-fights occur across thousands and perhaps millions of species, in the animal kingdom. But if this were all there was, Homo sapiens would never have evolved moral intuitions. Why would the human animal evolve to cry morality, if the cry had no effect?"

Zaire: "To make themselves feel better."

Yancy: "Ha! You fail at evolutionary biology."

Xannon: "A murderer accosts a victim, in a dark alley; the murderer desires the victim to die, and the victim desires to live. Is there nothing more to the universe than their conflict? No, because if I happen along, I will side with the victim, and not with the murderer. The victim's plea crosses the gap of persons, to me; it is not locked up inside the victim's own mind. But the murderer cannot obtain my sympathy, nor incite me to help murder. Morality crosses the gap between persons; you might not see it in a conflict between two people, but you would see it in a society."

Yancy: "So you define morality as that which crosses the gap of persons?"

Xannon: "It seems to me that social arguments over disputed goals are how human moral intuitions arose, beyond the simple clash over bananas. So that is how I define the term."

Yancy: "Then I disagree. If someone wants to murder me, and the two of us are alone, then I am still in the right and they are still in the wrong, even if no one else is present."

Zaire: "And the murderer says, 'I am in the right, you are in the wrong'. So what?"

Xannon: "How does your statement that you are in the right, and the murderer is in the wrong, impinge upon the universe - if there is no one else present to be persuaded?"

Yancy: "It licenses me to resist being murdered; which I might not do, if I thought that my desire to avoid being murdered was wrong, and the murderer's desire to kill me was right. I can distinguish between things I merely want, and things that are right - though alas, I do not always live up to my own standards. The murderer is blind to the morality, perhaps, but that doesn't change the morality. And if we were both blind, the morality still would not change."

Xannon: "Blind? What is being seen, what sees it?"

Yancy: "You're trying to treat fairness as... I don't know, something like an array-mapped 2-place function that goes out and eats a list of human minds, and returns a list of what each person thinks is 'fair', and then averages it together. The problem with this isn't just that different people could have different ideas about fairness. It's not just that they could have different ideas about how to combine the results. It's that it leads to infinite recursion outright - passing the recursive buck. You want there to be some level on which everyone agrees, but at least some possible minds will disagree with any statement you make."

Xannon: "Isn't the whole point of fairness to let people agree on a division, instead of fighting over it?"

Yancy: "What is fair is one question, and whether someone else accepts that this is fair is another question. What is fair? That's easy: an equal division of the pie is fair. Anything else won't be fair no matter what kind of pretty arguments you put around it. Even if I gave Zaire a sixth of my pie, that might be a voluntary division but it wouldn't be a fair division. Let fairness be a simple and object-level procedure, instead of this infinite meta-recursion, and the buck will stop immediately."

Zaire: "If the word 'fair' simply means 'equal division' then why not just say 'equal division' instead of this strange additional word, 'fair'? You want the pie divided equally, I want half the pie for myself. That's the whole fact of the matter; this word 'fair' is merely an attempt to get more of the pie for yourself."

Xannon: "If that's the whole fact of the matter, why would anyone talk about 'fairness' in the first place, I wonder?"

Zaire: "Because they all share the same delusion."

Yancy: "A delusion of what? What is it that you are saying people think incorrectly the universe is like?"

Zaire: "I am under no obligation to describe other people's confusions."

Yancy: "If you can't dissolve their confusion, how can you be sure they're confused? But it seems clear enough to me that if the word fair is going to have any meaning at all, it has to finally add up to each of us getting one-third of the pie."

Xannon: "How odd it is to have a procedure of which we are more sure of the result than the procedure itself."

Zaire: "Speak for yourself."

Sequence: Metaethics

Created Already In Motion

Moral Complexities

Referenced by: Moral Complexities • Is Morality Preference? • Rebelling Within Nature • Setting Up Metaethics • The Meaning of Right • Interpersonal Morality • Is Fairness Arbitrary? • That Tiny Note of Discord • What I Think, If Not Why

Original with comments: The Bedrock of Fairness

2 of 10, not 3 total

Eliezer Yudkowsky, 04 July 2008 01:10AM

There is no rule against commenting more than 3 times in a thread. Sorry if anyone has gotten this impression.

However, among the 10 "Recent Comments" visible in the sidebar at right, usually no more than 2, rarely 3, and never 4, should be yours. This is meant to ensure no one person dominates a thread; it gives others a chance to respond to others' responses. One-line comments that quickly correct an error may be common-sensically excepted from this rule.

You need not refrain from commenting, just wait a bit.

Original with comments: 2 of 10, not 3 total

Moral Complexities

Eliezer Yudkowsky, 04 July 2008 06:43AM

Followup to: The Bedrock of Fairness

Discussions of morality seem to me to often end up turning around two different intuitions, which I might label morality-as-preference and morality-as-given. The former crowd tends to equate morality with what people want; the latter to regard morality as something you can't change by changing people.

As for me, I have my own notions, which I am working up to presenting. But above all, I try to avoid avoiding difficult questions. Here are what I see as (some of) the difficult questions for the two intuitions:

- For morality-as-preference:

- Why do people seem to mean different things by "I want the pie" and "It is right that I should get the pie"? Why are the two propositions argued in different ways?

- When and why do people change their terminal values? Do the concepts of "moral error" and "moral progress" have referents? Why would anyone want to change what they want?

- Why and how does anyone ever "do something they know they shouldn't", or "want something they know is wrong"? Does the notion of morality-as-preference really add up to moral normality?

- For morality-as-given:

- Would it be possible for everyone in the world to be wrong about morality, and wrong about how to update their beliefs about morality, and wrong about how to choose between metamoralities, etcetera? So that there would be a morality, but it would be entirely outside our frame of reference? What distinguishes this state of affairs, from finding a random stone tablet showing the words "You should commit suicide"?

- How does a world in which a moral proposition is true, differ from a world in which that moral proposition is false? If the answer is "no", how does anyone perceive moral givens?

- Is it better for people to be happy than sad? If so, why does morality look amazingly like godshatter of natural selection?

- Am I not allowed to construct an alien mind that evaluates morality differently? What will stop me from doing so?

Sequence: Metaethics

The Bedrock of Fairness

Is Morality Preference?

Referenced by: Is Morality Preference? • Setting Up Metaethics

Original with comments: Moral Complexities

Is Morality Preference?

Eliezer Yudkowsky, 05 July 2008 12:55AM

Followup to: Moral Complexities

In the dialogue "The Bedrock of Fairness", I intended Yancy to represent morality-as-raw-fact, Zaire to represent morality-as-raw-whim, and Xannon to be a particular kind of attempt at compromising between them. Neither Xannon, Yancy, or Zaire represent my own views - rather they are, in their disagreement, showing the problem that I am trying to solve. It is futile to present answers to which questions are lacking.

But characters have independent life in the minds of all readers; when I create a dialogue, I don't view my authorial intent as primary. Any good interpretation can be discussed. I meant Zaire to be asking for half the pie out of pure selfishness; many readers interpreted this as a genuine need... which is as interesting a discussion to have as any, though it's a different discussion.

With this in mind, I turn to Subhan and Obert, who shall try to answer yesterday's questions on behalf of their respective viewpoints.

Subhan makes the opening statement:

Subhan: "I defend this proposition: that there is no reason to talk about a 'morality' distinct from what people want."

Obert: "I challenge. Suppose someone comes to me and says, 'I want a slice of that pie you're holding.' It seems to me that they have just made a very different statement from 'It is right that I should get a slice of that pie'. I have no reason at all to doubt the former statement - to suppose that they are lying to me about their desires. But when it comes to the latter proposition, I have reason indeed to be skeptical. Do you say that these two statements mean the same thing?"

Subhan: "I suggest that when the pie-requester says to you, 'It is right for me to get some pie', this asserts that you want the pie-requester to get a slice."

Obert: "Why should I need to be told what I want?"

Subhan: "You take a needlessly restrictive view of wanting, Obert; I am not setting out to reduce humans to creatures of animal instinct. Your wants include those desires you label 'moral values', such as wanting the hungry to be fed -"

Obert: "And you see no distinction between my desire to feed the hungry, and my desire to eat all the delicious pie myself?"

Subhan: "No! They are both desires - backed by different emotions, perhaps, but both desires. To continue, the pie-requester hopes that you have a desire to feed the hungry, and so says, 'It is right that I should get a slice of this pie', to remind you of your own desire. We do not automatically know all the consequences of our own wants; we are not logically omniscient."

Obert: "This seems psychologically unrealistic - I don't think that's what goes through the mind of the person who says, 'I have a right to some pie'. In this latter case, if I deny them pie, they will feel indignant. If they are only trying to remind me of my own desires, why should they feel indignant?"

Subhan: "Because they didn't get any pie, so they're frustrated."

Obert: "Unrealistic! Indignation at moral transgressions has a psychological dimension that goes beyond struggling with a struck door."

Subhan: "Then consider the evolutionary psychology. The pie-requester's emotion of indignation would evolve as a display, first to remind you of the potential consequences of offending fellow tribe-members, and second, to remind any observing tribe-members of goals they may have to feed the hungry. By refusing to share, you would offend against a social norm - which is to say, a widely shared want."

Obert: "So you take refuge in social wants as the essence of morality? But people seem to see a difference between desire and morality, even in the quiet of their own minds. They say things like: 'I want X, but the right thing to do is Y... what shall I do?'"

Subhan: "So they experience a conflict between their want to eat pie, and their want to feed the hungry - which they know is also a want of society. It's not predetermined that the prosocial impulse will be victorious, but they are both impulses."

Obert: "And when, during WWII, a German hides Jews in their basement - against the wants of surrounding society - how then?"

Subhan: "People do not always define their in-group by looking at their next-door neighbors; they may conceive of their group as 'good Christians' or 'humanitarians'."

Obert: "I should sooner say that people choose their in-groups by looking for others who share their beliefs about morality - not that they construct their morality from their in-group."

Subhan: "Oh, really? I should not be surprised if that were experimentally testable - if so, how much do you want to bet?"

Obert: "That the Germans who hid Jews in their basements, chose who to call their people by looking at their beliefs about morality? Sure. I'd bet on that."

Subhan: "But in any case, even if a German resister has a desire to preserve life which is so strong as to go against their own perceived 'society', it is still their desire."

Obert: "Yet they would attribute to that desire, the same distinction they make between 'right' and 'want' - even when going against society. They might think to themselves, 'How dearly I wish I could stay out of this, and keep my family safe. But it is my duty to hide these Jews from the Nazis, and I must fulfill that duty.' There is an interesting moral question, as to whether it reveals greater heroism, to fulfill a duty eagerly, or to fulfill your duties when you are not eager. For myself I should just total up the lives saved, and call that their score. But I digress... The distinction between 'right' and 'want' is not explained by your distinction of socially shared and individual wants. The distinction between desire and duty seems to me a basic thing, which someone could experience floating alone in a spacesuit a thousand light-years from company."

Subhan: "Even if I were to grant this psychological distinction, perhaps that is simply a matter of emotional flavoring. Why should I not describe perceived duties as a differently flavored want?"

Obert: "Duties, and should-ness, seem to have a dimension that goes beyond our whims. If we want different pizza toppings today, we can order a different pizza without guilt; but we cannot choose to make murder a good thing."

Subhan: "Schopenhauer: 'A man can do as he wills, but not will as he wills.' You cannot decide to make salad taste better to you than cheeseburgers, and you cannot decide not to dislike murder. Furthermore, people do change, albeit rarely, those wants that you name 'values'; indeed they are easier to change than our food tastes."

Obert: "Ah! That is something I meant to ask you about. People sometimes change their morals; I would call this updating their beliefs about morality, but you would call it changing their wants. Why would anyone want to change their wants?"

Subhan: "Perhaps they simply find that their wants have changed; brains do change over time. Perhaps they have formed a verbal belief about what they want, which they have discovered to be mistaken. Perhaps society has changed, or their perception of society has changed. But really, in most cases you don't have to go that far, to explain apparent changes of morality."

Obert: "Oh?"

Subhan: "Let's say that someone begins by thinking that Communism is a good social system, has some arguments, and ends by believing that Communism is a bad social system. This does not mean that their ends have changed - they may simply have gotten a good look at the history of Russia, and decided that Communism is a poor means to the end of raising standards of living. I challenge you to find me a case of changing morality in which people change their terminal values, and not just their beliefs about which acts have which consequences."

Obert: "Someone begins by believing that God ordains against premarital sex; they find out there is no God; subsequently they approve of premarital sex. This, let us specify, is not because of fear of Hell; but because previously they believed that God had the power to ordain, or knowledge to tell them, what is right; in ceasing to believe in God, they updated their belief about what is right."

Subhan: "I am not responsible for straightening others' confusions; this one is merely in a general state of disarray around the 'God' concept."

Obert: "All right; suppose I get into a moral argument with a man from a society that practices female circumcision. I do not think our argument is about the consequences to the woman; the argument is about the morality of these consequences."

Subhan: "Perhaps the one falsely believes that women have no feelings -"

Obert: "Unrealistic, unrealistic! It is far more likely that the one hasn't really considered whether the woman has feelings, because he doesn't see any obligation to care. The happiness of women is not a terminal value to him. Thousands of years ago, most societies devalued consequences to women. They also had false beliefs about women, true - and false beliefs about men as well, for that matter - but nothing like the Victorian era's complex rationalizations for how paternalistic rules really benefited women. The Old Testament doesn't explain why it levies the death penalty for a woman wearing men's clothing. It certainly doesn't explain how this rule really benefits women after all. It's not the sort of argument it would have occurred to the authors to rationalize! They didn't care about the consequences to women."

Subhan: "So they wanted different things than you; what of it?"

Obert: "See, now that is exactly why I cannot accept your viewpoint. Somehow, societies went from Old Testament attitudes, to democracies with female suffrage. And this transition - however it occurred - was caused by people saying, 'What this society does to women is a great wrong!', not, 'I would personally prefer to treat women better.' That's not just a change in semantics - it's the difference between being obligated to stand and deliver a justification, versus being able to just say, 'Well, I prefer differently, end of discussion.' And who says that humankind has finished with its moral progress? You're yanking the ladder out from underneath a very important climb."

Subhan: "Let us suppose that the change of human societies over the last ten thousand years, has been accompanied by a change in terminal values -"

Obert: "You call this a supposition? Modern political debates turn around vastly different valuations of consequences than in ancient Greece!"

Subhan: "I am not so sure; human cognitive psychology has not had time to change evolutionarily over that period. Modern democracies tend to appeal to our empathy for those suffering; that empathy existed in ancient Greece as well, but it was invoked less often. In each single moment of argument, I doubt you would find modern politicians appealing to emotions that didn't exist in ancient Greece."

Obert: "I'm not saying that emotions have changed; I'm saying that beliefs about morality have changed. Empathy merely provides emotional depth to an argument that can be made on a purely logical level: 'If it's wrong to enslave you, if it's wrong to enslave your family and your friends, then how can it be right to enslave people who happen to be a different color? What difference does the color make?' If morality is just preference, then there's a very simple answer: 'There is no right or wrong, I just like my own family better.' You see the problem here?"

Subhan: "Logical fallacy: Appeal to consequences."

Obert: "I'm not appealing to consequences. I'm showing that when I reason about 'right' or 'wrong', I am reasoning about something that does not behave like 'want' and 'don't want'."

Subhan: "Oh? But I think that in reality, your rejection of morality-as-preference has a great deal to do with your fear of where the truth leads."

Obert: "Logical fallacy: Ad hominem."

Subhan: "Fair enough. Where were we?"

Obert: "If morality is preference, why would you want to change your wants to be more inclusive? Why would you want to change your wants at all?"

Subhan: "The answer to your first question probably has to do with a fairness instinct, I would suppose - a notion that the tribe should have the same rules for everyone."

Obert: "I don't think that's an instinct. I think that's a triumph of three thousand years of moral philosophy."

Subhan: "That could be tested."

Obert: "And my second question?"

Subhan: "Even if terminal values change, it doesn't mean that terminal values are stored on a great stone tablet outside humanity. Indeed, it would seem to argue against it! It just means that some of the events that go on in our brains, can change what we want."

Obert: "That's your concept of moral progress? That's your view of the last three thousand years? That's why we have free speech, democracy, mass street protests against wars, nonlethal weapons, no more slavery -"

Subhan: "If you wander on a random path, and you compare all past states to your present state, you will see continuous 'advancement' toward your present condition -"

Obert: "Wander on a random path?"

Subhan: "I'm just pointing out that saying, 'Look how much better things are now', when your criterion for 'better' is comparing past moral values to yours, does not establish any directional trend in human progress."

Obert: "Your strange beliefs about the nature of morality have destroyed your soul. I don't even believe in souls, and I'm saying that."

Subhan: "Look, depending on which arguments do, in fact, move us, you might be able to regard the process of changing terminal values as a directional progress. You might be able to show that the change had a consistent trend as we thought of more and more arguments. But that doesn't show that morality is something outside us. We could even - though this is psychologically unrealistic - choose to regard you as computing a converging approximation to your 'ideal wants', so that you would have meta-values that defined both your present value and the rules for updating them. But these would be your meta-values and your ideals and your computation, just as much as pepperoni is your own taste in pizza toppings. You may not know your real favorite ever pizza topping, until you've tasted many possible flavors."

Obert: "Leaving out what it is that you just compared to pizza toppings, I begin to be suspicious of the all-embracingness of your viewpoint. No matter what my mind does, you can simply call it a still-more-modified 'want'. I think that you are the one suffering from meta-level confusion, not I. Appealing to right is not the same as appealing to desire. Just because the appeal is judged inside my brain, doesn't mean that the appeal is not to something more than my desires. Why can't my brain compute duties as well as desires?"

Subhan: "What is the difference between duty and desire?"

Obert: "A duty is something you must do whether you want to or not."

Subhan: "Now you're just being incoherent. Your brain computes something it wants to do whether it wants to or not?"

Obert: "No, you are the one whose theory makes this incoherent. Which is why your theory ultimately fails to add up to morality."

Subhan: "I say again that you underestimate the power of mere wanting. And more: You accuse me of incoherence? You say that I suffer from meta-level confusion?"

Obert: "Er... yes?"

To be continued...

Sequence: Metaethics

Moral Complexities

Is Morality Given?

Referenced by: Is Morality Given? • Whither Moral Progress? • Setting Up Metaethics

Original with comments: Is Morality Preference?

Is Morality Given?

Eliezer Yudkowsky, 06 July 2008 08:12AM

Continuation of: Is Morality Preference?

(Disclaimer: Neither Subhan nor Obert represent my own position on morality; rather they represent different sides of the questions I hope to answer.)

Subhan: "What is this 'morality' stuff, if it is not a preference within you?"

Obert: "I know that my mere wants, don't change what is right; but I don't claim to have absolute knowledge of what is right -"

Subhan: "You're not escaping that easily! How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally? If the answer to that is 'No', then how does any human being come to know that murder is wrong?"

Obert: "Am I allowed to say 'I don't know'?"

Subhan: "No. You believe now that murder is wrong. You must believe you already have evidence and you should be able to present it now."

Obert: "That's too strict! It's like saying to a hunter-gatherer, 'Why is the sky blue?' and expecting an immediate answer."

Subhan: "No, it's like saying to a hunter-gatherer: Why do you believe the sky is blue?"

Obert: "Because it seems blue, just as murder seems wrong. Just don't ask me what the sky is, or how I can see it."

Subhan: "But - aren't we discussing the nature of morality?"

Obert: "That, I confess, is not one of my strong points. I specialize in plain old morality. And as a matter of morality, I know that I can't make murder right just by wanting to kill someone."

Subhan: "But if you wanted to kill someone, you would say, 'I know murdering this guy is right, and I couldn't make it wrong just by not wanting to do it.'"

Obert: "Then, if I said that, I would be wrong. That's common moral sense, right?"

Subhan: "Argh! It's difficult to even argue with you, since you won't tell me exactly what you think morality is made of, or where you're getting all these amazing moral truths -"

Obert: "Well, I do regret having to frustrate you. But it's more important that I act morally, than that I come up with amazing new theories of the nature of morality. I don't claim that my strong point is in explaining the fundamental nature of morality. Rather, my strong point is coming up with theories of morality that give normal moral answers to questions like, 'If you feel like killing someone, does that make it right to do so?' The common-sense answer is 'No' and I really see no reason to adopt a theory that makes the answer 'Yes'. Adding up to moral normality - that is my theory's strong point."

Subhan: "Okay... look. You say that, if you believed it was right to murder someone, you would be wrong."

Obert: "Yes, of course! And just to cut off any quibbles, we'll specify that we're not talking about going back in time and shooting Stalin, but rather, stalking some innocent bystander through a dark alley and slitting their throat for no other reason but my own enjoyment. That's wrong."

Subhan: "And anyone who says murder is right, is mistaken."

Obert: "Yes."

Subhan: "Suppose there's an alien species somewhere in the vastness of the multiverse, who evolved from carnivores. In fact, through most of their evolutionary history, they were cannibals. They've evolved different emotions from us, and they have no concept that murder is wrong -"

Obert: "Why doesn't their society fall apart in an orgy of mutual killing?"

Subhan: "That doesn't matter for our purposes of theoretical metaethical investigation. But since you ask, we'll suppose that the Space Cannibals have a strong sense of honor - they won't kill someone they promise not to kill; they have a very strong idea that violating an oath is wrong. Their society holds together on that basis, and on the basis of vengeance contracts with private assassination companies. But so far as the actual killing is concerned, the aliens just think it's fun. When someone gets executed for, say, driving through a traffic light, there's a bidding war for the rights to personally tear out the offender's throat."

Obert: "Okay... where is this going?"

Subhan: "I'm proposing that the Space Cannibals not only have no sense that murder is wrong - indeed, they have a positive sense that killing is an important part of life - but moreover, there's no path of arguments you could use to persuade a Space Cannibal of your view that murder is wrong. There's no fact the aliens can learn, and no chain of reasoning they can discover, which will ever cause them to conclude that murder is a moral wrong. Nor is there any way to persuade them that they should modify themselves to perceive things differently."

Obert: "I'm not sure I believe that's possible -"

Subhan: "Then you believe in universally compelling arguments processed by a ghost in the machine. For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value -1. A mind is a physical device and you can't have a little blue woman pop out of nowhere and make it say 1 when the physics calls for it to say 0."

Obert: "Suppose I were to concede this. Then?"

Subhan: "Then it's possible to have an alien species that believes murder is not wrong, and moreover, will continue to believe this given knowledge of every possible fact and every possible argument. Can you say these aliens are mistaken?"

Obert: "Maybe it's the right thing to do in their very different, alien world -"

Subhan: "And then they land on Earth and start slitting human throats, laughing all the while, because they don't believe it's wrong. Are they mistaken?"

Obert: "Yes."

Subhan: "Where exactly is the mistake? In which step of reasoning?"

Obert: "I don't know exactly. My guess is that they've got a bad axiom."

Subhan: "Dammit! Okay, look. Is it possible that - by analogy with the Space Cannibals - there are true moral facts of which the human species is not only presently unaware, but incapable of perceiving in principle? Could we have been born defective - incapable even of being compelled by the arguments that would lead us to the light? Moreover, born without any desire to modify ourselves to be capable of understanding such arguments? Could we be irrevocably mistaken about morality - just like you say the Space Cannibals are?"

Obert: "I... guess so..."

Subhan: "You guess so? Surely this is an inevitable consequence of believing that morality is a given, independent of anyone's preferences! Now, is it possible that we, not the Space Cannibals, are the ones who are irrevocably mistaken in believing that murder is wrong?"

Obert: "That doesn't seem likely."

Subhan: "I'm not asking you if it's likely, I'm asking you if it's logically possible! If it's not possible, then you have just confessed that human morality is ultimately determined by our human constitutions. And if it is possible, then what distinguishes this scenario of 'humanity is irrevocably mistaken about morality', from finding a stone tablet on which is written the phrase 'Thou Shalt Murder' without any known justification attached? How is a given morality any different from an unjustified stone tablet?"

Obert: "Slow down. Why does this argument show that morality is determined by our own constitutions?"

Subhan: "Once upon a time, theologians tried to say that God was the foundation of morality. And even since the time of the ancient Greeks, philosophers were sophisticated enough to go on and ask the next question - 'Why follow God's commands?' Does God have knowledge of morality, so that we should follow Its orders as good advice? But then what is this morality, outside God, of which God has knowledge? Do God's commands determine morality? But then why, morally, should one follow God's orders?"

Obert: "Yes, this demolishes attempts to answer questions about the nature of morality just by saying 'God!', unless you answer the obvious further questions. But so what?"

Subhan: "And furthermore, let us castigate those who made the argument originally, for the sin of trying to cast off responsibility - trying to wave a scripture and say, 'I'm just following God's orders!' Even if God had told them to do a thing, it would still have been their own decision to follow God's orders."

Obert: "I agree - as a matter of morality, there is no evading of moral responsibility. Even if your parents, or your government, or some kind of hypothetical superintelligence, tells you to do something, you are responsible for your decision in doing it."

Subhan: "But you see, this also demolishes the idea of any morality that is outside, beyond, or above human preference. Just substitute 'morality' for 'God' in the argument!"

Obert: "What?"

Subhan: "John McCarthy said: 'You say you couldn't live if you thought the world had no purpose. You're saying that you can't form purposes of your own-that you need someone to tell you what to do. The average child has more gumption than that.' For every kind of stone tablet that you might imagine anywhere, in the trends of the universe or in the structure of logic, you are still left with the question: 'And why obey this morality?' It would be your decision to follow this trend of the universe, or obey this structure of logic. Your decision - and your preference."

Obert: "That doesn't follow! Just because it is my decision to be moral - and even because there are drives in me that lead me to make that decision - it doesn't follow that the morality I follow consists merely of my preferences. If someone gives me a pill that makes me prefer to not be moral, to commit murder, then this just alters my preference - but not the morality; murder is still wrong. That's common moral sense -"

Subhan: "I beat my head against my keyboard! What about scientific common sense? If morality is this mysterious given thing, from beyond space and time - and I don't even see why we should follow it, in that case - but in any case, if morality exists independently of human nature, then isn't it a remarkable coincidence that, say, love is good?"

Obert: "Coincidence? How so?"

Subhan: "Just where on Earth do you think the emotion of love comes from? If the ancient Greeks had ever thought of the theory of natural selection, they could have looked at the human institution of sexual romance, or parental love for that matter, and deduced in one flash that human beings had evolved - or at least derived tremendous Bayesian evidence for human evolution. Parental bonds and sexual romance clearly display the signature of evolutionary psychology - they're archetypal cases, in fact, so obvious we usually don't even see it."

Obert: "But love isn't just about reproduction -"

Subhan: "Of course not; individual organisms are adaptation-executers, not fitness-maximizers. But for something independent of humans, morality looks remarkably like godshatter of natural selection. Indeed, it is far too much coincidence for me to credit. Is happiness morally preferable to pain? What a coincidence! And if you claim that there is any emotion, any instinctive preference, any complex brain circuitry in humanity which was created by some external morality thingy and not natural selection, then you are infringing upon science and you will surely be torn to shreds - science has never needed to postulate anything but evolution to explain any feature of human psychology -"

Obert: "I'm not saying that humans got here by anything except evolution."

Subhan: "Then why does morality look so amazingly like a product of an evolved psychology?"

Obert: "I don't claim perfect access to moral truth; maybe, being human, I've made certain mistakes about morality -"

Subhan: "Say that - forsake love and life and happiness, and follow some useless damn trend of the universe or whatever - and you will lose every scrap of the moral normality that you once touted as your strong point. And I will be right here, asking, 'Why even bother?' It would be a pitiful mind indeed that demanded authoritative answers so strongly, that it would forsake all good things to have some authority beyond itself to follow."

Obert: "All right... then maybe the reason morality seems to bear certain similarities to our human constitutions, is that we could only perceive morality at all, if we happened, by luck, to evolve in consonance with it."

Subhan: "Horsemanure."

Obert: "Fine... you're right, that wasn't very plausible. Look, I admit you've driven me into quite a corner here. But even if there were nothing more to morality than preference, I would still prefer to act as morality were real. I mean, if it's all just preference, that way is as good as anything else -"

Subhan: "Now you're just trying to avoid facing reality! Like someone who says, 'If there is no Heaven or Hell, then I may as well still act as if God's going to punish me for sinning.'"

Obert: "That may be a good metaphor, in fact. Consider two theists, in the process of becoming atheists. One says, 'There is no Heaven or Hell, so I may as well cheat and steal, if I can get away without being caught, since there's no God to watch me.' And the other says, 'Even though there's no God, I intend to pretend that God is watching me, so that I can go on being a moral person.' Now they are both mistaken, but the first is straying much further from the path."

Subhan: "And what is the second one's flaw? Failure to accept personal responsibility!"

Obert: "Well, and I admit I find that a more compelling argument than anything else you have said. Probably because it is a moral argument, and it has always been morality, not metaethics, with which I claimed to be concerned. But even so, after our whole conversation, I still maintain that wanting to murder someone does not make murder right. Everything that you have said about preference is interesting, but it is ultimately about preference - about minds and what they are designed to desire - and not about this other thing that humans sometimes talk about, 'morality'. I can just ask Moore's Open Question: Why should I care about human preferences? What makes following human preferences right? By changing a mind, you can change what it prefers; you can even change what it believes to be right; but you cannot change what is right. Anything you talk about, that can be changed in this way, is not 'right-ness'."

Subhan: "So you take refuge in arguing from definitions?"

Obert: "You know, when I reflect on this whole argument, it seems to me that your position has the definite advantage when it comes to arguments about ontology and reality and all that stuff -"

Subhan: "'All that stuff'? What else is there, besides reality?"

Obert: "Okay, the morality-as-preference viewpoint is a lot easier to shoehorn into a universe of quarks. But I still think the morality-as-given viewpoint has the advantage when it comes to, you know, the actual morality part of it - giving answers that are good in the sense of being morally good, not in the sense of being a good reductionist. Because, you know, there are such things as moral errors, there is moral progress, and you really shouldn't go around thinking that murder would be right if you wanted it to be right."

Subhan: "That sounds to me like the logical fallacy of appealing to consequences."

Obert: "Oh? Well, it sounds to me like an incomplete reduction - one that doesn't quite add up to normality."

Sequence: Metaethics

Is Morality Preference?

Where Recursive Justification Hits Bottom

Referenced by: Rebelling Within Nature • The Gift We Give To Tomorrow • Does Your Morality Care What You Think? • Changing Your Metaethics • Setting Up Metaethics • Emotional Involvement

Original with comments: Is Morality Given?

Will As Thou Wilt

Eliezer Yudkowsky, 07 July 2008 10:37AM

Followup to: Possibility and Could-ness

Arthur Schopenhauer (1788-1860) said:

"A man can do as he wills, but not will as he wills."

For this fascinating sentence, I immediately saw two interpretations; and then, after some further thought, two more interpretations.

On the first interpretation, Schopenhauer forbids us to build circular causal models of human psychology. The explanation for someone's current will cannot be their current will - though it can include their past will.

On the second interpretation, the sentence says that alternate choices are not reachable - that we couldn't have taken other options even "if we had wanted to do so".

On the third interpretation, the sentence says that we cannot control our own desires - that we are the prisoners of our own passions, even when we struggle against them.

On the fourth interpretation, the sentence says that we cannot control our own desires, because our desires themselves will determine which desires we want, and so protect themselves.

I count two true interpretations and two false interpretations. How about you?

Original with comments: Will As Thou Wilt

Where Recursive Justification Hits Bottom

Eliezer Yudkowsky, 08 July 2008 10:16AM

Followup to: No Universally Compelling Arguments, Passing the Recursive Buck, Wrong Questions, A Priori

Why do I believe that the Sun will rise tomorrow?

Because I've seen the Sun rise on thousands of previous days.

Ah... but why do I believe the future will be like the past?

Even if I go past the mere surface observation of the Sun rising, to the apparently universal and exceptionless laws of gravitation and nuclear physics, then I am still left with the question: "Why do I believe this will also be true tomorrow?"

I could appeal to Occam's Razor, the principle of using the simplest theory that fits the facts... but why believe in Occam's Razor? Because it's been successful on past problems? But who says that this means Occam's Razor will work tomorrow?

And lo, the one said:

"Science also depends on unjustified assumptions. Thus science is ultimately based on faith, so don't you criticize me for believing in [silly-belief-#238721]."

As I've previously observed:

It's a most peculiar psychology - this business of "Science is based on faith too, so there!" Typically this is said by people who claim that faith is a good thing. Then why do they say "Science is based on faith too!" in that angry-triumphal tone, rather than as a compliment?

Arguing that you should be immune to criticism is rarely a good sign.

But this doesn't answer the legitimate philosophical dilemma: If every belief must be justified, and those justifications in turn must be justified, then how is the infinite recursion terminated?

And if you're allowed to end in something assumed-without-justification, then why aren't you allowed to assume anything without justification?

A similar critique is sometimes leveled against Bayesianism - that it requires assuming some prior - by people who apparently think that the problem of induction is a particular problem of Bayesianism, which you can avoid by using classical statistics. I will speak of this later, perhaps.

But first, let it be clearly admitted that the rules of Bayesian updating, do not of themselves solve the problem of induction.

Suppose you're drawing red and white balls from an urn. You observe that, of the first 9 balls, 3 are red and 6 are white. What is the probability that the next ball drawn will be red?

That depends on your prior beliefs about the urn. If you think the urn-maker generated a uniform random number between 0 and 1, and used that number as the fixed probability of each ball being red, then the answer is 4/11 (by Laplace's Law of Succession). If you think the urn originally contained 10 red balls and 10 white balls, then the answer is 7/11.

Which goes to say that, with the right prior - or rather the wrong prior - the chance of the Sun rising tomorrow, would seem to go down with each succeeding day... if you were absolutely certain, a priori, that there was a great barrel out there from which, on each day, there was drawn a little slip of paper that determined whether the Sun rose or not; and that the barrel contained only a limited number of slips saying "Yes", and the slips were drawn without replacement.

There are possible minds in mind design space who have anti-Occamian and anti-Laplacian priors; they believe that simpler theories are less likely to be correct, and that the more often something happens, the less likely it is to happen again.

And when you ask these strange beings why they keep using priors that never seem to work in real life... they reply, "Because it's never worked for us before!"

Now, one lesson you might derive from this, is "Don't be born with a stupid prior." This is an amazingly helpful principle on many real-world problems, but I doubt it will satisfy philosophers.

Here's how I treat this problem myself: I try to approach questions like "Should I trust my brain?" or "Should I trust Occam's Razor?" as though they were nothing special - or at least, nothing special as deep questions go.

Should I trust Occam's Razor? Well, how well does (any particular version of) Occam's Razor seem to work in practice? What kind of probability-theoretic justifications can I find for it? When I look at the universe, does it seem like the kind of universe in which Occam's Razor would work well?

Should I trust my brain? Obviously not; it doesn't always work. But nonetheless, the human brain seems much more powerful than the most sophisticated computer programs I could consider trusting otherwise. How well does my brain work in practice, on which sorts of problems?

When I examine the causal history of my brain - its origins in natural selection - I find, on the one hand, all sorts of specific reasons for doubt; my brain was optimized to run on the ancestral savanna, not to do math. But on the other hand, it's also clear why, loosely speaking, it's possible that the brain really could work. Natural selection would have quickly eliminated brains so completely unsuited to reasoning, so anti-helpful, as anti-Occamian or anti-Laplacian priors.

So what I did in practice, does not amount to declaring a sudden halt to questioning and justification. I'm not halting the chain of examination at the point that I encounter Occam's Razor, or my brain, or some other unquestionable. The chain of examination continues - but it continues, unavoidably, using my current brain and my current grasp on reasoning techniques. What else could I possibly use?

Indeed, no matter what I did with this dilemma, it would be me doing it. Even if I trusted something else, like some computer program, it would be my own decision to trust it.

The technique of rejecting beliefs that have absolutely no justification, is in general an extremely important one. I sometimes say that the fundamental question of rationality is "Why do you believe what you believe?" I don't even want to say something that sounds like it might allow a single exception to the rule that everything needs justification.

Which is, itself, a dangerous sort of motivation; you can't always avoid everything that might be risky, and when someone annoys you by saying something silly, you can't reverse that stupidity to arrive at intelligence.

But I would nonetheless emphasize the difference between saying:

"Here is this assumption I cannot justify, which must be simply taken, and not further examined."

Versus saying:

"Here the inquiry continues to examine this assumption, with the full force of my present intelligence - as opposed to the full force of something else, like a random number generator or a magic 8-ball - even though my present intelligence happens to be founded on this assumption."

Still... wouldn't it be nice if we could examine the problem of how much to trust our brains without using our current intelligence? Wouldn't it be nice if we could examine the problem of how to think, without using our current grasp of rationality?

When you phrase it that way, it starts looking like the answer might be "No".

E. T. Jaynes used to say that you must always use all the information available to you - he was a Bayesian probability theorist, and had to clean up the paradoxes other people generated when they used different information at different points in their calculations. The principle of "Always put forth your true best effort" has at least as much appeal as "Never do anything that might look circular." After all, the alternative to putting forth your best effort is presumably doing less than your best.

But still... wouldn't it be nice if there were some way to justify using Occam's Razor, or justify predicting that the future will resemble the past, without assuming that those methods of reasoning which have worked on previous occasions are better than those which have continually failed?

Wouldn't it be nice if there were some chain of justifications that neither ended in an unexaminable assumption, nor was forced to examine itself under its own rules, but, instead, could be explained starting from absolute scratch to an ideal philosophy student of perfect emptiness?

Well, I'd certainly be interested, but I don't expect to see it done any time soon. I've argued elsewhere in several places against the idea that you can have a perfectly empty ghost-in-the-machine; there is no argument that you can explain to a rock.

Even if someone cracks the First Cause problem and comes up with the actual reason the universe is simple, which does not itself presume a simple universe... then I would still expect that the explanation could only be understood by a mindful listener, and not by, say, a rock. A listener that didn't start out already implementing modus ponens might be out of luck.

So, at the end of the day, what happens when someone keeps asking me "Why do you believe what you believe?"

At present, I start going around in a loop at the point where I explain, "I predict the future as though it will resemble the past on the simplest and most stable level of organization I can identify, because previously, this rule has usually worked to generate good results; and using the simple assumption of a simple universe, I can see why it generates good results; and I can even see how my brain might have evolved to be able to observe the universe with some degree of accuracy, if my observations are correct."

But then... haven't I just licensed circular logic?

Actually, I've just licensed reflecting on your mind's degree of trustworthiness, using your current mind as opposed to something else.

Reflection of this sort is, indeed, the reason we reject most circular logic in the first place. We want to have a coherent causal story about how our mind comes to know something, a story that explains how the process we used to arrive at our beliefs, is itself trustworthy. This is the essential demand behind the rationalist's fundamental question, "Why do you believe what you believe?"

Now suppose you write on a sheet of paper: "(1) Everything on this sheet of paper is true, (2) The mass of a helium atom is 20 grams." If that trick actually worked in real life, you would be able to know the true mass of a helium atom just by believing some circular logic which asserted it. Which would enable you to arrive at a true map of the universe sitting in your living room with the blinds drawn. Which would violate the second law of thermodynamics by generating information from nowhere. Which would not be a plausible story about how your mind could end up believing something true.

Even if you started out believing the sheet of paper, it would not seem that you had any reason for why the paper corresponded to reality. It would just be a miraculous coincidence that (a) the mass of a helium atom was 20 grams, and (b) the paper happened to say so.

Believing, in general, self-validating statement sets, does not seem like it should work to map external reality - when we reflect on it as a causal story about minds - using, of course, our current minds to do so.

But what about evolving to give more credence to simpler beliefs, and to believe that algorithms which have worked in the past are more likely to work in the future? Even when we reflect on this as a causal story of the origin of minds, it still seems like this could plausibly work to map reality.

And what about trusting reflective coherence in general? Wouldn't most possible minds, randomly generated and allowed to settle into a state of reflective coherence, be incorrect? Ah, but we evolved by natural selection; we were not generated randomly.

If trusting this argument seems worrisome to you, then forget about the problem of philosophical justifications, and ask yourself whether it's really truly true.

(You will, of course, use your own mind to do so.)

Is this the same as the one who says, "I believe that the Bible is the word of God, because the Bible says so"?

Couldn't they argue that their blind faith must also have been placed in them by God, and is therefore trustworthy?

In point of fact, when religious people finally come to reject the Bible, they do not do so by magically jumping to a non-religious state of pure emptiness, and then evaluating their religious beliefs in that non-religious state of mind, and then jumping back to a new state with their religious beliefs removed.

People go from being religious, to being non-religious, because even in a religious state of mind, doubt seeps in. They notice their prayers (and worse, the prayers of seemingly much worthier people) are not being answered. They notice that God, who speaks to them in their heart in order to provide seemingly consoling answers about the universe, is not able to tell them the hundredth digit of pi (which would be a lot more reassuring, if God's purpose were reassurance). They examine the story of God's creation of the world and damnation of unbelievers, and it doesn't seem to make sense even under their own religious premises.

Being religious doesn't make you less than human. Your brain still has the abilities of a human brain. The dangerous part is that being religious might stop you from applying those native abilities to your religion - stop you from reflecting fully on yourself. People don't heal their errors by resetting themselves to an ideal philosopher of pure emptiness and reconsidering all their sensory experiences from scratch. They heal themselves by becoming more willing to question their current beliefs, using more of the power of their current mind.

This is why it's important to distinguish between reflecting on your mind using your mind (it's not like you can use anything else) and having an unquestionable assumption that you can't reflect on.

"I believe that the Bible is the word of God, because the Bible says so." Well, if the Bible were an astoundingly reliable source of information about all other matters, if it had not said that grasshoppers had four legs or that the universe was created in six days, but had instead contained the Periodic Table of Elements centuries before chemistry - if the Bible had served us only well and told us only truth - then we might, in fact, be inclined to take seriously the additional statement in the Bible, that the Bible had been generated by God. We might not trust it entirely, because it could also be aliens or the Dark Lords of the Matrix, but it would at least be worth taking seriously.

Likewise, if everything else that priests had told us, turned out to be true, we might take more seriously their statement that faith had been placed in us by God and was a systematically trustworthy source - especially if people could divine the hundredth digit of pi by faith as well.

So the important part of appreciating the circularity of "I believe that the Bible is the word of God, because the Bible says so," is not so much that you are going to reject the idea of reflecting on your mind using your current mind. But, rather, that you realize that anything which calls into question the Bible's trustworthiness, also calls into question the Bible's assurance of its trustworthiness.

This applies to rationality too: if the future should cease to resemble the past - even on its lowest and simplest and most stable observed levels of organization - well, mostly, I'd be dead, because my brain's processes require a lawful universe where chemistry goes on working. But if somehow I survived, then I would have to start questioning the principle that the future should be predicted to be like the past.

But for now... what's the alternative to saying, "I'm going to believe that the future will be like the past on the most stable level of organization I can identify, because that's previously worked better for me than any other algorithm I've tried"?

Is it saying, "I'm going to believe that the future will not be like the past, because that algorithm has always failed before"?

At this point I feel obliged to drag up the point that rationalists are not out to win arguments with ideal philosophers of perfect emptiness; we are simply out to win. For which purpose we want to get as close to the truth as we can possibly manage. So at the end of the day, I embrace the principle: "Question your brain, question your intuitions, question your principles of rationality, using the full current force of your mind, and doing the best you can do at every point."

If one of your current principles does come up wanting - according to your own mind's examination, since you can't step outside yourself - then change it! And then go back and look at things again, using your new improved principles.

The point is not to be reflectively consistent. The point is to win. But if you look at yourself and play to win, you are making yourself more reflectively consistent - that's what it means to "play to win" while "looking at yourself".

Everything, without exception, needs justification. Sometimes - unavoidably, as far as I can tell - those justifications will go around in reflective loops. I do think that reflective loops have a meta-character which should enable one to distinguish them, by common sense, from circular logics. But anyone seriously considering a circular logic in the first place, is probably out to lunch in matters of rationality; and will simply insist that their circular logic is a "reflective loop" even if it consists of a single scrap of paper saying "Trust me". Well, you can't always optimize your rationality techniques according to the sole consideration of preventing those bent on self-destruction from abusing them.

The important thing is to hold nothing back in your criticisms of how to criticize; nor should you regard the unavoidability of loopy justifications as a warrant of immunity from questioning.

Always apply full force, whether it loops or not - do the best you can possibly do, whether it loops or not - and play, ultimately, to win.

Sequence: Metaethics

Is Morality Given?

My Kind of Reflection

Referenced by: My Kind of Reflection • Fundamental Doubts • Rebelling Within Nature • Could Anything Be Right? • Changing Your Metaethics • Setting Up Metaethics • The Meaning of Right • "Arbitrary" • Is Fairness Arbitrary? • You Provably Can't Trust Yourself • No License To Be Human • Mirrors and Paintings • High Challenge • The Sin of Underconfidence

Original with comments: Where Recursive Justification Hits Bottom

The Fear of Common Knowledge

Eliezer Yudkowsky, 09 July 2008 09:48AM

Followup to: Belief in Belief

One of those insights that made me sit upright and say "Aha!" From The Uncredible Hallq:

Minor acts of dishonesty are integral to human life, ranging from how we deal with casual acquaintances to writing formal agreements between nation states. Steven Pinker has an excellent chapter on this in The Stuff of Thought, a version of which can be found at TIME magazine’s website. What didn’t make it into the TIME version is Pinker’s proposal that, while there are several reasons we do this, the most important reason is to avoid mutual knowledge: "She probably knows I just blew a pass at her, but does she know I know she knows? Does she know I know she knows I know she knows?" Etc. Mutual knowledge is that nightmare where, for all intents and purposes, the known-knows can be extended out to infinity. The ultimate example of this has to be the joke "No, it wasn’t awkward until you said, 'well, this is awkward.'" A situation might be a little awkward, but what’s really awkward is mutual knowledge, created when someone blurts out what’s going on for all to hear...

The story of the Emperor’s New Clothes is another example of the power of mutual knowledge...

The power of real deception - outright lies - is easy for even us nerds to understand.

The notion of a lie that the other person knows is a lie, seems very odd at first. Up until I read the Hallq's explanation of Pinker, I had thought in terms of people suppressing uncomfortable thoughts: "If it isn't said out loud, I don't have to deal with it."

Like the friends of a terminal patient, whose disease has progressed to a stage that - if you look it up online - turns out to be nearly universally fatal. So the friends gather around, and wish the patient best hopes for their medical treatment. No one says, "Well, we all know you're going to die; and now it's too late for you to get life insurance and sign up for cryonics. I hope it isn't too painful; let me know if you want me to smuggle you a heroin overdose."

So even that is possible for a nerd to understand - in terms of, as Vassar puts it, thinking of non-nerds as defective nerds...

But the notion of a lie that the other person knows is a lie, but they aren't sure that you know they know it's a lie, and so the social situation occupies a different state from common knowledge...

I think that's the closest I've ever seen life get to imitating a Raymond Smullyan logic puzzle.

Added: Richard quotes Nagel on a further purpose of mutual hypocrisy: preventing an issue from rising to the level where it must be publicly acknowledged and dealt with, because common ground on that issue is not easily available.

Original with comments: The Fear of Common Knowledge

My Kind of Reflection

Eliezer Yudkowsky, 10 July 2008 07:21AM

Followup to: Where Recursive Justification Hits Bottom

In "Where Recursive Justification Hits Bottom", I concluded that it's okay to use induction to reason about the probability that induction will work in the future, given that it's worked in the past; or to use Occam's Razor to conclude that the simplest explanation for why Occam's Razor works is that the universe itself is fundamentally simple.

Now I am far from the first person to consider reflective application of reasoning principles. Chris Hibbert compared my view to Bartley's Pan-Critical Rationalism (I was wondering whether that would happen). So it seems worthwhile to state what I see as the distinguishing features of my view of reflection, which may or may not happen to be shared by any other philosopher's view of reflection.

• All of my philosophy here actually comes from trying to figure out how to build a self-modifying AI that applies its own reasoning principles to itself in the process of rewriting its own source code. So whenever I talk about using induction to license induction, I'm really thinking about an inductive AI considering a rewrite of the part of itself that performs induction. If you wouldn't want the AI to rewrite its source code to not use induction, your philosophy had better not label induction as unjustifiable.

• One of the most powerful general principles I know for AI in general, is that the true Way generally turns out to be naturalistic - which for reflective reasoning, means treating transistors inside the AI, just as if they were transistors found in the environment; not an ad-hoc special case. This is the real source of my insistence in "Recursive Justification" that questions like "How well does my version of Occam's Razor work?" should be considered just like an ordinary question - or at least an ordinary very deep question. I strongly suspect that a correctly built AI, in pondering modifications to the part of its source code that implements Occamian reasoning, will not have to do anything special as it ponders - in particular, it shouldn't have to make a special effort to avoid using Occamian reasoning.

• I don't think that "reflective coherence" or "reflective consistency" should be considered as a desideratum in itself. As I said in the Twelve Virtues and the Simple Truth, if you make five accurate maps of the same city, then the maps will necessarily be consistent with each other; but if you draw one map by fantasy and then make four copies, the five will be consistent but not accurate. In the same way, no one is deliberately pursuing reflective consistency, and reflective consistency is not a special warrant of trustworthiness; the goal is to win. But anyone who pursues the goal of winning, using their current notion of winning, and modifying their own source code, will end up reflectively consistent as a side effect - just like someone continually striving to improve their map of the world should find the parts becoming more consistent among themselves, as a side effect. If you put on your AI goggles, then the AI, rewriting its own source code, is not trying to make itself "reflectively consistent" - it is trying to optimize the expected utility of its source code, and it happens to be doing this using its current mind's anticipation of the consequences.

• One of the ways I license using induction and Occam's Razor to consider "induction" and "Occam's Razor", is by appealing to E. T. Jaynes's principle that we should always use all the information available to us (computing power permitting) in a calculation. If you think induction works, then you should use it in order to use your maximum power, including when you're thinking about induction.

• In general, I think it's valuable to distinguish a defensive posture where you're imagining how to justify your philosophy to a philosopher that questions you, from an aggressive posture where you're trying to get as close to the truth as possible. So it's not that being suspicious of Occam's Razor, but using your current mind and intelligence to inspect it, shows that you're being fair and defensible by questioning your foundational beliefs. Rather, the reason why you would inspect Occam's Razor is to see if you could improve your application of it, or if you're worried it might really be wrong. I tend to deprecate mere dutiful doubts.

• If you run around inspecting your foundations, I expect you to actually improve them, not just dutifully investigate. Our brains are built to assess "simplicity" in a certain intuitive way that makes Thor sound simpler than Maxwell's Equations as an explanation for lightning. But, having gotten a better look at the way the universe really works, we've concluded that differential equations (which few humans master) are actually simpler (in an information-theoretic sense) than heroic mythology (which is how most tribes explain the universe). This being the case, we've tried to import our notions of Occam's Razor into math as well.

• On the other hand, the improved foundations should still add up to normality; 2 + 2 should still end up equalling 4, not something new and amazing and exciting like "fish".

• I think it's very important to distinguish between the questions "Why does induction work?" and "Does induction work?" The reason why the universe itself is regular is still a mysterious question unto us, for now. Strange speculations here may be temporarily needful. But on the other hand, if you start claiming that the universe isn't actually regular, that the answer to "Does induction work?" is "No!", then you're wandering into 2 + 2 = 3 territory. You're trying too hard to make your philosophy interesting, instead of correct. An inductive AI asking what probability assignment to make on the next round is asking "Does induction work?", and this is the question that it may answer by inductive reasoning. If you ask "Why does induction work?" then answering "Because induction works" is circular logic, and answering "Because I believe induction works" is magical thinking.

• I don't think that going around in a loop of justifications through the meta-level is the same thing as circular logic. I think the notion of "circular logic" applies within the object level, and is something that is definitely bad and forbidden, on the object level. Forbidding reflective coherence doesn't sound like a good idea. But I haven't yet sat down and formalized the exact difference - my reflective theory is something I'm trying to work out, not something I have in hand.

Sequence: Metaethics

Where Recursive Justification Hits Bottom

The Genetic Fallacy

Referenced by: Fundamental Doubts • Rebelling Within Nature • Probability is Subjectively Objective • Changing Your Metaethics • Setting Up Metaethics

Original with comments: My Kind of Reflection

The Genetic Fallacy

Eliezer Yudkowsky, 11 July 2008 05:47AM

In lists of logical fallacies, you will find included "the genetic fallacy" - the fallacy attacking a belief, based on someone's causes for believing it.

This is, at first sight, a very strange idea - if the causes of a belief do not determine its systematic reliability, what does? If Deep Blue advises us of a chess move, we trust it based on our understanding of the code that searches the game tree, being unable to evaluate the actual game tree ourselves. What could license any probability assignment as "rational", except that it was produced by some systematically reliable process?

Articles on the genetic fallacy will tell you that genetic reasoning is not always a fallacy - that the origin of evidence can be relevant to its evaluation, as in the case of a trusted expert. But other times, say the articles, it is a fallacy; the chemist Kekulé first saw the ring structure of benzene in a dream, but this doesn't mean we can never trust this belief.

So sometimes the genetic fallacy is a fallacy, and sometimes it's not?

The genetic fallacy is formally a fallacy, because the original cause of a belief is not the same as its current justificational status, the sum of all the support and antisupport currently known.

Yet we change our minds less often than we think. Genetic accusations have a force among humans that they would not have among ideal Bayesians.

Clearing your mind is a powerful heuristic when you're faced with new suspicion that many of your ideas may have come from a flawed source.

Once an idea gets into our heads, it's not always easy for evidence to root it out. Consider all the people out there who grew up believing in the Bible; later came to reject (on a deliberate level) the idea that the Bible was written by the hand of God; and who nonetheless think that the Bible contains indispensable ethical wisdom. They have failed to clear their minds; they could do significantly better by doubting anything the Bible said because the Bible said it.

At the same time, they would have to bear firmly in mind the principle that reversed stupidity is not intelligence; the goal is to genuinely shake your mind loose and do independent thinking, not to negate the Bible and let that be your algorithm.

Once an idea gets into your head, you tend to find support for it everywhere you look - and so when the original source is suddenly cast into suspicion, you would be very wise indeed to suspect all the leaves that originally grew on that branch...

If you can! It's not easy to clear your mind. It takes a convulsive effort to actually reconsider, instead of letting your mind fall into the pattern of rehearsing cached arguments. "It ain't a true crisis of faith unless things could just as easily go either way," said Thor Shenkel.

You should be extremely suspicious if you have many ideas suggested by a source that you now know to be untrustworthy, but by golly, it seems that all the ideas still ended up being right - the Bible being the obvious archetypal example.

On the other hand... there's such a thing as sufficiently clear-cut evidence, that it no longer significantly matters where the idea originally came from. Accumulating that kind of clear-cut evidence is what Science is all about. It doesn't matter any more that Kekulé first saw the ring structure of benzene in a dream - it wouldn't matter if we'd found the hypothesis to test by generating random computer images, or from a spiritualist revealed as a fraud, or even from the Bible. The ring structure of benzene is pinned down by enough experimental evidence to make the source of the suggestion irrelevant.

In the absence of such clear-cut evidence, then you do need to pay attention to the original sources of ideas - to give experts more credence than layfolk, if their field has earned respect - to suspect ideas you originally got from suspicious sources - to distrust those whose motives are untrustworthy, if they cannot present arguments independent of their own authority.

The genetic fallacy is a fallacy when there exist justifications beyond the genetic fact asserted, but the genetic accusation is presented as if it settled the issue.

Some good rules of thumb (for humans):

- Be suspicious of genetic accusations against beliefs that you dislike, especially if the proponent claims justifications beyond the simple authority of a speaker. "Flight is a religious idea, so the Wright Brothers must be liars" is one of the classically given examples.

- By the same token, don't think you can get good information about a technical issue just by sagely psychoanalyzing the personalities involved and their flawed motives. If technical arguments exist, they get priority.

- When new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves that grew from that root. You are not licensed to reject them outright as conclusions, because reversed stupidity is not intelligence, but...

- Be extremely suspicious if you find that you still believe the early suggestions of a source you later rejected.

Added: Hal Finney suggests that we should call it "the genetic heuristic".

Sequence: Seeing with Fresh Eyes

Lonely Dissent

Sequence: Metaethics

My Kind of Reflection

Fundamental Doubts

Referenced by: Fundamental Doubts • Rebelling Within Nature • Changing Your Metaethics • Fighting a Rearguard Action Against the Truth • Crisis of Faith • Dark Side Epistemology • Formative Youth • That Crisis thing seems pretty useful

Original with comments: The Genetic Fallacy

Fundamental Doubts

Eliezer Yudkowsky, 12 July 2008 05:21AM

Followup to: The Genetic Fallacy, Where Recursive Justification Hits Bottom

Yesterday I said that - because humans are not perfect Bayesians - the genetic fallacy is not entirely a fallacy; when new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves of that root, even if they seem to have accumulated new evidence in the meanwhile.

This is one of the most difficult techniques of rationality (on which I will separately post, one of these days). Descartes, setting out to "doubt, insofar as possible, all things", ended up trying to prove the existence of God - which, if he wasn't a secret atheist trying to avoid getting burned at the stake, is pretty pathetic. It is hard to doubt an idea to which we are deeply attached; our mind naturally reaches for cached thoughts and rehearsed arguments.

But today's post concerns a different kind of difficulty - the case where the doubt is so deep, of a source so fundamental, that you can't make a true fresh beginning.

Case in point: Remember when, in the The Matrix, Morpheus told Neo that the machines were harvesting the body heat of humans for energy, and liquefying the dead to feed to babies? I suppose you thought something like, "Hey! That violates the second law of thermodynamics."

Well, it does violate the second law of thermodynamics. But if the Matrix's makers had cared about the flaw once it was pointed out to them, they could have fixed the plot hole in any of the sequels, in fifteen seconds, this easily:

Neo: "Doesn't harvesting human body heat for energy, violate the laws of thermodynamics?"

Morpheus: "Where'd you learn about thermodynamics, Neo?"

Neo: "In school."

Morpheus: "Where'd you go to school, Neo?"

Neo: "Oh."

Morpheus: "The machines tell elegant lies."

Now, mind you, I am not saying that this excuses the original mistake in the script. When my mind generated this excuse, it came clearly labeled with that warning sign of which I have spoken, "Tada! Your mind can generate an excuse for anything!" You do not need to tell me that my plot-hole-patch is a nitwit idea, I am well aware of that...

...but, in point of fact, if you woke up out of a virtual reality pod one day, you would have to suspect all the physics you knew. Even if you looked down and saw that you had hands, you couldn't rely on there being blood and bone inside them. Even if you looked up and saw stars, you couldn't rely on their being trillions of miles away. And even if you found yourself thinking, you couldn't rely on your head containing a brain.

You could still try to doubt, even so. You could do your best to unwind your thoughts past every lesson in school, every science paper read, every sensory experience, every math proof whose seeming approval by other mathematicians might have been choreographed to conceal a subtle flaw...

But suppose you discovered that you were a computer program and that the Dark Lords of the Matrix were actively tampering with your thoughts.

Well... in that scenario, you're pretty much screwed, I'd have to say.

Descartes vastly underestimated the powers of an infinitely powerful deceiving demon when he supposed he could trust "I think therefore I am." Maybe that's just what they want you to think. Maybe they just inserted that conclusion into your mind with a memory of it seeming to have an irrefutable chain of logical support, along with some peer pressure to label it "unquestionable" just like all your friends.

(Personally, I don't trust "I think therefore I am" even in real life, since it contains a term "am" whose meaning I find confusing, and I've learned to spread my confidence intervals very widely in the presence of basic confusion. As for absolute certainty, don't be silly.)

Every memory of justification could be faked. Every feeling of support could be artificially induced. Modus ponens could be a lie. Your concept of "rational justification" - not just your specific concept, but your notion that any such thing exists at all - could have been manufactured to mislead you. Your trust in Reason itself could have been inculcated to throw you off the trail.

So you might as well not think about the possibility that you're a brain with choreographed thoughts, because there's nothing you can do about it...

Unless, of course, that's what they want you to think.

Past a certain level of doubt, it's not possible to start over fresh. There's nothing you can unassume to find some firm rock on which to stand. You cannot unwind yourself into a perfectly empty and perfectly reliable ghost in the machine.

This level of meta-suspicion should be a rare occasion. For example, suspecting that all academic science is an organized conspiracy, should not run into anything like these meta-difficulties. Certainly, someone does not get to plead that unwinding past the Bible is impossible because it is too foundational; atheists walk the Earth without falling into comas. Remember, when Descartes tried to outwit an infinitely powerful deceiving demon, he first tried to make himself absolutely certain of a highly confusing statement, and then proved the existence of God. Consider that a caution about what you try to claim is "too basic for a fresh beginning". And even basic things can still be doubted, it is only that we use our untrustworthy brains to doubt them.

Or consider the case of our existence as evolved brains. Natural selection isn't trustworthy, and we have specific reason to suspect it. We know that evolution is stupid. We know many specific ways in which our human brains fail, taken beyond the savanna. But you can't clear your mind of evolutionary influences and start over. It would be like deciding that you don't trust neurons, so you're going to clear your mind of brains.

And evolution certainly gets a chance to influence every single thought that runs through your mind! It is the very reason why you exist as a thinker, rather than a lump of carbon - and that doesn't mean evolution summoned a ghost-in-the-machine into you; it designed the ghost. If you learn culture, it is because you were built to learn culture.

But in fact, we don't run into unmanageable meta-trouble in trying to come up with specific patches for specific known evolved biases. And evolution is stupid, so even though it has set up self-deceptive circuits in us, these circuits are not infinitely difficult to comprehend and outwit.

Or so it seems! But it really does seem that way, on reflection.

There is no button you can press to rewind past your noisy brain, and become a perfectly reliable ghost of perfect emptiness. That's not just because your brain is you. It's also because you can't unassume things like modus ponens or belief updating. You can unassume them as explicit premises for deliberate reasoning - a hunter-gatherer has no explicit concept of modus ponens - but you can't delete the actual dynamics (and all their products!)

So, in the end, I think we must allow the use of brains to think about thinking; and the use of evolved brains to think about evolution; and the use of inductive brains to think about induction; and the use of brains with an Occam prior to think about whether the universe appears to be simple; for these things we really cannot unwind entirely, even when we have reason to distrust them. Strange loops through the meta level, I think, are not the same as circular logic.

Sequence: Metaethics

The Genetic Fallacy

Rebelling Within Nature

Referenced by: Rebelling Within Nature

Original with comments: Fundamental Doubts

Rebelling Within Nature

Eliezer Yudkowsky, 13 July 2008 12:32PM

Followup to: Fundamental Doubts, Where Recursive Justification Hits Bottom, No Universally Compelling Arguments, Joy in the Merely Real, Evolutionary Psychology

"Let us understand, once and for all, that the ethical progress of society depends, not on imitating the cosmic process, still less in running away from it, but in combating it."
-- T. H. Huxley ("Darwin's bulldog", early advocate of evolutionary theory)

There is a quote from some Zen Master or other, who said something along the lines of:

"Western man believes that he is rebelling against nature, but he does not realize that, in doing so, he is acting according to nature."

The Reductionist Masters of the West, strong in their own Art, are not so foolish; they do realize that they always act within Nature.

You can narrow your focus and rebel against a facet of existing Nature - polio, say - but in so doing, you act within the whole of Nature. The syringe that carries the polio vaccine is forged of atoms; our minds, that understood the method, embodied in neurons. If Jonas Salk had to fight laziness, he fought something that evolution instilled in him - a reluctance to work that conserves energy. And he fought it with other emotions that natural selection also inscribed in him: feelings of friendship that he extended to humanity, heroism to protect his tribe, maybe an explicit desire for fame that he never acknowledged to himself - who knows? (I haven't actually read a biography of Salk.)

The point is, you can't fight Nature from beyond Nature, only from within it. There is no acausal fulcrum on which to stand outside reality and move it. There is no ghost of perfect emptiness by which you can judge your brain from outside your brain. You can fight the cosmic process, but only by recruiting other abilities that evolution originally gave to you.

And if you fight one emotion within yourself - looking upon your own nature, and judging yourself less than you think should be - saying perhaps, "I should not want to kill my enemies" - then you make that judgment, by...

How exactly does one go about rebelling against one's own goal system?

From within it, naturally.

This is perhaps the primary thing that I didn't quite understand as a teenager.

At the age of fifteen (fourteen?), I picked up a copy of TIME magazine and read an article on evolutionary psychology. It seemed like one of the most massively obvious-in-retrospect ideas I'd ever heard. I went on to read The Moral Animal by Robert Wright. And later The Adapted Mind - but from the perspective of personal epiphanies, The Moral Animal pretty much did the job.

I'm reasonably sure that if I had not known the basics of evolutionary psychology from my teenage years, I would not currently exist as the Eliezer Yudkowsky you know.

Indeed, let me drop back a bit further:

At the age of... I think it was nine... I discovered the truth about sex by looking it up in my parents' home copy of the Encyclopedia Britannica (stop that laughing). Shortly after, I learned a good deal more by discovering where my parents had hidden the secret 15th volume of my long-beloved Childcraft series. I'd been avidly reading the first 14 volumes - some of them, anyway - since the age of five. But the 15th volume wasn't meant for me- it was the "Guide for Parents".

The 15th volume of Childcraft described the life cycle of children. It described the horrible confusion of the teenage years - teenagers experimenting with alcohol, with drugs, with unsafe sex, with reckless driving, the hormones taking over their minds, the overwhelming importance of peer pressure, the tearful accusations of "You don't love me!" and "I hate you!"

I took one look at that description, at the tender age of nine, and said to myself in quiet revulsion, I'm not going to do that.

And I didn't.

My teenage years were not untroubled. But I didn't do any of the things that the Guide to Parents warned me against. I didn't drink, drive, drug, lose control to hormones, pay any attention to peer pressure, or ever once think that my parents didn't love me.

In a safer world, I would have wished for my parents to have hidden that book better.

But in this world, which needs me as I am, I don't regret finding it.

I still rebelled, of course. I rebelled against the rebellious nature the Guide to Parents described to me. That was part of how I defined my identity in my teenage years - "I'm not doing the standard stupid stuff." Some of the time, this just meant that I invented amazing new stupidity, but in fact that was a major improvement.

Years later, The Moral Animal made suddenly obvious the why of all that disastrous behavior I'd been warned against. Not that Robert Wright pointed any of this out explicitly, but it was obvious given the elementary concept of evolutionary psychology:

Physiologically adult humans are not meant to spend an additional 10 years in a school system; their brains map that onto "I have been assigned low tribal status". And so, of course, they plot rebellion - accuse the existing tribal overlords of corruption - plot perhaps to split off their own little tribe in the savanna, not realizing that this is impossible in the Modern World. The teenage males map their own fathers onto the role of "tribal chief"...

Echoes in time, thousands of repeated generations in the savanna carving the pattern, ancient repetitions of form, reproduced in the present in strange twisted mappings, across genes that didn't know anything had changed...

The world grew older, of a sudden.

And I'm not going to go into the evolutionary psychology of "teenagers" in detail, not now, because that would deserve its own post.

But when I read The Moral Animal, the world suddenly acquired causal depth. Human emotions existed for reasons, they weren't just unexamined givens. I might previously have questioned whether an emotion was appropriate to its circumstance - whether it made sense to hate your parents, if they did really love you - but I wouldn't have thought, before then, to judge the existence of hatred as an evolved emotion.

And then, having come so far, and having avoided with instinctive ease all the classic errors that evolutionary psychologists are traditionally warned against - I was never once tempted to confuse evolutionary causation with psychological causation - I went wrong at the last turn.

The echo in time that was teenage psychology was obviously wrong and stupid - a distortion in the way things should be - so clearly you were supposed to unwind past it, compensate in the opposite direction or disable the feeling, to arrive at the correct answer.

It's hard for me to remember exactly what I was thinking in this era, but I think I tended to focus on one facet of human psychology at any given moment, trying to unwind myself a piece at a time. IIRC I did think, in full generality, "Evolution is bad; the effect it has on psychology is bad." (Like it had some kind of "effect" that could be isolated!) But somehow, I managed not to get to "Evolutionary psychology is the cause of altruism; altruism is bad."

It was easy for me to see all sorts of warped altruism as having been warped by evolution.

People who wanted to trust themselves with power, for the good of their tribe - that had an obvious evolutionary explanation; it was, therefore, a distortion to be corrected.

People who wanted to be altruistic in ways their friends would approve of - obvious evolutionary explanation; therefore a distortion to be corrected.

People who wanted to be altruistic in a way that would optimize their fame and repute - obvious evolutionary distortion to be corrected.

People who wanted to help only their family, or only their nation - acting out ancient selection pressures on the savanna; move past it.

But the fundamental will to help people?

Well, the notion of that being merely evolved, was something that, somehow, I managed to never quite accept. Even though, in retrospect, the causality is just as obvious as teen revolutionism.

IIRC, I did think something along the lines of: "Once you unwind past evolution, then the true morality isn't likely to contain a clause saying, 'This person matters but this person doesn't', so everyone should matter equally, so you should be as eager to help others as help yourself." And so I thought that even if the emotion of altruism had merely evolved, it was a right emotion, and I should keep it.

But why think that people mattered at all, if you were trying to unwind past all evolutionary psychology? Why think that it was better for people to be happy than sad, rather than the converse?

If I recall correctly, I did ask myself that, and sort of waved my hands mentally and said, "It just seems like one of the best guesses - I mean, I don't know that people are valuable, but I can't think of what else could be."

This is the Avoiding Your Belief's Real Weak Points / Not Spontaneously Thinking About Your Belief's Most Painful Weaknesses antipattern in full glory: Get just far enough to place yourself on the first fringes of real distress, and then stop thinking.

And also the antipattern of trying to unwind past everything that is causally responsible for your existence as a mind, to arrive at a perfectly reliable ghost of perfect emptiness.

Later, having also seen others making similar mistakes, it seems to me that the general problem is an illusion of mind-independence that comes from picking something that appeals to you, while still seeming philosophically simple.

As if the appeal to you, of the moral argument, weren't still a feature of your particular point in mind design space.

As if there weren't still an ordinary and explicable causal history behind the appeal, and your selection of that particular principle.

As if, by making things philosophically simpler-seeming, you could enhance their appeal to a ghost-in-the-machine who would hear your justifications starting from scratch, as fairness demands.

As if your very sense of simplicity were not an aesthetic sense inscribed in you by evolution.

As if your very intuitions of "moral argument" and "justification", were not an architecture-of-reasoning inscribed in you by natural selection, and just as causally explicable as any other feature of human psychology...

You can't throw away evolution, and end up with a perfectly moral creature that humans would have been, if only we had never evolved; that's really not how it works.

Why accept intuitively appealing arguments about the nature of morality, rather than intuitively unappealing ones, if you're going to distrust everything in you that ever evolved?

Then what is right? What should we do, having been inscribed by a blind mad idiot god whose incarnation-into-reality takes the form of millions of years of ancestral murder and war?

But even this question - every fragment of it - the notion that a blind mad idiocy is an ugly property for a god to have, or that murder is a poisoned well of order, even the words "right" and "should" - all a phenomenon within nature. All traceable back to debates built around arguments appealing to intuitions that evolved in me.

You can't jump out of the system. You really can't. Even wanting to jump out of the system - the sense that something isn't justified "just because it evolved" - is something that you feel from within the system. Anything you might try to use to jump - any sense of what morality should be like, if you could unwind past evolution - is also there as a causal result of evolution.

Not everything we think about morality is directly inscribed by evolution, of course. We have values that we got from our parents teaching them to us as we grew up; after it won out in a civilizational debate conducted with reference to other moral principles; that were themselves argued into existence by appealing to built-in emotions; using an architecture-of-interpersonal-moral-argument that evolution burped into existence.

It all goes back to evolution. This doesn't just include things like instinctive concepts of fairness, or empathy, it includes the whole notion of arguing morals as if they were propositional beliefs. Evolution created within you that frame of reference within which you can formulate the concept of moral questioning. Including questioning evolution's fitness to create our moral frame of reference. If you really try to unwind outside the system, you'll unwind your unwinders.

That's what I didn't quite get, those years ago.

I do plan to dissolve the cognitive confusion that makes words like "right" and "should" seem difficult to grasp. I've been working up to that for a while now.

But I'm not there yet, and so, for now, I'm going to jump ahead and peek at an answer I'll only later be able to justify as moral philosophy:

Embrace reflection. You can't unwind to emptiness, but you can bootstrap from a starting point.

Go on morally questioning the existence (and not just appropriateness) of emotions. But don't treat the mere fact of their having evolved as a reason to reject them. Yes, I know that "X evolved" doesn't seem like a good justification for having an emotion; but don't let that be a reason to reject X, any more than it's a reason to accept it. Hence the post on the Genetic Fallacy: causation is conceptually distinct from justification. If you try to apply the Genetic Accusation to automatically convict and expel your genes, you're going to run into foundational trouble - so don't!

Just ask if the emotion is justified - don't treat its evolutionary cause as proof of mere distortion. Use your current mind to examine the emotion's pluses and minuses, without being ashamed; use your full strength of morality.

Judge emotions as emotions, not as evolutionary relics. When you say, "motherly love outcompeted its alternative alleles because it protected children that could carry the allele for motherly love", this is only a cause, not a sum of all moral arguments. The evolutionary psychology may grant you helpful insight into the pattern and process of motherly love, but it neither justifies the emotion as natural, nor convicts it as coming from an unworthy source. You don't make the Genetic Accusation either way. You just, y'know, think about motherly love, and ask yourself if it seems like a good thing or not; considering its effects, not its source.

You tot up the balance of moral justifications, using your current mind - without worrying about the fact that the entire debate takes place within an evolved framework.

That's the moral normality to which my yet-to-be-revealed moral philosophy will add up.

And if, in the meanwhile, it seems to you like I've just proved that there is no morality... well, I haven't proved any such thing. But, meanwhile, just ask yourself if you might want to help people even if there were no morality. If you find that the answer is yes, then you will later discover that you discovered morality.

Sequence: Metaethics

Fundamental Doubts

Probability is Subjectively Objective

Referenced by: The Gift We Give To Tomorrow • Could Anything Be Right? • Does Your Morality Care What You Think? • Changing Your Metaethics • The Meaning of Right • Detached Lever Fallacy • Against Maturity

Original with comments: Rebelling Within Nature

Probability is Subjectively Objective

Eliezer Yudkowsky, 14 July 2008 09:16AM

Followup to: Probability is in the Mind

"Reality is that which, when you stop believing in it, doesn't go away."
-- Philip K. Dick

There are two kinds of Bayesians, allegedly. Subjective Bayesians believe that "probabilities" are degrees of uncertainty existing in our minds; if you are uncertain about a phenomenon, that is a fact about your state of mind, not a property of the phenomenon itself; probability theory constrains the logical coherence of uncertain beliefs. Then there are objective Bayesians, who... I'm not quite sure what it means to be an "objective Bayesian"; there are multiple definitions out there. As best I can tell, an "objective Bayesian" is anyone who uses Bayesian methods and isn't a subjective Bayesian.

If I recall correctly, E. T. Jaynes, master of the art, once described himself as a subjective-objective Bayesian. Jaynes certainly believed very firmly that probability was in the mind; Jaynes was the one who coined the term Mind Projection Fallacy. But Jaynes also didn't think that this implied a license to make up whatever priors you liked. There was only one correct prior distribution to use, given your state of partial information at the start of the problem.

How can something be in the mind, yet still be objective?

It appears to me that a good deal of philosophical maturity consists in being able to keep separate track of nearby concepts, without mixing them up.

For example, to understand evolutionary psychology, you have to keep separate track of the psychological purpose of an act, and the evolutionary pseudo-purposes of the adaptations that execute as the psychology; this is a common failure of newcomers to evolutionary psychology, who read, misunderstand, and thereafter say, "You think you love your children, but you're just trying to maximize your fitness!"

What is it, exactly, that the terms "subjective" and "objective", mean? Let's say that I hand you a sock. Is it a subjective or an objective sock? You believe that 2 + 3 = 5. Is your belief subjective or objective? What about two plus three actually equaling five - is that subjective or objective? What about a specific act of adding two apples and three apples and getting five apples?

I don't intend to confuse you in shrouds of words; but I do mean to point out that, while you may feel that you know very well what is "subjective" or "objective", you might find that you have a bit of trouble saying out loud what those words mean.

Suppose there's a calculator that computes "2 + 3 = 5". We punch in "2", then "+", then "3", and lo and behold, we see "5" flash on the screen. We accept this as evidence that 2 + 3 = 5, but we wouldn't say that the calculator's physical output defines the answer to the question 2 + 3 = ?. A cosmic ray could strike a transistor, which might give us misleading evidence and cause us to believe that 2 + 3 = 6, but it wouldn't affect the actual sum of 2 + 3.

Which proposition is common-sensically true, but philosophically interesting: while we can easily point to the physical location of a symbol on a calculator screen, or observe the result of putting two apples on a table followed by another three apples, it is rather harder to track down the whereabouts of 2 + 3 = 5. (Did you look in the garage?)

But let us leave aside the question of where the fact 2 + 3 = 5 is located - in the universe, or somewhere else - and consider the assertion that the proposition is "objective". If a cosmic ray strikes a calculator and makes it output "6" in response to the query "2 + 3 = ?", and you add two apples to a table followed by three apples, then you'll still see five apples on the table. If you do the calculation in your own head, expending the necessary computing power - we assume that 2 + 3 is a very difficult sum to compute, so that the answer is not immediately obvious to you - then you'll get the answer "5". So the cosmic ray strike didn't change anything.

And similarly - exactly similarly - what if a cosmic ray strikes a neuron inside your brain, causing you to compute "2 + 3 = 7"? Then, adding two apples to three apples, you will expect to see seven apples, but instead you will be surprised to see five apples.

If instead we found that no one was ever mistaken about addition problems, and that, moreover, you could change the answer by an act of will, then we might be tempted to call addition "subjective" rather than "objective". I am not saying that this is everything people mean by "subjective" and "objective", just pointing to one aspect of the concept. One might summarize this aspect thus: "If you can change something by thinking differently, it's subjective; if you can't change it by anything you do strictly inside your head, it's objective."

Mind is not magic. Every act of reasoning that we human beings carry out, is computed within some particular human brain. But not every computation is about the state of a human brain. Not every thought that you think is about something that can be changed by thinking. Herein lies the opportunity for confusion-of-levels. The quotation is not the referent. If you are going to consider thoughts as referential at all - if not, I'd like you to explain the mysterious correlation between my thought "2 + 3 = 5" and the observed behavior of apples on tables - then, while the quoted thoughts will always change with thoughts, the referents may or may not be entities that change with changing human thoughts.

The calculator computes "What is 2 + 3?", not "What does this calculator compute as the result of 2 + 3?" The answer to the former question is 5, but if the calculator were to ask the latter question instead, the result could self-consistently be anything at all! If the calculator returned 42, then indeed, "What does this calculator compute as the result of 2 + 3?" would in fact be 42.

So just because a computation takes place inside your brain, does not mean that the computation explicitly mentions your brain, that it has your brain as a referent, any more than the calculator mentions the calculator. The calculator does not attempt to contain a representation of itself, only of numbers.

Indeed, in the most straightforward implementation, the calculator that asks "What does this calculator compute as the answer to the query 2 + 3 = ?" will never return a result, just simulate itself simulating itself until it runs out of memory.

But if you punch the keys "2", "+", and "3", and the calculator proceeds to compute "What do I output when someone punches '2 + 3'?", the resulting computation does have one interesting characteristic: the referent of the computation is highly subjective, since it depends on the computation, and can be made to be anything just by changing the computation.

Is probability, then, subjective or objective?

Well, probability is computed within human brains or other calculators. A probability is a state of partial information that is possessed by you; if you flip a coin and press it to your arm, the coin is showing heads or tails, but you assign the probability 1/2 until you reveal it. A friend, who got a tiny but not fully informative peek, might assign a probability of 0.6.

So can you make the probability of winning the lottery be anything you like?

Forget about many-worlds for the moment - you should almost always be able to forget about many-worlds - and pretend that you're living in a single Small World where the lottery has only a single outcome. You will nonetheless have a need to call upon probability. Or if you prefer, we can discuss the ten trillionth decimal digit of pi, which I believe is not yet known. (If you are foolish enough to refuse to assign a probability distribution to this entity, you might pass up an excellent bet, like betting $1 to win $1000 that the digit is not 4.) Your uncertainty is a state of your mind, of partial information that you possess. Someone else might have different information, complete or partial. And the entity itself will only ever take on a single value.

So can you make the probability of winning the lottery, or the probability of the ten trillionth decimal digit of pi equaling 4, be anything you like?

You might be tempted to reply: "Well, since I currently think the probability of winning the lottery is one in a hundred million, then obviously, I will currently expect that assigning any other probability than this to the lottery, will decrease my expected log-score - or if you prefer a decision-theoretic formulation, I will expect this modification to myself to decrease expected utility. So, obviously, I will not choose to modify my probability distribution. It wouldn't be reflectively coherent."

So reflective coherency is the goal, is it? Too bad you weren't born with a prior that assigned probability 0.9 to winning the lottery! Then, by exactly the same line of argument, you wouldn't want to assign any probability except 0.9 to winning the lottery. And you would still be reflectively coherent. And you would have a 90% probability of winning millions of dollars! Hooray!

"No, then I would think I had a 90% probability of winning the lottery, but actually, the probability would only be one in a hundred million."

Well, of course you would be expected to say that. And if you'd been born with a prior that assigned 90% probability to your winning the lottery, you'd consider an alleged probability of 10^-8, and say, "No, then I would think I had almost no probability of winning the lottery, but actually, the probability would be 0.9."

"Yeah? Then just modify your probability distribution, and buy a lottery ticket, and then wait and see what happens."

What happens? Either the ticket will win, or it won't. That's what will happen. We won't get to see that some particular probability was, in fact, the exactly right probability to assign.

"Perform the experiment a hundred times, and -"

Okay, let's talk about the ten trillionth digit of pi, then. Single-shot problem, no "long run" you can measure.

Probability is subjectively objective: Probability exists in your mind: if you're ignorant of a phenomenon, that's an attribute of you, not an attribute of the phenomenon. Yet it will seem to you that you can't change probabilities by wishing.

You could make yourself compute something else, perhaps, rather than probability. You could compute "What do I say is the probability?" (answer: anything you say) or "What do I wish were the probability?" (answer: whatever you wish) but these things are not the probability, which is subjectively objective.

The thing about subjectively objective quantities is that they really do seem objective to you. You don't look them over and say, "Oh, well, of course I don't want to modify my own probability estimate, because no one can just modify their probability estimate; but if I'd been born with a different prior I'd be saying something different, and I wouldn't want to modify that either; and so none of us is superior to anyone else." That's the way a subjectively subjective quantity would seem.

No, it will seem to you that, if the lottery sells a hundred million tickets, and you don't get a peek at the results, then the probability of a ticket winning, is one in a hundred million. And that you could be born with different priors but that wouldn't give you any better odds. And if there's someone next to you saying the same thing about their 90% probability estimate, you'll just shrug and say, "Good luck with that." You won't expect them to win.

Probability is subjectively really objective, not just subjectively sort of objective.

Jaynes used to recommend that no one ever write out an unconditional probability: That you never, ever write simply P(A), but always write P(A|I), where I is your prior information. I'll use Q instead of I, for ease of reading, but Jaynes used I. Similarly, one would not write P(A|B) for the posterior probability of A given that we learn B, but rather P(A|B,Q), the probability of A given that we learn B and had background information Q.

This is good advice in a purely pragmatic sense, when you see how many false "paradoxes" are generated by accidentally using different prior information in different places.

But it also makes a deep philosophical point as well, which I never saw Jaynes spell out explicitly, but I think he would have approved: there is no such thing as a probability that isn't in any mind. Any mind that takes in evidence and outputs probability estimates of the next event, remember, can be viewed as a prior - so there is no probability without priors/minds.

You can't unwind the Q. You can't ask "What is the unconditional probability of our background information being true, P(Q)?" To make that estimate, you would still need some kind of prior. No way to unwind back to an ideal ghost of perfect emptiness...

You might argue that you and the lottery-ticket buyer do not really have a disagreement about probability. You say that the probability of the ticket winning the lottery is one in a hundred million given your prior, P(W|Q1) = 10^-8. The other fellow says the probability of the ticket winning given his prior is P(W|Q2) = 0.9. Every time you say "The probability of X is Y", you really mean, "P(X|Q1) = Y". And when he says, "No, the probability of X is Z", he really means, "P(X|Q2) = Z".

Now you might, if you traced out his mathematical calculations, agree that, indeed, the conditional probability of the ticket winning, given his weird prior is 0.9. But you wouldn't agree that "the probability of the ticket winning" is 0.9. Just as he wouldn't agree that "the probability of the ticket winning" is 10^-8.

Even if the two of you refer to different mathematical calculations when you say the word "probability", you don't think that puts you on equal ground, neither of you being better than the other. And neither does he, of course.

So you see that, subjectively, probability really does feel objective - even after you have subjectively taken all apparent subjectivity into account.

And this is not mistaken, because, by golly, the probability of winning the lottery really is 10^-8, not 0.9. It's not as if you're doing your probability calculation wrong, after all. If you weren't worried about being fair or about justifying yourself to philosophers, if you only wanted to get the correct answer, your betting odds would be 10^-8.

Somewhere out in mind design space, there's a mind with any possible prior; but that doesn't mean that you'll say, "All priors are created equal."

When you judge those alternate minds, you'll do so using your own mind - your own beliefs about the universe - your own posterior that came out of your own prior, your own posterior probability assignments P(X|A,B,C,...,Q1). But there's nothing wrong with that. It's not like you could judge using something other than yourself. It's not like you could have a probability assignment without any prior, a degree of uncertainty that isn't in any mind.

And so, when all that is said and done, it still seems like the probability of winning the lottery really is 10^-8, not 0.9. No matter what other minds in design space say differently.

Which shouldn't be surprising. When you compute probabilities, you're thinking about lottery balls, not thinking about brains or mind designs or other people with different priors. Your probability computation makes no mention of that, any more than it explicitly represents itself. Your goal, after all, is to win, not to be fair. So of course probability will seem to be independent of what other minds might think of it.

Okay, but... you still can't win the lottery by assigning a higher probability to winning.

If you like, we could regard probability as an idealized computation, just like 2 + 2 = 4 seems to be independent of any particular error-prone calculator that computes it; and you could regard your mind as trying to approximate this ideal computation. In which case, it is good that your mind does not mention people's opinions, and only thinks of the lottery balls; the ideal computation makes no mention of people's opinions, and we are trying to reflect this ideal as accurately as possible...

But what you will calculate is the "ideal calculation" to plug into your betting odds, will depend on your prior, even though the calculation won't have an explicit dependency on "your prior". Someone who thought the universe was anti-Occamian, would advocate an anti-Occamian calculation, regardless of whether or not anyone thought the universe was anti-Occamian.

Your calculations get checked against reality, in a probabilistic way; you either win the lottery or not. But interpreting these results, is done with your prior; once again there is no probability that isn't in any mind.

I am not trying to argue that you can win the lottery by wishing, of course. Rather, I am trying to inculcate the ability to distinguish between levels.

When you think about the ontological nature of probability, and perform reductionism on it - when you try to explain how "probability" fits into a universe in which states of mind do not exist fundamentally - then you find that probability is computed within a brain; and you find that other possible minds could perform mostly-analogous operations with different priors and arrive at different answers.

But, when you consider probability as probability, think about the referent instead of the thought process - which thinking you will do in your own thoughts, which are physical processes - then you will conclude that the vast majority of possible priors are probably wrong. (You will also be able to conceive of priors which are, in fact, better than yours, because they assign more probability to the actual outcome; you just won't know in advance which alternative prior is the truly better one.)

If you again swap your goggles to think about how probability is implemented in the brain, the seeming objectivity of probability is the way the probability algorithm feels from inside; so it's no mystery that, considering probability as probability, you feel that it's not subject to your whims. That's just what the probability-computation would be expected to say, since the computation doesn't represent any dependency on your whims.

But when you swap out those goggles and go back to thinking about probabilities, then, by golly, your algorithm seems to be right in computing that probability is not subject to your whims. You can't win the lottery just by changing your beliefs about it. And if that is the way you would be expected to feel, then so what? The feeling has been explained, not explained away; it is not a mere feeling. Just because a calculation is implemented in your brain, doesn't mean it's wrong, after all.

Your "probability that the ten trillionth decimal digit of pi is 4", is an attribute of yourself, and exists in your mind; the real digit is either 4 or not. And if you could change your belief about the probability by editing your brain, you wouldn't expect that to change the probability.

Therefore I say of probability that it is "subjectively objective".

Sequence: Metaethics

Rebelling Within Nature

Whither Moral Progress?

Referenced by: Can Counterfactuals Be True? • Math is Subjunctively Objective • Changing Your Metaethics • The Meaning of Right • You Provably Can't Trust Yourself

Original with comments: Probability is Subjectively Objective

Lawrence Watt-Evans's Fiction

Eliezer Yudkowsky, 15 July 2008 03:00AM

One of my pet topics, on which I will post more one of these days, is the Rationalist in Fiction. Most of the time - it goes almost without saying - the Rationalist is done completely wrong. In Hollywood, the Rationalist is a villain, or a cold emotionless foil, or a child who has to grow into a real human being, or a fool whose probabilities are all wrong, etcetera. Even in science fiction, the Rationalist character is rarely done right - bearing the same resemblance to a real rationalist, as the mad scientist genius inventor who designs a new nuclear reactor in a month, bears to real scientists and engineers.

Perhaps this is because most speculative fiction, generally speaking, is interested in someone battling monsters or falling in love or becoming a vampire, or whatever, not in being rational... and it would probably be worse fiction, if the author tried to make that the whole story. But that can't be the entire problem. I've read at least one author whose plots are not about rationality, but whose characters are nonetheless, in passing, realistically rational.

That author is Lawrence Watt-Evans. His work stands out for a number of reasons, the first being that it is genuinely unpredictable. Not because of a postmodernist contempt for coherence, but because there are events going on outside the hero's story, just like real life.

Most authors, if they set up a fantasy world with a horrible evil villain, and give their main character the one sword that can kill that villain, you could guess that, at the end of the book, the main character is going to kill the evil villain with the sword.

Not Lawrence Watt-Evans. In a Watt-Evans book, it's entirely possible that the evil villain will die of a heart attack halfway through the book, then the character will decide to sell the sword because they'd rather have the money, and then the character uses the money to set up an investment banking company.

That didn't actually happen in any particular Watt-Evans book - I don't believe in spoilers - but it gives you something of the flavor.

And Watt-Evans doesn't always do this, either - just as, even in real life, things sometimes do go as you expect.

It's this strange realism that charms me about Watt-Evans's work. It's not done as a schtick, but as faithfulness-to-reality. Real life doesn't run on perfect rails of dramatic necessity; neither is it random postmodern chaos. I admire an author who can be faithful to that, and still tell a story.

Watt-Evans's characters, if they happen to be rationalists, are realistic rationalists - they think the same things that you or I would, in their situations.

If the character gets catapulted into a fantasy world, they actually notice the resemblance to their fantasy books, wonder about it, and think to themselves, "If this were a fantasy book, the next thing that would happen is X..." (which may or may not happen, because Watt-Evans doesn't write typical fantasy books). It's not done as a postmodern self-referential schtick, but as a faithfulness-to-reality; they think what a real rational person would think, in their shoes.

If the character finds out that it is their destiny to destroy the world, they don't waste time on immense dramatic displays - after they get over the shock, they land on their feet and start thinking about it in more or less the fashion that you or I would in their shoes. Not just, "How do I avoid this? Are there any possibilities I've overlooked?" but also "Am I sure this is really what's going on? How reliable is this information?"

If a Watt-Evans character gets their hands on a powerful cheat, they are going to exploit it to the fullest and actively think about creative new ways to use it. If they find a staff of healing, they're going to set up a hospital. If they invent a teleportation spell, they're going to think about new industrial uses.

I hate it when some artifact of world-cracking power is introduced and then used as a one-time plot device. Eventually you get numb, though.

But if a Watt-Evans character finds a device with some interesting magical power halfway through the book, and there's some clever way that the magical power can be used to take over the world, and that character happens to want to take over the world, she's going to say the hell with whatever she was previously doing and go for it.

Most fictional characters are stupid, because they have to be. This occurs for several reasons; but in speculative fiction, a primary reason is that the author wants to throw around wish-fulfillment superpowers, and the author isn't competent enough to depict the real consequences of halfway intelligent people using that power.

Lawrence Watt-Evans's stories aren't about the intelligence or rationality of his characters. Nonetheless, Watt-Evans writes intelligent characters, and he's willing to deal with the consequences, which are huge.

Maybe that's the main reason we don't see many realistic rationalists in fiction.

I'd like to see more rationalist fiction. Not necessarily in Watt-Evans's exact vein, because we already have Watt-Evans, and thus, there is no need to invent him. But rationalist fiction is hard to do well; there are plenty of cliches out there, but few depictions that say something new or true.

Lawrence Watt-Evans is not going to be everyone's cuppa tea, but if you like SF&F already, give it a try. Suggested starting books: The Unwilling Warlord (fantasy), Denner's Wreck (SF).

Referenced by: Rationalist Fiction

Original with comments: Lawrence Watt-Evans's Fiction

Posting May Slow

Eliezer Yudkowsky, 16 July 2008 03:00AM

Greetings, fearless readers:

Due to the Oxford conference on Global Catastrophic Risk, I may miss some posts - possibly quite a few.

Or possibly not.

Just so you don't think I'm dead.

Sincerely,
Eliezer.

Original with comments: Posting May Slow

Whither Moral Progress?

Eliezer Yudkowsky, 16 July 2008 05:04AM

Followup to: Is Morality Preference?

In the dialogue "Is Morality Preference?", Obert argues for the existence of moral progress by pointing to free speech, democracy, mass street protests against wars, the end of slavery... and we could also cite female suffrage, or the fact that burning a cat alive was once a popular entertainment... and many other things that our ancestors believed were right, but which we have come to see as wrong, or vice versa.

But Subhan points out that if your only measure of progress is to take a difference against your current state, then you can follow a random walk, and still see the appearance of inevitable progress.

One way of refuting the simplest version of this argument, would be to say that we don't automatically think ourselves the very apex of possible morality; that we can imagine our descendants being more moral than us.

But can you concretely imagine a being morally wiser than yourself - one who knows that some particular thing is wrong, when you believe it to be right?

Certainly: I am not sure of the moral status of chimpanzees, and hence I find it easy to imagine that a future civilization will label them definitely people, and castigate us for failing to cryopreserve the chimpanzees who died in human custody.

Yet this still doesn't prove the existence of moral progress. Maybe I am simply mistaken about the nature of changes in morality that have previously occurred - like looking at a time chart of "differences between past and present", noting that the difference has been steadily decreasing, and saying, without being able to visualize it, "Extrapolating this chart into the future, we find that the future will be even less different from the present than the present."

So let me throw the question open to my readers: Whither moral progress?

You might say, perhaps, "Over time, people have become more willing to help one another - that is the very substance and definition of moral progress."

But as John McCarthy put it:

"If everyone were to live for others all the time, life would be like a procession of ants following each other around in a circle."

Once you make "People helping each other more" the definition of moral progress, then people helping each other all the time, is by definition the apex of moral progress.

At the very least we have Moore's Open Question: It is not clear that helping others all the time is automatically moral progress, whether or not you argue that it is; and so we apparently have some notion of what constitutes "moral progress" that goes beyond the direct identification with "helping others more often".

Or if you identify moral progress with "Democracy!", then at some point there was a first democratic civilization - at some point, people went from having no notion of democracy as a good thing, to inventing the idea of democracy as a good thing. If increasing democracy is the very substance of moral progress, then how did this moral progress come about to exist in the world? How did people invent, without knowing it, this very substance of moral progress?

It's easy to come up with concrete examples of moral progress. Just point to a moral disagreement between past and present civilizations; or point to a disagreement between yourself and present civilization, and claim that future civilizations might agree with you.

It's harder to answer Subhan's challenge - to show directionality, rather than a random walk, on the meta-level. And explain how this directionality is implemented, on the meta-level: how people go from not having a moral ideal, to having it.

(I have my own ideas about this, as some of you know. And I'll thank you not to link to them in the comments, or quote them and attribute them to me, until at least 24 hours have passed from this post.)

Sequence: Metaethics

Probability is Subjectively Objective

The Gift We Give To Tomorrow

Referenced by: Setting Up Metaethics • The Meaning of Right • Inseparably Right; or, Joy in the Merely Good • Lawful Creativity

Original with comments: Whither Moral Progress?

The Gift We Give To Tomorrow

Eliezer Yudkowsky, 17 July 2008 06:07AM

Followup to: Thou Art Godshatter, Joy in the Merely Real, Is Morality Given?, Rebelling Within Nature

How, oh how, did an unloving and mindless universe, cough up minds who were capable of love?

"No mystery in that," you say, "it's just a matter of natural selection."

But natural selection is cruel, bloody, and bloody stupid. Even when, on the surface of things, biological organisms aren't directly fighting each other - aren't directly tearing at each other with claws - there's still a deeper competition going on between the genes. Genetic information is created when genes increase their relative frequency in the next generation - what matters for "genetic fitness" is not how many children you have, but that you have more children than others. It is quite possible for a species to evolve to extinction, if the winning genes are playing negative-sum games.

How, oh how, could such a process create beings capable of love?

"No mystery," you say, "there is never any mystery-in-the-world; mystery is a property of questions, not answers. A mother's children share her genes, so the mother loves her children."

But sometimes mothers adopt children, and still love them. And mothers love their children for themselves, not for their genes.

"No mystery," you say, "Individual organisms are adaptation-executers, not fitness-maximizers. Evolutionary psychology is not about deliberately maximizing fitness - through most of human history, we didn't know genes existed. We don't calculate our acts' effect on genetic fitness consciously, or even subconsciously."

But human beings form friendships even with non-relatives: how, oh how, can it be?

"No mystery, for hunter-gatherers often play Iterated Prisoner's Dilemmas, the solution to which is reciprocal altruism. Sometimes the most dangerous human in the tribe is not the strongest, the prettiest, or even the smartest, but the one who has the most allies."

Yet not all friends are fair-weather friends; we have a concept of true friendship - and some people have sacrificed their life for their friends. Would not such a devotion tend to remove itself from the gene pool?

"You said it yourself: we have a concept of true friendship and fair-weather friendship. We can tell, or try to tell, the difference between someone who considers us a valuable ally, and someone executing the friendship adaptation. We wouldn't be true friends with someone who we didn't think was a true friend to us - and someone with many true friends is far more formidable than someone with many fair-weather allies."

And Mohandas Gandhi, who really did turn the other cheek? Those who try to serve all humanity, whether or not all humanity serves them in turn?

"That perhaps is a more complicated story. Human beings are not just social animals. We are political animals who argue linguistically about policy in adaptive tribal contexts. Sometimes the formidable human is not the strongest, but the one who can most skillfully argue that their preferred policies match the preferences of others."

Um... that doesn't explain Gandhi, or am I missing something?

"The point is that we have the ability to argue about 'What should be done?' as a proposition - we can make those arguments and respond to those arguments, without which politics could not take place."

Okay, but Gandhi?

"Believed certain complicated propositions about 'What should be done?' and did them."

That sounds like it could explain any possible human behavior.

"If we traced back the chain of causality through all the arguments, it would involve: a moral architecture that had the ability to argue general abstract moral propositions like 'What should be done to people?'; appeal to hardwired intuitions like fairness, a concept of duty, pain aversion + empathy; something like a preference for simple moral propositions, probably reused from our previous Occam prior; and the end result of all this, plus perhaps memetic selection effects, was 'You should not hurt people' in full generality -"

And that gets you Gandhi.

"Unless you think it was magic, it has to fit into the lawful causal development of the universe somehow."

Well... I certainly won't postulate magic, under any name.

"Good."

But come on... doesn't it seem a little... amazing... that hundreds of millions of years worth of evolution's death tournament could cough up mothers and fathers, sisters and brothers, husbands and wives, steadfast friends and honorable enemies, true altruists and guardians of causes, police officers and loyal defenders, even artists sacrificing themselves for their art, all practicing so many kinds of love? For so many things other than genes? Doing their part to make their world less ugly, something besides a sea of blood and violence and mindless replication?

"Are you claiming to be surprised by this? If so, question your underlying model, for it has led you to be surprised by the true state of affairs. Since the beginning, not one unusual thing has ever happened."

But how is it not surprising?

"What are you suggesting, that some sort of shadowy figure stood behind the scenes and directed evolution?"

Hell no. But -

"Because if you were suggesting that, I would have to ask how that shadowy figure originally decided that love was a desirable outcome of evolution. I would have to ask where that figure got preferences that included things like love, friendship, loyalty, fairness, honor, romance, and so on. On evolutionary psychology, we can see how that specific outcome came about - how those particular goals rather than others were generated in the first place. You can call it 'surprising' all you like. But when you really do understand evolutionary psychology, you can see how parental love and romance and honor, and even true altruism and moral arguments, bear the specific design signature of natural selection in particular adaptive contexts of the hunter-gatherer savanna. So if there was a shadowy figure, it must itself have evolved - and that obviates the whole point of postulating it."

I'm not postulating a shadowy figure! I'm just asking how human beings ended up so nice.

"Nice! Have you looked at this planet lately? We also bear all those other emotions that evolved, too - which would tell you very well that we evolved, should you begin to doubt it. Humans aren't always nice."

We're one hell of a lot nicer than the process that produced us, which lets elephants starve to death when they run out of teeth, and doesn't anesthetize a gazelle even as it lays dying and is of no further importance to evolution one way or the other. It doesn't take much to be nicer than evolution. To have the theoretical capacity to make one single gesture of mercy, to feel a single twinge of empathy, is to be nicer than evolution. How did evolution, which is itself so uncaring, create minds on that qualitatively higher moral level than itself? How did evolution, which is so ugly, end up doing anything so beautiful?

"Beautiful, you say? Bach's Little Fugue in G Minor may be beautiful, but the sound waves, as they travel through the air, are not stamped with tiny tags to specify their beauty. If you wish to find explicitly encoded a measure of the fugue's beauty, you will have to look at a human brain - nowhere else in the universe will you find it. Not upon the seas or the mountains will you find such judgments written: they are not minds, they cannot think."

Perhaps that is so, but still I ask: How did evolution end up doing anything so beautiful, as giving us the ability to admire the beauty of a flower?

"Can you not see the circularity in your question? If beauty were like some great light in the sky that shined from outside humans, then your question might make sense - though there would still be the question of how humans came to perceive that light. You evolved with a psychology unlike evolution: Evolution has nothing like the intelligence or the precision required to exactly quine its goal system. In coughing up the first true minds, evolution's simple fitness criterion shattered into a thousand values. You evolved with a psychology that attaches utility to things which evolution does not care about, like human life and happiness. And then you look back and say, 'How marvelous, that uncaring evolution produced minds that care about sentient life!' So your great marvel and wonder, that seems like far too much coincidence, is really no coincidence at all."

But then it is still amazing that this particular circular loop, happened to loop around such important things as beauty and altruism.

"I don't think you're following me here. To you, it seems natural to privilege the beauty and altruism as special, as preferred, because you value them highly; and you don't see this as a unusual fact about yourself, because many of your friends do likewise. So you expect that a ghost of perfect emptiness would also value life and happiness - and then, from this standpoint outside reality, a great coincidence would indeed have occurred."

But you can make arguments for the importance of beauty and altruism from first principles - that our aesthetic senses lead us to create new complexity, instead of repeating the same things over and over; and that altruism is important because it takes us outside ourselves, gives our life a higher meaning than sheer brute selfishness.

"Oh, and that argument is going to move even a ghost of perfect emptiness - now that you've appealed to slightly different values? Those aren't first principles, they're just different principles. Even if you've adopted a high-falutin' philosophical tone, still there are no universally compelling arguments. All you've done is pass the recursive buck."

You don't think that, somehow, we evolved to tap into something beyond -

"What good does it do to suppose something beyond? Why should we pay more attention to that beyond thing, than we pay to our existence as humans? How does it alter your personal responsibility, to say that you were only following the orders of the beyond thing? And you would still have evolved to let the beyond thing, rather than something else, direct your actions. You are only passing the recursive buck. Above all, it would be too much coincidence."

Too much coincidence?

"A flower is beautiful, you say. Do you think there is no story behind that beauty, or that science does not know the story? Flower pollen is transmitted by bees, so by sexual selection, flowers evolved to attract bees - by imitating certain mating signs of bees, as it happened; the flowers' patterns would look more intricate, if you could see in the ultraviolet. Now healthy flowers are a sign of fertile land, likely to bear fruits and other treasures, and probably prey animals as well; so is it any wonder that humans evolved to be attracted to flowers? But for there to be some great light written upon the very stars - those huge unsentient balls of burning hydrogen - which also said that flowers were beautiful, now that would be far too much coincidence."

So you explain away the beauty of a flower?

"No, I explain it. Of course there's a story behind the beauty of flowers and the fact that we find them beautiful. Behind ordered events, one finds ordered stories; and what has no story is the product of random noise, which is hardly any better. If you cannot take joy in things that have stories behind them, your life will be empty indeed. I don't think I take any less joy in a flower than you do; more so, perhaps, because I take joy in its story as well."

Perhaps as you say, there is no surprise from a causal viewpoint - no disruption of the physical order of the universe. But it still seems to me that, in this creation of humans by evolution, something happened that is precious and marvelous and wonderful. If we cannot call it a physical miracle, then call it a moral miracle.

"Because it's only a miracle from the perspective of the morality that was produced, thus explaining away all of the apparent coincidence from a merely causal and physical perspective?"

Well... I suppose you could interpret the term that way, yes. I just meant something that was immensely surprising and wonderful on a moral level, even if it is not surprising on a physical level.

"I think that's what I said."

But it still seems to me that you, from your own view, drain something of that wonder away.

"Then you have problems taking joy in the merely real. Love has to begin somehow, it has to enter the universe somewhere. It is like asking how life itself begins - and though you were born of your father and mother, and they arose from their living parents in turn, if you go far and far and far away back, you will finally come to a replicator that arose by pure accident - the border between life and unlife. So too with love.

"A complex pattern must be explained by a cause which is not already that complex pattern. Not just the event must be explained, but the very shape and form. For love to first enter Time, it must come of something that is not love; if this were not possible, then love could not be.

"Even as life itself required that first replicator to come about by accident, parentless but still caused: far, far back in the causal chain that led to you: 3.85 billion years ago, in some little tidal pool.

"Perhaps your children's children will ask how it is that they are capable of love.

"And their parents will say: Because we, who also love, created you to love.

"And your children's children will ask: But how is it that you love?

"And their parents will reply: Because our own parents, who also loved, created us to love in turn.

"Then your children's children will ask: But where did it all begin? Where does the recursion end?

"And their parents will say: Once upon a time, long ago and far away, ever so long ago, there were intelligent beings who were not themselves intelligently designed. Once upon a time, there were lovers created by something that did not love.

"Once upon a time, when all of civilization was a single galaxy and a single star: and a single planet, a place called Earth.

"Long ago, and far away, ever so long ago."

Sequence: Metaethics

Whither Moral Progress?

Could Anything Be Right?

Referenced by: Could Anything Be Right? • Does Your Morality Care What You Think? • Changing Your Metaethics • Setting Up Metaethics • The Meaning of Right • Humans in Funny Suits • Inseparably Right; or, Joy in the Merely Good • The Bedrock of Morality: Arbitrary? • No License To Be Human • Mirrors and Paintings • Ethical Inhibitions • Sympathetic Minds • Value is Fragile • An Especially Elegant Evpsych Experiment

Original with comments: The Gift We Give To Tomorrow

Could Anything Be Right?

Eliezer Yudkowsky, 18 July 2008 07:19AM

Followup to: Where Recursive Justification Hits Bottom, Rebelling Within Nature

Years ago, Eliezer_1999 was convinced that he knew nothing about morality.

For all he knew, morality could require the extermination of the human species; and if so he saw no virtue in taking a stand against morality, because he thought that, by definition, if he postulated that moral fact, that meant human extinction was what "should" be done.

I thought I could figure out what was right, perhaps, given enough reasoning time and enough facts, but that I currently had no information about it. I could not trust evolution which had built me. What foundation did that leave on which to stand?

Well, indeed Eliezer_1999 was massively mistaken about the nature of morality, so far as his explicitly represented philosophy went.

But as Davidson once observed, if you believe that "beavers" live in deserts, are pure white in color, and weigh 300 pounds when adult, then you do not have any beliefs about beavers, true or false. You must get at least some of your beliefs right, before the remaining ones can be wrong about anything.

My belief that I had no information about morality was not internally consistent.

Saying that I knew nothing felt virtuous, for I had once been taught that it was virtuous to confess my ignorance. "The only thing I know is that I know nothing," and all that. But in this case I would have been better off considering the admittedly exaggerated saying, "The greatest fool is the one who is not aware they are wise." (This is nowhere near the greatest kind of foolishness, but it is a kind of foolishness.)

Was it wrong to kill people? Well, I thought so, but I wasn't sure; maybe it was right to kill people, though that seemed less likely.

What kind of procedure would answer whether it was right to kill people? I didn't know that either, but I thought that if you built a generic superintelligence (what I would later label a "ghost of perfect emptiness") then it could, you know, reason about what was likely to be right and wrong; and since it was superintelligent, it was bound to come up with the right answer.

The problem that I somehow managed not to think too hard about, was where the superintelligence would get the procedure that discovered the procedure that discovered the procedure that discovered morality - if I couldn't write it into the start state that wrote the successor AI that wrote the successor AI.

As Marcello Herreshoff later put it, "We never bother running a computer program unless we don't know the output and we know an important fact about the output." If I knew nothing about morality, and did not even claim to know the nature of morality, then how could I construct any computer program whatsoever - even a "superintelligent" one or a "self-improving" one - and claim that it would output something called "morality"?

There are no-free-lunch theorems in computer science - in a maxentropy universe, no plan is better on average than any other. If you have no knowledge at all about "morality", there's also no computational procedure that will seem more likely than others to compute "morality", and no meta-procedure that's more likely than others to produce a procedure that computes "morality".

I thought that surely even a ghost of perfect emptiness, finding that it knew nothing of morality, would see a moral imperative to think about morality.

But the difficulty lies in the word think. Thinking is not an activity that a ghost of perfect emptiness is automatically able to carry out. Thinking requires running some specific computation that is the thought. For a reflective AI to decide to think, requires that it know some computation which it believes is more likely to tell it what it wants to know, than consulting an Ouija board; the AI must also have a notion of how to interpret the output.

If one knows nothing about morality, what does the word "should" mean, at all? If you don't know whether death is right or wrong - and don't know how you can discover whether death is right or wrong - and don't know whether any given procedure might output the procedure for saying whether death is right or wrong - then what do these words, "right" and "wrong", even mean?

If the words "right" and "wrong" have nothing baked into them - no starting point - if everything about morality is up for grabs, not just the content but the structure and the starting point and the determination procedure - then what is their meaning? What distinguishes, "I don't know what is right" from "I don't know what is wakalixes"?

A scientist may say that everything is up for grabs in science, since any theory may be disproven; but then they have some idea of what would count as evidence that could disprove the theory. Could there be something that would change what a scientist regarded as evidence?

Well, yes, in fact; a scientist who read some Karl Popper and thought they knew what "evidence" meant, could be presented with the coherence and uniqueness proofs underlying Bayesian probability, and that might change their definition of evidence. They might not have had any explicit notion, in advance, that such a proof could exist. But they would have had an implicit notion. It would have been baked into their brains, if not explicitly represented therein, that such-and-such an argument would in fact persuade them that Bayesian probability gave a better definition of "evidence" than the one they had been using.

In the same way, you could say, "I don't know what morality is, but I'll know it when I see it," and make sense.

But then you are not rebelling completely against your own evolved nature. You are supposing that whatever has been baked into you to recognize "morality", is, if not absolutely trustworthy, then at least your initial condition with which you start debating. Can you trust your moral intuitions to give you any information about morality at all, when they are the product of mere evolution?

But if you discard every procedure that evolution gave you and all its products, then you discard your whole brain. You discard everything that could potentially recognize morality when it sees it. You discard everything that could potentially respond to moral arguments by updating your morality. You even unwind past the unwinder: you discard the intuitions underlying your conclusion that you can't trust evolution to be moral. It is your existing moral intuitions that tell you that evolution doesn't seem like a very good source of morality. What, then, will the words "right" and "should" and "better" even mean?

Humans do not perfectly recognize truth when they see it, and hunter-gatherers do not have an explicit concept of the Bayesian criterion of evidence. But all our science and all our probability theory was built on top of a chain of appeals to our instinctive notion of "truth". Had this core been flawed, there would have been nothing we could do in principle to arrive at the present notion of science; the notion of science would have just sounded completely unappealing and pointless.

One of the arguments that might have shaken my teenage self out of his mistake, if I could have gone back in time to argue with him, was the question:

Could there be some morality, some given rightness or wrongness, that human beings do not perceive, do not want to perceive, will not see any appealing moral argument for adopting, nor any moral argument for adopting a procedure that adopts it, etcetera? Could there be a morality, and ourselves utterly outside its frame of reference? But then what makes this thing morality - rather than a stone tablet somewhere with the words 'Thou shalt murder' written on them, with absolutely no justification offered?

So all this suggests that you should be willing to accept that you might know a little about morality. Nothing unquestionable, perhaps, but an initial state with which to start questioning yourself. Baked into your brain but not explicitly known to you, perhaps; but still, that which your brain would recognize as right is what you are talking about. You will accept at least enough of the way you respond to moral arguments as a starting point, to identify "morality" as something to think about.

But that's a rather large step.

It implies accepting your own mind as identifying a moral frame of reference, rather than all morality being a great light shining from beyond (that in principle you might not be able to perceive at all). It implies accepting that even if there were a light and your brain decided to recognize it as "morality", it would still be your own brain that recognized it, and you would not have evaded causal responsibility - or evaded moral responsibility either, on my view.

It implies dropping the notion that a ghost of perfect emptiness will necessarily agree with you, because the ghost might occupy a different moral frame of reference, respond to different arguments, be asking a different question when it computes what-to-do-next.

And if you're willing to bake at least a few things into the very meaning of this topic of "morality", this quality of rightness that you are talking about when you talk about "rightness" - if you're willing to accept even that morality is what you argue about when you argue about "morality" - then why not accept other intuitions, other pieces of yourself, into the starting point as well?

Why not accept that, ceteris paribus, joy is preferable to sorrow?

You might later find some ground within yourself or built upon yourself with which to criticize this - but why not accept it for now? Not just as a personal preference, mind you; but as something baked into the question you ask when you ask "What is truly right"?

But then you might find that you know rather a lot about morality! Nothing certain - nothing unquestionable - nothing unarguable - but still, quite a bit of information. Are you willing to relinquish your Socratean ignorance?

I don't argue by definitions, of course. But if you claim to know nothing at all about morality, then you will have problems with the meaning of your words, not just their plausibility.

Sequence: Metaethics

The Gift We Give To Tomorrow

Existential Angst Factory

Referenced by: Does Your Morality Care What You Think? • Setting Up Metaethics • The Meaning of Right • The Bedrock of Morality: Arbitrary? • Lawful Creativity • Prolegomena to a Theory of Fun • Emotional Involvement

Original with comments: Could Anything Be Right?

Existential Angst Factory

Eliezer Yudkowsky, 19 July 2008 06:55AM

Followup to: The Moral Void

A widespread excuse for avoiding rationality is the widespread belief that it is "rational" to believe life is meaningless, and thus suffer existential angst. This is one of the secondary reasons why it is worth discussing the nature of morality. But it's also worth attacking existential angst directly.

I suspect that most existential angst is not really existential. I think that most of what is labeled "existential angst" comes from trying to solve the wrong problem.

Let's say you're trapped in an unsatisfying relationship, so you're unhappy. You consider going on a skiing trip, or you actually go on a skiing trip, and you're still unhappy. You eat some chocolate, but you're still unhappy. You do some volunteer work at a charity (or better yet, work the same hours professionally and donate the money, thus applying the Law of Comparative Advantage) and you're still unhappy because you're in an unsatisfying relationship.

So you say something like: "Skiing is meaningless, chocolate is meaningless, charity is meaningless, life is doomed to be an endless stream of woe." And you blame this on the universe being a mere dance of atoms, empty of meaning. Not necessarily because of some kind of subconsciously deliberate Freudian substitution to avoid acknowledging your real problem, but because you've stopped hoping that your real problem is solvable. And so, as a sheer unexplained background fact, you observe that you're always unhappy.

Maybe you're poor, and so always unhappy. Nothing you do solves your poverty, so it starts to seem like a universal background fact, along with your unhappiness. So when you observe that you're always unhappy, you blame this on the universe being a mere dance of atoms. Not as some kind of Freudian substitution, but because it has ceased to occur to you that there does exist some possible state of affairs in which life is not painful.

What about rich heiresses with everything in the world available to buy, who still feel unhappy? Perhaps they can't get themselves into satisfying romantic relationships. One way or another, they don't know how to use their money to create happiness - they lack the expertise in hedonic psychology and/or self-awareness and/or simple competence.

So they're constantly unhappy - and they blame it on existential angst, because they've already solved the only problem they know how to solve. They already have enough money and they've already bought all the toys. Clearly, if there's still a problem, it's because life is meaningless.

If someone who weighs 560 pounds suffers from "existential angst", allegedly because the universe is a mere dance of particles, then stomach reduction surgery might drastically change their views of the metaphysics of morality.

I'm not a fan of Timothy Ferris, but The Four-Hour Workweek does make an interesting fun-theoretic observation:

Let's assume we have 10 goals and we achieve them - what is the desired outcome that makes all the effort worthwhile? The most common response is what I also would have suggested five years ago: happiness. I no longer believe this is a good answer. Happiness can be bought with a bottle of wine and has become ambiguous through overuse. There is a more precise alternative that reflects what I believe the actual objective is.

Bear with me. What is the opposite of happiness? Sadness? No. Just as love and hate are two sides of the same coin, so are happiness and sadness. Crying out of happiness is a perfect illustration of this. The opposite of love is indifference, and the opposite of happiness is - here's the clincher - boredom.

Excitement is the more practical synonym for happiness, and it is precisely what you should strive to chase. It is the cure-all. When people suggest you follow your "passion" or your "bliss," I propose that they are, in fact, referring to the same singular concept: excitement.

This brings us full circle. The question you should be asking isn't "What do I want?" or "What are my goals?" but "What would excite me?"

Remember - boredom is the enemy, not some abstract "failure."

Living like a millionaire requires doing interesting things and not just owning enviable things.

I don't endorse all of the above, of course. But note the SolvingTheWrongProblem anti-pattern Ferris describes. It was on reading the above that I first generalized ExistentialAngstFactory.

Now, if someone is in a unproblematic, loving relationship; and they have enough money; and no major health problems; and they're signed up for cryonics so death is not approaching inexorably; and they're doing exciting work that they enjoy; and they believe they're having a positive effect on the world...

...and they're still unhappy because it seems to them that the universe is a mere dance of atoms empty of meaning, then we may have a legitimate problem here. One that, perhaps, can only be resolved by a very long discussion of the nature of morality and how it fits into a reductionist universe.

But, mostly, I suspect that when people complain about the empty meaningless void, it is because they have at least one problem that they aren't thinking about solving - perhaps because they never identified it. Being able to identify your own problems is a feat of rationality that schools don't explicitly train you to perform. And they haven't even been told that an un-focused-on problem might be the source of their "existential angst" - they've just been told to blame it on existential angst.

That's the other reason it might be helpful to understand the nature of morality - even if it just adds up to moral normality - because it tells you that if you're constantly unhappy, it's not because the universe is empty of meaning.

Or maybe believing the universe is a "mere dance of particles" is one more factor contributing to human unhappiness; in which case, again, people can benefit from eliminating that factor.

If it seems to you like nothing you do makes you happy, and you can't even imagine what would make you happy, it's not because the universe is made of particle fields. It's because you're still solving the wrong problem. Keep searching, until you find the visualizable state of affairs in which the existential angst seems like it should go away - that might (or might not) tell you the real problem; but at least, don't blame it on reductionism.

Added: Several commenters pointed out that random acts of brain chemistry may also be responsible for depression, even if your life is otherwise fine. As far as I know, this is true. But, once again, it won't help to mistake that random act of brain chemistry as being about existential issues; that might prevent you from trying neuropharmaceutical interventions.

Sequence: Metaethics

Could Anything Be Right?

Can Counterfactuals Be True?

Referenced by: Changing Your Metaethics • The Meaning of Right • High Challenge • Dunbar's Function • Emotional Involvement • Justified Expectation of Pleasant Surprises • 31 Laws of Fun • Incremental Progress and the Valley

Original with comments: Existential Angst Factory

Touching the Old

Eliezer Yudkowsky, 20 July 2008 09:19AM

I'm in Oxford right now, for the Global Catastrophic Risks conference.

There's a psychological impact in walking down a street where where any given building might be older than your whole country.

Toby Ord and Anders Sandberg pointed out to me an old church tower in Oxford, that is a thousand years old.

At the risk conference I heard a talk from someone talking about what the universe will look like in 10^100 years (barring intelligent modification thereof, which he didn't consider).

The psychological impact of seeing that old church tower was greater. I'm not defending this reaction, only admitting it.

I haven't traveled as much as I would travel if I were free to follow my whims; I've never seen the Pyramids. I don't think I've ever touched anything that has endured in the world for longer than that church tower.

A thousand years... I've lived less than half of 70, and sometimes it seems like a long time to me. What would it be like, to be as old as that tower? To have lasted through that much of the world, that much history and that much change?

Transhumanism does scare me. I shouldn't wonder if it scares me more than it scares arch-luddites like Leon Kass. Kass doesn't take it seriously; he doesn't expect to live that long.

Yet I know - and I doubt the thought ever occurred to Kass - that even if something scares you, you can still have the courage to confront it. Even time. Even life.

But sometimes it's such a strange thought that our world really is that old.

The inverse failure of the logical fallacy of generalization from fictional evidence, is failure to generalize from things that actually happened. We see movies, and in the ancestral environment, what you saw with your own eyes was real; we have to avoid treating them as available examples.

Conversely, history books seem like writing on paper - but those are things that really happened, even if we hear about them selectively. What happened there was as real to the people who lived it, as your own life, and equally evidence.

Sometimes it's such a strange thought that the people in the history books really lived and experienced and died - that there's so much more depth to history than anything I've seen with my own eyes; so much more life than anything I've lived.

Original with comments: Touching the Old

Should We Ban Physics?

Eliezer Yudkowsky, 21 July 2008 08:12AM

Nobel laureate Marie Curie died of aplastic anemia, the victim of radiation from the many fascinating glowing substances she had learned to isolate.

How could she have known? And the answer, as far as I can tell, is that she couldn't. The only way she could have avoided death was by being too scared of anything new to go near it. Would banning physics experiments have saved Curie from herself?

But far more cancer patients than just one person have been saved by radiation therapy. And the real cost of banning physics is not just losing that one experiment - it's losing physics. No more Industrial Revolution.

Some of us fall, and the human species carries on, and advances; our modern world is built on the backs, and sometimes the bodies, of people who took risks. My father is fond of saying that if the automobile were invented nowadays, the saddle industry would arrange to have it outlawed.

But what if the laws of physics had been different from what they are? What if Curie, by isolating and purifying the glowy stuff, had caused something akin to a fission chain reaction gone critical... which, the laws of physics being different, had ignited the atmosphere or produced a strangelet?

At the recent Global Catastrophic Risks conference, someone proposed a policy prescription which, I argued, amounted to a ban on all physics experiments involving the production of novel physical situations - as opposed to measuring existing phenomena. You can weigh a rock, but you can't purify radium, and you can't even expose the rock to X-rays unless you can show that exactly similar X-rays hit rocks all the time. So the Large Hadron Collider, which produces collisions as energetic as cosmic rays, but not exactly the same as cosmic rays, would be off the menu.

After all, whenever you do something new, even if you calculate that everything is safe, there is surely some probability of being mistaken in the calculation - right?

So the one who proposed the policy, disagreed that their policy cashed out to a blanket ban on physics experiments. And discussion is in progress, so I won't talk further about their policy argument.

But if you consider the policy of "Ban Physics", and leave aside the total political infeasibility, I think the strongest way to frame the issue - from the pro-ban viewpoint - would be as follows:

Suppose that Tegmark's Level IV Multiverse is real - that all possible mathematical objects, including all possible physical universes with all possible laws of physics, exist. (Perhaps anthropically weighted by their simplicity.)

Somewhere in Tegmark's Level IV Multiverse, then, there have undoubtedly been cases where intelligence arises somewhere in a universe with physics unlike this one - i.e., instead of a planet, life arises on a gigantic triangular plate hanging suspended in the void - and that intelligence accidentally destroys its world, perhaps its universe, in the course of a physics experiment.

Maybe they experiment with alchemy, bring together some combination of substances that were never brought together before, and catalyze a change in their atmosphere. Or maybe they manage to break their triangular plate, whose pieces fall and break other triangular plates.

So, across the whole of the Tegmark Level IV multiverse - containing all possible physical universes with all laws of physics, weighted by the laws' simplicity:

What fraction of sentient species that try to follow the policy "Ban all physics experiments involving situations with a remote possibility of being novel, until you can augment your own intelligence enough to do error-free cognition";

And what fraction of sentient species that go ahead and do physics experiments;

Survive in the long term, on average?

In the case of the human species, trying to ban chemistry would hardly have been effective - but supposing that a species actually could make a collective decision like that, it's at least not clear-cut which fraction would be larger across the whole multiverse. (We, in our universe, have already learned that you can't easily destroy the world with alchemy.)

Or an even tougher question: On average, across the multiverse, do you think you would advise an intelligent species to stop performing novel physics experiments during the interval after it figures out how to build transistors and before it builds AI?

Referenced by: When (Not) To Use Probabilities • Horrible LHC Inconsistency

Original with comments: Should We Ban Physics?

Fake Norms, or "Truth" vs. Truth

Eliezer Yudkowsky, 22 July 2008 10:23AM

Followup to: Applause Lights

When you say the word "truth", people know that "truth" is a good thing, and that they're supposed to applaud. So it might seem like there is a social norm in favor of "truth". But when it comes to some particular truth, like whether God exists, or how likely their startup is to thrive, people will say: "I just want to believe" or "you've got to be optimistic to succeed".

So Robin and I were talking about this, and Robin asked me how it is that people prevent themselves from noticing the conflict.

I replied that I don't think active prevention is required. First, as I quoted Michael Vassar:

"It seems to me that much of the frustration in my life prior to a few years ago has been due to thinking that all other human minds necessarily and consistently implement modus ponens."

But more importantly, I don't think there does exist any social norm in favor of truth. There's a social norm in favor of "truth". There's a difference.

How would a norm in favor of truth actually be expressed, or acquired?

If you were told many stories, as a kid, about specific people who accepted specific hard truths - like a story of a scientist accepting that their theory was wrong, say - then your brain would generalize over its experiences, and compress them, and form a concept of that-which-is-the-norm: the wordless act of accepting reality.

If you heard someone say "I don't care about the evidence, I just want to believe in God", and you saw everyone else in the room gasp and regard them in frozen shock, then your brain would generalize a social norm against self-deception. (E.g., the sort of thing that would happen if a scientist said "I don't care about the evidence, I just want to believe in my-favorite-theory" in front of their fellow scientists.)

If, on the other hand, you see lots of people saying "Isn't the truth wonderful?" or "I am in favor of truth", then you learn that when someone says "truth", you are supposed to applaud.

Now there are certain particular cases where someone will be castigated if they admit they refuse to see the truth: for example, "I've seen the evidence on global warming but I don't want to believe it." You couldn't get away with that in modern society. But this indignation doesn't have to derive from violating a norm in favor of truth - it can derive from the widely held norm, "'global warming' is bad".

But (said Robin) we see a lot of trees and hear the word "tree", and somehow we learn that the word refers to the thing - why don't people learn something similar about "truth", which is supposed to be good?

I suggested in reply that the brain is capable of distinguishing different uses of the same syllables - a child is quite capable of learning that a right turn and the right answer are not the same kind of "right". You won't necessarily assume that the right answer is always the one printed on the right side of the page. Maybe the word "truth" is overloaded in the same way.

Or maybe it's not exactly the same, but analogous: the social norms of which words we are meant to praise, and which deeds, are stored as separately as left hands and leftovers.

There's a social norm in favor of "diversity", but not diversity. There's a social norm in favor of "free speech", but not pornography. There's a social norm in favor of "democracy", but it doesn't spontaneously occur to most people to suggest voting on their arguments. There's a social norm in favor of "love", but not for letting some damn idiot marry your daughter even if the two of them are stupid and besotted.

There's a social norm in favor of "honesty". And there are in fact social norms for honesty about e.g. who cut down the cherry tree. But not a social norm favoring saying what you think about someone else's appearance.

I'm not suggesting that you ignore all the words that people praise. Sometimes the things people praise with their lips, really are the things that matter, and our deeds are what fail to live up. Neither am I suggesting that you should ignore what people really do, because sometimes that also embodies wisdom. I would just say to be aware of any differences, and judge deliberately, and choose knowingly.

Sounds good, doesn't it? Everyone knows that being "aware" and "choosing knowingly" must surely be good things. But is it a real norm or a fake norm? Can you think of any stories you were told that illustrate the point? (Not a rhetorical question, but a question one should learn to ask.)

It's often not hard to find a norm in favor of "rationality" - but norms favoring rationality are rarer.

Referenced by: Honesty: Beyond Internal Truth

Original with comments: Fake Norms, or "Truth" vs. Truth

When (Not) To Use Probabilities

Eliezer Yudkowsky, 23 July 2008 10:58AM

Followup to: Should We Ban Physics?

It may come as a surprise to some readers of this blog, that I do not always advocate using probabilities.

Or rather, I don't always advocate that human beings, trying to solve their problems, should try to make up verbal probabilities, and then apply the laws of probability theory or decision theory to whatever number they just made up, and then use the result as their final belief or decision.

The laws of probability are laws, not suggestions, but often the true Law is too difficult for us humans to compute. If P != NP and the universe has no source of exponential computing power, then there are evidential updates too difficult for even a superintelligence to compute - even though the probabilities would be quite well-defined, if we could afford to calculate them.

So sometimes you don't apply probability theory. Especially if you're human, and your brain has evolved with all sorts of useful algorithms for uncertain reasoning, that don't involve verbal probability assignments.

Not sure where a flying ball will land? I don't advise trying to formulate a probability distribution over its landing spots, performing deliberate Bayesian updates on your glances at the ball, and calculating the expected utility of all possible strings of motor instructions to your muscles.

Trying to catch a flying ball, you're probably better off with your brain's built-in mechanisms, then using deliberative verbal reasoning to invent or manipulate probabilities.

But this doesn't mean you're going beyond probability theory or above probability theory.

The Dutch Book arguments still apply. If I offer you a choice of gambles ($10,000 if the ball lands in this square, versus $10,000 if I roll a die and it comes up 6), and you answer in a way that does not allow consistent probabilities to be assigned, then you will accept combinations of gambles that are certain losses, or reject gambles that are certain gains...

Which still doesn't mean that you should try to use deliberative verbal reasoning. I would expect that for professional baseball players, at least, it's more important to catch the ball than to assign consistent probabilities. Indeed, if you tried to make up probabilities, the verbal probabilities might not even be very good ones, compared to some gut-level feeling - some wordless representation of uncertainty in the back of your mind.

There is nothing privileged about uncertainty that is expressed in words, unless the verbal parts of your brain do, in fact, happen to work better on the problem.

And while accurate maps of the same territory will necessarily be consistent among themselves, not all consistent maps are accurate. It is more important to be accurate than to be consistent, and more important to catch the ball than to be consistent.

In fact, I generally advise against making up probabilities, unless it seems like you have some decent basis for them. This only fools you into believing that you are more Bayesian than you actually are.

To be specific, I would advise, in most cases, against using non-numerical procedures to create what appear to be numerical probabilities. Numbers should come from numbers.

Now there are benefits from trying to translate your gut feelings of uncertainty into verbal probabilities. It may help you spot problems like the conjunction fallacy. It may help you spot internal inconsistencies - though it may not show you any way to remedy them.

But you shouldn't go around thinking that, if you translate your gut feeling into "one in a thousand", then, on occasions when you emit these verbal words, the corresponding event will happen around one in a thousand times. Your brain is not so well-calibrated. If instead you do something nonverbal with your gut feeling of uncertainty, you may be better off, because at least you'll be using the gut feeling the way it was meant to be used.

This specific topic came up recently in the context of the Large Hadron Collider, and an argument given at the Global Catastrophic Risks conference:

That we couldn't be sure that there was no error in the papers which showed from multiple angles that the LHC couldn't possibly destroy the world. And moreover, the theory used in the papers might be wrong. And in either case, there was still a chance the LHC could destroy the world. And therefore, it ought not to be turned on.

Now if the argument had been given in just this way, I would not have objected to its epistemology.

But the speaker actually purported to assign a probability of at least 1 in 1000 that the theory, model, or calculations in the LHC paper were wrong; and a probability of at least 1 in 1000 that, if the theory or model or calculations were wrong, the LHC would destroy the world.

After all, it's surely not so improbable that future generations will reject the theory used in the LHC paper, or reject the model, or maybe just find an error. And if the LHC paper is wrong, then who knows what might happen as a result?

So that is an argument - but to assign numbers to it?

I object to the air of authority given these numbers pulled out of thin air. I generally feel that if you can't use probabilistic tools to shape your feelings of uncertainty, you ought not to dignify them by calling them probabilities.

The alternative I would propose, in this particular case, is to debate the general rule of banning physics experiments because you cannot be absolutely certain of the arguments that say they are safe.

I hold that if you phrase it this way, then your mind, by considering frequencies of events, is likely to bring in more consequences of the decision, and remember more relevant historical cases.

If you debate just the one case of the LHC, and assign specific probabilities, it (1) gives very shaky reasoning an undue air of authority, (2) obscures the general consequences of applying similar rules, and even (3) creates the illusion that we might come to a different decision if someone else published a new physics paper that decreased the probabilities.

The authors at the Global Catastrophic Risk conference seemed to be suggesting that we could just do a bit more analysis of the LHC and then switch it on. This struck me as the most disingenuous part of the argument. Once you admit the argument "Maybe the analysis could be wrong, and who knows what happens then," there is no possible physics paper that can ever get rid of it.

No matter what other physics papers had been published previously, the authors would have used the same argument and made up the same numerical probabilities at the Global Catastrophic Risk conference. I cannot be sure of this statement, of course, but it has a probability of 75%.

In general a rationalist tries to make their minds function at the best achievable power output; sometimes this involves talking about verbal probabilities, and sometimes it does not, but always the laws of probability theory govern.

If all you have is a gut feeling of uncertainty, then you should probably stick with those algorithms that make use of gut feelings of uncertainty, because your built-in algorithms may do better than your clumsy attempts to put things into words.

Now it may be that by reasoning thusly, I may find myself inconsistent. For example, I would be substantially more alarmed about a lottery device with a well-defined chance of 1 in 1,000,000 of destroying the world, than I am about the Large Hadron Collider being switched on.

On the other hand, if you asked me whether I could make one million statements of authority equal to "The Large Hadron Collider will not destroy the world", and be wrong, on average, around once, then I would have to say no.

What should I do about this inconsistency? I'm not sure, but I'm certainly not going to wave a magic wand to make it go away. That's like finding an inconsistency in a pair of maps you own, and quickly scribbling some alterations to make sure they're consistent.

I would also, by the way, be substantially more worried about a lottery device with a 1 in 1,000,000,000 chance of destroying the world, than a device which destroyed the world if the Judeo-Christian God existed. But I would not suppose that I could make one billion statements, one after the other, fully independent and equally fraught as "There is no God", and be wrong on average around once.

I can't say I'm happy with this state of epistemic affairs, but I'm not going to modify it until I can see myself moving in the direction of greater accuracy and real-world effectiveness, not just moving in the direction of greater self-consistency. The goal is to win, after all. If I make up a probability that is not shaped by probabilistic tools, if I make up a number that is not created by numerical methods, then maybe I am just defeating my built-in algorithms that would do better by reasoning in their native modes of uncertainty.

Of course this is not a license to ignore probabilities that are well-founded. Any numerical founding at all is likely to be better than a vague feeling of uncertainty; humans are terrible statisticians. But pulling a number entirely out of your butt, that is, using a non-numerical procedure to produce a number, is nearly no foundation at all; and in that case you probably are better off sticking with the vague feelings of uncertainty.

Which is why my Overcoming Bias posts generally use words like "maybe" and "probably" and "surely" instead of assigning made-up numerical probabilities like "40%" and "70%" and "95%". Think of how silly that would look. I think it actually would be silly; I think I would do worse thereby.

I am not the kind of straw Bayesian who says that you should make up probabilities to avoid being subject to Dutch Books. I am the sort of Bayesian who says that in practice, humans end up subject to Dutch Books because they aren't powerful enough to avoid them; and moreover it's more important to catch the ball than to avoid Dutch Books. The math is like underlying physics, inescapably governing, but too expensive to calculate. Nor is there any point in a ritual of cognition which mimics the surface forms of the math, but fails to produce systematically better decision-making. That would be a lost purpose; this is not the true art of living under the law.

Referenced by: Horrible LHC Inconsistency • Shut up and do the impossible!

Original with comments: When (Not) To Use Probabilities

Can Counterfactuals Be True?

Eliezer Yudkowsky, 24 July 2008 04:40AM

Followup to: Probability is Subjectively Objective

The classic explanation of counterfactuals begins with this distinction:

1. If Lee Harvey Oswald didn't shoot John F. Kennedy, then someone else did.

2. If Lee Harvey Oswald hadn't shot John F. Kennedy, someone else would have.

In ordinary usage we would agree with the first statement, but not the second (I hope).

If, somehow, we learn the definite fact that Oswald did not shoot Kennedy, then someone else must have done so, since Kennedy was in fact shot.

But if we went back in time and removed Oswald, while leaving everything else the same, then - unless you believe there was a conspiracy - there's no particular reason to believe Kennedy would be shot:

We start by imagining the same historical situation that existed in 1963 - by a further act of imagination, we remove Oswald from our vision - we run forward the laws that we think govern the world - visualize Kennedy parading through in his limousine - and find that, in our imagination, no one shoots Kennedy.

It's an interesting question whether counterfactuals can be true or false. We never get to experience them directly.

If we disagree on what would have happened if Oswald hadn't been there, what experiment could we perform to find out which of us is right?

And if the counterfactual is something unphysical - like, "If gravity had stopped working three days ago, the Sun would have exploded" - then there aren't even any alternate histories out there to provide a truth-value.

It's not as simple as saying that if the bucket contains three pebbles, and the pasture contains three sheep, the bucket is true.

Since the counterfactual event only exists in your imagination, how can it be true or false?

So... is it just as fair to say that "If Oswald hadn't shot Kennedy, the Sun would have exploded"?

After all, the event only exists in our imaginations - surely that means it's subjective, so we can say anything we like?

But so long as we have a lawful specification of how counterfactuals are constructed - a lawful computational procedure - then the counterfactual result of removing Oswald, depends entirely on the empirical state of the world.

If there was no conspiracy, then any reasonable computational procedure that simulates removing Oswald's bullet from the course of history, ought to return an answer of Kennedy not getting shot.

"Reasonable!" you say. "Ought!" you say.

But that's not the point; the point is that if you do pick some fixed computational procedure, whether it is reasonable or not, then either it will say that Kennedy gets shot, or not, and what it says will depend on the empirical state of the world. So that, if you tell me, "I believe that this-and-such counterfactual construal, run over Oswald's removal, preserves Kennedy's life", then I can deduce that you don't believe in the conspiracy.

Indeed, so long as we take this computational procedure as fixed, then the actual state of the world (which either does include a conspiracy, or does not) presents a ready truth-value for the output of the counterfactual.

In general, if you give me a fixed computational procedure, like "multiply by 7 and add 5", and then you point to a 6-sided die underneath a cup, and say, "The result-of-procedure is 26!" then it's not hard at all to assign a truth value to this statement. Even if the actual die under the cup only ever takes on the values between 1 and 6, so that "26" is not found anywhere under the cup. The statement is still true if and only if the die is showing 3; that is its empirical truth-condition.

And what about the statement ((3 * 7) + 5) = 26? Where is the truth-condition for that statement located? This I don't know; but I am nonetheless quite confident that it is true. Even though I am not confident that this 'true' means exactly the same thing as the 'true' in "the bucket is 'true' when it contains the same number of pebbles as sheep in the pasture".

So if someone I trust - presumably someone I really trust - tells me, "If Oswald hadn't shot Kennedy, someone else would have", and I believe this statement, then I believe the empirical reality is such as to make the counterfactual computation come out this way. Which would seem to imply the conspiracy. And I will anticipate accordingly.

Or if I find out that there was a conspiracy, then this will confirm the truth-condition of the counterfactual - which might make a bit more sense than saying, "Confirm that the counterfactual is true."

But how do you actually compute a counterfactual? For this you must consult Judea Pearl. Roughly speaking, you perform surgery on graphical models of causal processes; you sever some variables from their ordinary parents and surgically set them to new values, and then recalculate the probability distribution.

There are other ways of defining counterfactuals, but I confess they all strike me as entirely odd. Even worse, you have philosophers arguing over what the value of a counterfactual really is or really means, as if there were some counterfactual world actually floating out there in the philosophical void. If you think I'm attacking a strawperson here, I invite you to consult the philosophical literature on Newcomb's Problem.

A lot of philosophy seems to me to suffer from "naive philosophical realism" - the belief that philosophical debates are about things that automatically and directly exist as propertied objects floating out there in the void.

You can talk about an ideal computation, or an ideal process, that would ideally be applied to the empirical world. You can talk about your uncertain beliefs about the output of this ideal computation, or the result of the ideal process.

So long as the computation is fixed, and so long as the computational itself is only over actually existent things. Or the results of other computations previously defined - you should not have your computation be over "nearby possible worlds" unless you can tell me how to compute those, as well.

A chief sign of naive philosophical realism is that it does not tell you how to write a computer program that computes the objects of its discussion.

I have yet to see a camera that peers into "nearby possible worlds" - so even after you've analyzed counterfactuals in terms of "nearby possible worlds", I still can't write an AI that computes counterfactuals.

But Judea Pearl tells me just how to compute a counterfactual, given only my beliefs about the actual world.

I strongly privilege the real world that actually exists, and to a slightly lesser degree, logical truths about mathematical objects (preferably finite ones). Anything else you want to talk about, I need to figure out how to describe in terms of the first two - for example, as the output of an ideal computation run over the empirical state of the real universe.

The absence of this requirement as a condition, or at least a goal, of modern philosophy, is one of the primary reasons why modern philosophy is often surprisingly useless in my AI work. I've read whole books about decision theory that take counterfactual distributions as givens, and never tell you how to compute the counterfactuals.

Oh, and to talk about "the probability that John F. Kennedy was shot, given that Lee Harvey Oswald didn't shoot him", we write:

P(Kennedy_shot|Oswald_not)

And to talk about "the probability that John F. Kennedy would have been shot, if Lee Harvey Oswald hadn't shot him", we write:

P(Oswald_not []-> Kennedy_shot)

That little symbol there is supposed to be a box with an arrow coming out of it, but I don't think Unicode has it.

Sequence: Metaethics

Existential Angst Factory

Math is Subjunctively Objective

Referenced by: Math is Subjunctively Objective • The Meaning of Right • Moral Error and Moral Disagreement

Original with comments: Can Counterfactuals Be True?

Math is Subjunctively Objective

Eliezer Yudkowsky, 25 July 2008 11:06AM

Followup to: Probability is Subjectively Objective, Can Counterfactuals Be True?

I am quite confident that the statement 2 + 3 = 5 is true; I am far less confident of what it means for a mathematical statement to be true.

In "The Simple Truth" I defined a pebble-and-bucket system for tracking sheep, and defined a condition for whether a bucket's pebble level is "true" in terms of the sheep. The bucket is the belief, the sheep are the reality. I believe 2 + 3 = 5. Not just that two sheep plus three sheep equal five sheep, but that 2 + 3 = 5. That is my belief, but where is the reality?

So now the one comes to me and says: "Yes, two sheep plus three sheep equals five sheep, and two stars plus three stars equals five stars. I won't deny that. But this notion that 2 + 3 = 5, exists only in your imagination, and is purely subjective."

So I say: Excuse me, what?

And the one says: "Well, I know what it means to observe two sheep and three sheep leave the fold, and five sheep come back. I know what it means to press '2' and '+' and '3' on a calculator, and see the screen flash '5'. I even know what it means to ask someone 'What is two plus three?' and hear them say 'Five.' But you insist that there is some fact beyond this. You insist that 2 + 3 = 5."

Well, it kinda is.

"Perhaps you just mean that when you mentally visualize adding two dots and three dots, you end up visualizing five dots. Perhaps this is the content of what you mean by saying, 2 + 3 = 5. I have no trouble with that, for brains are as real as sheep."

No, for it seems to me that 2 + 3 equaled 5 before there were any humans around to do addition. When humans showed up on the scene, they did not make 2 + 3 equal 5 by virtue of thinking it. Rather, they thought that '2 + 3 = 5' because 2 + 3 did in fact equal 5.

"Prove it."

I'd love to, but I'm busy; I've got to, um, eat a salad.

"The reason you believe that 2 + 3 = 5, is your mental visualization of two dots plus three dots yielding five dots. Does this not imply that this physical event in your physical brain is the meaning of the statement '2 + 3 = 5'?"

But I honestly don't think that is what I mean. Suppose that by an amazing cosmic coincidence, a flurry of neutrinos struck my neurons, causing me to imagine two dots colliding with three dots and visualize six dots. I would then say, '2 + 3 = 6'. But this wouldn't mean that 2 + 3 actually had become equal to 6. Now, if what I mean by '2 + 3' consists entirely of what my mere physical brain merely happens to output, then a neutrino could make 2 + 3 = 6. But you can't change arithmetic by tampering with a calculator.

"Aha! I have you now!"

Is that so?

"Yes, you've given your whole game away!"

Do tell.

"You visualize a subjunctive world, a counterfactual, where your brain is struck by neutrinos, and says, '2 + 3 = 6'. So you know that in this case, your future self will say that '2 + 3 = 6'. But then you add up dots in your own, current brain, and your current self gets five dots. So you say: 'Even if I believed "2 + 3 = 6", then 2 + 3 would still equal 5.' You say: '2 + 3 = 5 regardless of what anyone thinks of it.' So your current brain, computing the same question while it imagines being different but is not actually different, finds that the answer seems to be the same. Thus your brain creates the illusion of an additional reality that exists outside it, independent of any brain."

Now hold on! You've explained my belief that 2 + 3 = 5 regardless of what anyone thinks, but that's not the same as explaining away my belief. Since 2 + 3 = 5 does not, in fact, depend on what any human being thinks of it, therefore it is right and proper that when I imagine counterfactual worlds in which people (including myself) think '2 + 3 = 6', and I ask what 2 + 3 actually equals in this counterfactual world, it still comes out as 5.

"Don't you see, that's just like trying to visualize motion stopping everywhere in the universe, by imagining yourself as an observer outside the universe who experiences time passing while nothing moves. But really there is no time without motion."

I see the analogy, but I'm not sure it's a deep analogy. Not everything you can imagine seeing, doesn't exist. It seems to me that a brain can easily compute quantities that don't depend on the brain.

"What? Of course everything that the brain computes depends on the brain! Everything that the brain computes, is computed inside the brain!"

That's not what I mean! I just mean that the brain can perform computations that refer to quantities outside the brain. You can set up a question, like 'How many sheep are in the field?', that isn't about any particular person's brain, and whose actual answer doesn't depend on any particular person's brain. And then a brain can faithfully compute that answer.

If I count two sheep and three sheep returning from the field, and Autrey's brain gets hit by neutrinos so that Autrey thinks there are six sheep in the fold, then that's not going to cause there to be six sheep in the fold - right? The whole question here is just not about what Autrey thinks, it's about how many sheep are in the fold.

Why should I care what my subjunctive future self thinks is the sum of 2 + 3, any more than I care what Autrey thinks is the sum of 2 + 3, when it comes to asking what is really the sum of 2 + 3?

"Okay... I'll take another tack. Suppose you're a psychiatrist, right? And you're an expert witness in court cases - basically a hired gun, but you try to deceive yourself about it. Now wouldn't it be a bit suspicious, to find yourself saying: 'Well, the only reason that I in fact believe that the defendant is insane, is because I was paid to be an expert psychiatric witness for the defense. And if I had been paid to witness for the prosecution, I undoubtedly would have come to the conclusion that the defendant is sane. But my belief that the defendant is insane, is perfectly justified; it is justified by my observation that the defendant used his own blood to paint an Elder Sign on the wall of his jail cell.'"

Yes, that does sound suspicious, but I don't see the point.

"My point is that the physical cause of your belief that 2 + 3 = 5, is the physical event of your brain visualizing two dots and three dots and coming up with five dots. If your brain came up six dots, due to a neutrino storm or whatever, you'd think '2 + 3 = 6'. How can you possibly say that your belief means anything other than the number of dots your brain came up with?"

Now hold on just a second. Let's say that the psychiatrist is paid by the judge, and when he's paid by the judge, he renders an honest and neutral evaluation, and his evaluation is that the defendant is sane, just played a bit too much Mythos. So it is true to say that if the psychiatrist had been paid by the defense, then the psychiatrist would have found the defendant to be insane. But that doesn't mean that when the psychiatrist is paid by the judge, you should dismiss his evaluation as telling you nothing more than 'the psychiatrist was paid by the judge'. On those occasions where the psychiatrist is paid by the judge, his opinion varies with the defendant, and conveys real evidence about the defendant.

"Okay, so now what's your point?"

That when my brain is not being hit by a neutrino storm, it yields honest and informative evidence that 2 + 3 = 5.

"And if your brain was hit by a neutrino storm, you'd be saying, '2 + 3 = 6 regardless of what anyone thinks of it'. Which shows how reliable that line of reasoning is."

I'm not claiming that my saying '2 + 3 = 5 no matter what anyone thinks' represents stronger numerical evidence than my saying '2 + 3 = 5'. My saying the former just tells you something extra about my epistemology, not numbers.

"And you don't think your epistemology is, oh, a little... incoherent?"

No! I think it is perfectly coherent to simultaneously hold all of the following:

- 2 + 3 = 5.

- If neutrinos make me believe "2 + 3 = 6", then 2 + 3 = 5.

- If neutrinos make me believe "2 + 3 = 6", then I will say "2 + 3 = 6".

- If neutrinos make me believe that "2 + 3 = 6", then I will thereafter assert that "If neutrinos make me believe '2 + 3 = 5', then 2 + 3 = 6".

- The cause of my thinking that "2 + 3 = 5 independently of what anyone thinks" is that my current mind, when it subjunctively recomputes the value of 2 + 3 under the assumption that my imagined self is hit by neutrinos, does not see the imagined self's beliefs as changing the dots, and my current brain just visualizes two dots plus three dots, as before, so that the imagination of my current brain shows the same result.

- If I were actually hit by neutrinos, my brain would compute a different result, and I would assert "2 + 3 = 6 independently of what anyone thinks."

- 2 + 3 = 5 independently of what anyone thinks.

- Since 2 + 3 will in fact go on equaling 5 regardless of what I imagine about it or how my brain visualizes cases where my future self has different beliefs, it's a good thing that my imagination doesn't visualize the result as depending on my beliefs.

"Now that's just crazy talk!"

No, you're the crazy one! You're collapsing your levels; you think that just because my brain asks a question, it should start mixing up queries about the state of my brain into the question. Not every question my brain asks is about my brain!

Just because something is computed in my brain, doesn't mean that my computation has to depend on my brain's representation of my brain. It certainly doesn't mean that the actual quantity depends on my brain! It's my brain that computes my beliefs about gravity, and if neutrinos hit me I will come to a different conclusion; but that doesn't mean that I can think different and fly. And I don't think I can think different and fly, either!

I am not a calculator who, when someone presses my "2" and "+" and "3" buttons, computes, "What do I output when someone presses 2 + 3?" I am a calculator who computes "What is 2 + 3?" The former is a circular question that can consistently return any answer - which makes it not very helpful.

Shouldn't we expect non-circular questions to be the normal case? The brain evolved to guess at the state of the environment, not guess at 'what the brain will think is the state of the environment'. Even when the brain models itself, it is trying to know itself, not trying to know what it will think about itself.

Judgments that depend on our representations of anyone's state of mind, like "It's okay to kiss someone only if they want to be kissed", are the exception rather than the rule.

Most quantities we bother to think about at all, will appear to be 'the same regardless of what anyone thinks of them'. When we imagine thinking differently about the quantity, we will imagine the quantity coming out the same; it will feel "subjunctively objective".

And there's nothing wrong with that! If something appears to be the same regardless of what anyone thinks, then maybe that's because it actually is the same regardless of what anyone thinks.

Even if you explain that the quantity appears to stay the same in my imagination, merely because my current brain computes it the same way - well, how else would I imagine something, except with my current brain? Should I imagine it using a rock?

"Okay, so it's possible for something that appears thought-independent, to actually be thought-independent. But why do you think that 2 + 3 = 5, in particular, has some kind of existence independently of the dots you imagine?"

Because two sheep plus three sheep equals five sheep, and this appears to be true in every mountain and every island, every swamp and every plain and every forest.

And moreover, it is also true of two rocks plus three rocks.

And further, when I press buttons upon a calculator and activate a network of transistors, it successfully predicts how many sheep or rocks I will find.

Since all these quantities, correlate with each other and successfully predict each other, surely they must have something like a common cause, a similarity that factors out? Something that is true beyond and before the concrete observations? Something that the concrete observations hold in common? And this commonality is then also the sponsor of my answer, 'five', that I find in my own brain.

"But my dear sir, if the fact of 2 + 3 = 5 exists somewhere outside your brain... then where is it?"

Damned if I know.

Sequence: Metaethics

Can Counterfactuals Be True?

Does Your Morality Care What You Think?

Referenced by: Does Your Morality Care What You Think? • Changing Your Metaethics • The Meaning of Right • Morality as Fixed Computation

Original with comments: Math is Subjunctively Objective

Does Your Morality Care What You Think?

Eliezer Yudkowsky, 26 July 2008 12:25AM

Followup to: Math is Subjunctively Objective, The Moral Void, Is Morality Given?

Thus I recall the study, though I cannot recall the citation:

Children, at some relatively young age, were found to distinguish between:

- The teacher, by saying that we're allowed to stand on our desks, can make it right to do so.

- The teacher, by saying that I'm allowed to take something from another child's backpack, cannot make it right to do so.

Obert: "Well, I don't know the citation, but it sounds like a fascinating study. So even children, then, realize that moral facts are givens, beyond the ability of teachers or parents to alter."

Subhan: "You say that like it's a good thing. Children may also think that people in Australia have to wear heavy boots from falling off the other side of the Earth."

Obert: "Call me Peter Pan, then, because I never grew up on this one. Of course it doesn't matter what the teacher says. It doesn't matter what I say. It doesn't even matter what I think. Stealing is wrong. Do you disagree?"

Subhan: "You don't see me picking your pockets, do you? Isn't it enough that I choose not to steal from you - do I have to pretend it's the law of the universe?"

Obert: "Yes, or I can't trust your commitment."

Subhan: "A... revealing remark. But really, I don't think that this experimental result seems at all confusing, in light of the recent discussion of subjunctive objectivity - a discussion in which Eliezer strongly supported my position, by the way."

Obert: "Really? I thought Eliezer was finally coming out in favor of my position."

Subhan: "Huh? How do you get that?"

Obert: "The whole subtext of 'Math is Subjunctively Objective' is that morality is just like math! Sure, we compute morality inside our own brains - where else would we compute it? But just because we compute a quantity inside our own brains, doesn't mean that what is computed has a dependency on our own state of mind."

Subhan: "I think we must have been reading different Overcoming Bias posts! The whole subtext of 'Math is Subjunctively Objective' is to explain away why morality seems objective - to show that the feeling of a fixed given can arise without any external referent. When you imagine yourself thinking that killing is right, your brain-that-imagines hasn't yet been altered, so you carry out that moral imagination with your current brain, and conclude: 'Even if I thought killing were right, killing would still be wrong.' But this doesn't show that killing-is-wrong is a fixed fact from outside you."

Obert: "Like, say, 2 + 3 = 5 is a fixed fact. Eliezer wrote: 'If something appears to be the same regardless of what anyone thinks, then maybe that's because it actually is the same regardless of what anyone thinks.' I'd say that subtext is pretty clear!"

Subhan: "On the contrary. Naively, you might imagine your future self thinking differently of a thing, and visualize that the thing wouldn't thereby change, and conclude that the thing existed outside you. Eliezer shows how this is not necessarily the case. So you shouldn't trust your intuition that the thing is objective - it might be that the thing exists outside you, or it might not. It has to be argued separately from the feeling of subjunctive objectivity. In the case of 2 + 3 = 5, it's at least reasonable to wonder if math existed before humans. Physics itself seems to be made of math, and if we don't tell a story where physics was around before humans could observe it, it's hard to give a coherent account of how we got here. But there's not the slightest evidence that morality was at work in the universe before humans got here. We created it."

Obert: "I know some very wise children who would disagree with you."

Subhan: "Then they're wrong! If children learned in school that it was okay to steal, they would grow up believing it was okay to steal."

Obert: "Not if they saw that stealing hurt the other person, and felt empathy for their pain. Empathy is a human universal."

Subhan: "So we take a step back and say that evolution created the emotions that gave rise to morality, it doesn't put morality anywhere outside us. But what you say might not even be true - if theft weren't considered a crime, the other child might not feel so hurt by it. And regardless, it is rare to find any child capable of fully reconsidering the moral teachings of its society."

Obert: "I hear that, in a remarkable similarity to Eliezer, your parents were Orthodox Jewish and you broke with religion as a very young child."

Subhan: "I doubt that I was internally generating de novo moral philosophy. I was probably just wielding, against Judaism, the morality of the science fiction that actually socialized me."

Obert: "Perhaps you underestimate yourself. How much science fiction had you read at the age of five, when you realized it was dumb to recite Hebrew prayers you couldn't understand? Children may see errors that adults are too adept at fooling themselves to realize."

Subhan: "Hah! In all probability, if the teacher had in fact said that it was okay to take things from other children's backpacks, the children would in fact have thought it was right to steal."

Obert: "Even if true, that doesn't prove anything. It is quite coherent to simultaneously hold that:"

- "Stealing is wrong."

- "If a neutrino storm makes me believe 'stealing is right', then stealing is wrong."

- "If a neutrino storm makes me believe 'stealing is right', then I will say, 'If a neutrino storm makes me believe ''stealing is wrong'', then stealing is right.'"

Subhan: "Fine, it's coherent, but that doesn't mean it's true. The morality that the child has in fact learned from the teacher - or their parents, or the other children, or the television, or their parents' science fiction collection - doesn't say, 'Don't steal because the teacher says so.' The learned morality just says, 'Don't steal.' The cognitive procedure by which the children were taught to judge, does not have an internal dependency on what the children believe the teacher believes. That's why, in their moral imagination, it feels objective. But where did they acquire that morality in the first place? From the teacher!"

Obert: "So? I don't understand - you're saying that because they learned about morality from the teacher, they should think that morality has to be about the teacher? That they should think the teacher has the power to make it right to steal? How does that follow? It is quite coherent to simultaneously hold that -"

Subhan: "I'm saying that they got the morality from the teacher! Not from some mysterious light in the sky!"

Obert: "Look, I too read science fiction and fantasy as a child, and I think I may have been to some degree socialized by it -"

Subhan: "What a remarkable coincidence."

Obert: "The stories taught me that it was right to care about people who were different from me - aliens with strange shapes, aliens made of something other than carbon atoms, AIs who had been created rather than evolved, even things that didn't think like a human. But none of the stories ever said, 'You should care about people of different shapes and substrates because science fiction told you to do it, and what science fiction says, goes.' I wouldn't have bought that."

Subhan: "Are you sure you wouldn't have? That's how religion works."

Obert: "Didn't work on you. Anyway, the novels said to care about the aliens because they had inner lives and joys - or because I wouldn't want aliens to mistreat humans - or because shape and substrate never had anything to do with what makes a person a person. And you know, that still seems to me like a good justification."

Subhan: "Of course; you were told it was a good justification - maybe not directly, but the author showed other characters responding to the argument."

Obert: "It's not like the science fiction writers were making up their morality from scratch. They were working at the end of a chain of moral arguments and debates that stretches back to the Greeks, probably to before writing, maybe to before the dawn of modern humanity. You can learn morality, not just get pressed into it like a Jello mold. If you learn 2 + 3 = 5 from a teacher, it doesn't mean the teacher has the power to add two sheep to three sheep and get six sheep. If you would have spouted back '2 + 3 = 6' if the teacher said so, that doesn't change the sheep, it just means that you don't really understand the subject. So too with morality."

Subhan: "Okay, let me try a different tack. You, I take it, agree with both of these statements:"

- "If I preferred to kill people, it would not become right to kill people."

- "If I preferred to eat anchovy pizzas, it would become right to eat anchovy pizzas."

Obert: "Well, there are various caveats I'd attach to both of those. Like, in any circumstance where I really did prefer to kill someone, there'd be a high probability he was about to shoot me, or something. And there's all kinds of ways that eating an anchovy pizza could be wrong, like if I was already overweight. And I don't claim to be certain of anything when it comes to morality. But on the whole, and omitting all objections and knock-on effects, I agree."

Subhan: "It's that second statement I'm really interested in. How does your wanting to eat an anchovy pizza make it right?"

Obert: "Because ceteris paribus, in the course of ordinary life as we know it, and barring unspecified side effects, it is good for sentient beings to get what they want."

Subhan: "And why doesn't that apply to the bit about killing, then?"

Obert: "Because the other person doesn't want to die. Look, the whole reason why it's right in the first place for me to eat pepperoni pizza - the original justification - is that I enjoy doing so. Eating pepperoni pizza makes me happy, which is ceteris paribus a good thing. And eating anchovy pizza - blegh! Ceteris paribus, it's not good for sentient beings to experience disgusting tastes. But if my taste in pizza changes, that changes the consequneces of eating, which changes the moral justification, and so the moral judgment changes as well. But the reasons for not killing are in terms of the other person having an inner life that gets snuffed out - a fact that doesn't change depending on my own state of mind."

Subhan: "Oh? I was guessing that the difference had something to do with the social disapproval that would be leveled at murder, but not at eating anchovy pizza."

Obert: "As usual, your awkward attempts at rationalism have put you out of touch with self-evident moral truths. That's just not how I, or other real people, actually think! If I want to bleep bleep bleep a consenting adult, it doesn't matter whether society approves. Society can go bleep bleep bleep bleep bleep -"

Subhan: "Or so science fiction taught you."

Obert: "Spider Robinson's science fiction, to be precise. 'Whatever turns you on' shall be the whole of the law. So long as the 'you' is plural."

Subhan: "So that's where you got that particular self-evident moral truth. Was it also Spider Robinson who told you that it was self-evident?"

Obert: "No, I thought about that for a while, and then decided myself."

Subhan: "You seem to be paying remarkably close attention to what people want. Yet you insist that what validates this attention, is some external standard that makes the satisfaction of desires, good. Can't you just admit that, by empathy and vicarious experience and evolved fellow-feeling, you want others to get what they want? When does this external standard ever say that it's good for something to happen that someone doesn't want?"

Obert: "Every time you've got to tell your child to lay off the ice cream, he'll grow more fat cells that will make it impossible for him to lose weight as an adult."

Subhan: "And could something good happen that no one wanted?"

Obert: "I rather expect so. I don't think we're all entirely past our childhoods. In some ways the human species itself strikes me as being a sort of toddler in the 'No!' stage."

Subhan: "Look, there's a perfectly normal and non-mysterious chain of causality that describes where morality comes from, and it's not from outside humans. If you'd been told that killing was right, or if you'd evolved to enjoy killing - much more than we already do, I mean - or if you really did have a mini-stroke that damaged your frontal lobe, then you'd be going around saying, 'Killing is right regardless of what anyone thinks of it'. No great light in the sky would correct you. There is nothing else to the story."

Obert: "Really, I think that in this whole debate between us, there is surprisingly litle information to be gained by such observations as 'You only say that because your brain makes you say it.' If a neutrino storm hit me, I might say '2 + 3 = 6', but that wouldn't change arithmetic. It would just make my brain compute something other than arithmetic. And these various misfortunes that you've described, wouldn't change the crime of murder. They would just make my brain compute something other than morality."

Sequence: Metaethics

Math is Subjunctively Objective

Changing Your Metaethics

Referenced by: Changing Your Metaethics • Setting Up Metaethics • The Meaning of Right

Original with comments: Does Your Morality Care What You Think?

Changing Your Metaethics

Eliezer Yudkowsky, 27 July 2008 12:36PM

Followup to: The Moral Void, Joy in the Merely Real, No Universally Compelling Arguments, Where Recursive Justification Hits Bottom, The Gift We Give To Tomorrow, Does Your Morality Care What You Think?, Existential Angst Factory, ...

If you say, "Killing people is wrong," that's morality. If you say, "You shouldn't kill people because God prohibited it," or "You shouldn't kill people because it goes against the trend of the universe", that's metaethics.

Just as there's far more agreement on Special Relativity than there is on the question "What is science?", people find it much easier to agree "Murder is bad" than to agree what makes it bad, or what it means for something to be bad.

People do get attached to their metaethics. Indeed they frequently insist that if their metaethic is wrong, all morality necessarily falls apart. It might be interesting to set up a panel of metaethicists - theists, Objectivists, Platonists, etc. - all of whom agree that killing is wrong; all of whom disagree on what it means for a thing to be "wrong"; and all of whom insist that if their metaethic is untrue, then morality falls apart.

Clearly a good number of people, if they are to make philosophical progress, will need to shift metathics at some point in their lives. You may have to do it.

At that point, it might be useful to have an open line of retreat - not a retreat from morality, but a retreat from Your-Current-Metaethic. (You know, the one that, if it is not true, leaves no possible basis for not killing people.)

And so I've been setting up these lines of retreat, in many and various posts, summarized below. For I have learned that to change metaethical beliefs is nigh-impossible in the presence of an unanswered attachment.

If, for example, someone believes the authority of "Thou Shalt Not Kill" derives from God, then there are several and well-known things to say that can help set up a line of retreat - as opposed to immediately attacking the plausibility of God. You can say, "Take personal responsibility! Even if you got orders from God, it would be your own decision to obey those orders. Even if God didn't order you to be moral, you could just be moral anyway."

The above argument actually generalizes to quite a number of metaethics - you just substitute Their-Favorite-Source-Of-Morality, or even the word "morality", for "God". Even if your particular source of moral authority failed, couldn't you just drag the child off the train tracks anyway? And indeed, who is it but you, that ever decided to follow this source of moral authority in the first place? What responsibility are you really passing on?

So the most important line of retreat is the one given in The Moral Void: If your metaethic stops telling you to save lives, you can just drag the kid off the train tracks anyway. To paraphrase Piers Anthony, only those who have moralities worry over whether or not they have them. If your metaethic tells you to kill people, why should you even listen? Maybe that which you would do even if there were no morality, is your morality.

The point being, of course, not that no morality exists; but that you can hold your will in place, and not fear losing sight of what's important to you, while your notions of the nature of morality change.

Other posts are there to set up lines of retreat specifically for more naturalistic metaethics. It may make more sense where I'm coming from on these, once I actually present my metaethic; but I thought it wiser to set them up in advance, to leave lines of retreat.

Joy in the Merely Real and Explaining vs. Explaining Away argue that you shouldn't be disappointed in any facet of life, just because it turns out to be explicable instead of inherently mysterious: for if we cannot take joy in the merely real, our lives shall be empty indeed.

No Universally Compelling Arguments sets up a line of retreat from the desire to have everyone agree with our moral arguments. There's a strong moral intuition which says that if our moral arguments are right, by golly, we ought to be able to explain them to people. This may be valid among humans, but you can't explain moral arguments to a rock. There is no ideal philosophy student of perfect emptiness who can be persuaded to implement modus ponens, starting without modus ponens. If a mind doesn't contain that which is moved by your moral arguments, it won't respond to them.

But then isn't all morality circular logic, in which case it falls apart? Where Recursive Justification Hits Bottom and My Kind of Reflection explain the difference between a self-consistent loop through the meta-level, and actual circular logic. You shouldn't find yourself saying "The universe is simple because it is simple", or "Murder is wrong because it is wrong"; but neither should you try to abandon Occam's Razor while evaluating the probability that Occam's Razor works, nor should you try to evaluate "Is murder wrong?" from somewhere outside your brain. There is no ideal philosophy student of perfect emptiness to which you can unwind yourself - try to find the perfect rock to stand upon, and you'll end up as a rock. So instead use the full force of your intelligence, your full rationality and your full morality, when you investigate the foundations of yourself.

The Gift We Give To Tomorrow sets up a line of retreat for those afraid to allow a causal role for evolution, in their account of how morality came to be. (Note that this is extremely distinct from granting evolution a justificational status in moral theories.) Love has to come into existence somehow - for if we cannot take joy in things that can come into existence, our lives will be empty indeed. Evolution may not be a particularly pleasant way for love to evolve, but judge the end product - not the source. Otherwise you would be committing what is known (appropriately) as The Genetic Fallacy: causation is not the same concept as justification. It's not like you can step outside the brain evolution gave you: Rebelling against nature is only possible from within nature.

The earlier series on Evolutionary Psychology should dispense with the metaethical confusion of believing that any normal human being thinks about their reproductive fitness, even unconsciously, in the course of making decisions. Only evolutionary biologists even know how to define genetic fitness, and they know better than to think it defines morality.

Alarming indeed is the thought that morality might be computed inside our own minds - doesn't this imply that morality is a mere thought? Doesn't it imply that whatever you think is right, must be right? Posts such as Does Your Morality Care What You Think? and its predecessors, Math is Subjunctively Objective and Probability is Subjectively Objective, set up the needed line of retreat: Just because a quantity is computed inside your head, doesn't mean that the quantity computed is about your thoughts. There's a difference between a calculator that calculates "What is 2 + 3?" and "What do I output when someone presses '2', '+', and '3'?"

And finally Existential Angst Factory offers the notion that if life seems painful, reductionism may not be the real source of your problem - if living in a world of mere particles seems too unbearable, maybe your life isn't exciting enough on its own?

If all goes well, my next post will set up the metaethical question and its methodology, and I'll present my actual answer on Monday.

And if you're wondering why I deem this business of metaethics important, when it is all going to end up adding up to moral normality... telling you to pull the child off the train tracks, rather than the converse...

Well, there is opposition to rationality from people who think it drains meaning from the universe.

And this is a special case of a general phenomenon, in which many many people get messed up by misunderstanding where their morality comes from. Poor metaethics forms part of the teachings of many a cult, including the big ones. My target audience is not just people who are afraid that life is meaningless, but also those who've concluded that love is a delusion because real morality has to involve maximizing your inclusive fitness, or those who've concluded that unreturned kindness is evil because real morality arises only from selfishness, etc.

But the real reason, of course...

Sequence: Metaethics

Does Your Morality Care What You Think?

Setting Up Metaethics

Referenced by: The Meaning of Right • Humans in Funny Suits • Mirrors and Paintings • Fighting a Rearguard Action Against the Truth • ...And Say No More Of It

Original with comments: Changing Your Metaethics

Setting Up Metaethics

Eliezer Yudkowsky, 28 July 2008 02:25AM

Followup to: Is Morality Given?, Is Morality Preference?, Moral Complexities, Could Anything Be Right?, The Bedrock of Fairness, ...

Intuitions about morality seem to split up into two broad camps: morality-as-given and morality-as-preference.

Some perceive morality as a fixed given, independent of our whims, about which we form changeable beliefs. This view's great advantage is that it seems more normal up at the level of everyday moral conversations: it is the intuition underlying our everyday notions of "moral error", "moral progress", "moral argument", or "just because you want to murder someone doesn't make it right".

Others choose to describe morality as a preference - as a desire in some particular person; nowhere else is it written. This view's great advantage is that it has an easier time living with reductionism - fitting the notion of "morality" into a universe of mere physics. It has an easier time at the meta level, answering questions like "What is morality?" and "Where does morality come from?"

Both intuitions must contend with seemingly impossible questions. For example, Moore's Open Question: Even if you come up with some simple answer that fits on T-Shirt, like "Happiness is the sum total of goodness!", you would need to argue the identity. It isn't instantly obvious to everyone that goodness is happiness, which seems to indicate that happiness and rightness were different concepts to start with. What was that second concept, then, originally?

Or if "Morality is mere preference!" then why care about human preferences? How is it possible to establish any "ought" at all, in a universe seemingly of mere "is"?

So what we should want, ideally, is a metaethic that:

1. Adds up to moral normality, including moral errors, moral progress, and things you should do whether you want to or not;

2. Fits naturally into a non-mysterious universe, postulating no exception to reductionism;

3. Does not oversimplify humanity's complicated moral arguments and many terminal values;

4. Answers all the impossible questions.

I'll present that view tomorrow.

Today's post is devoted to setting up the question.

Consider "free will", already dealt with in these posts. On one level of organization, we have mere physics, particles that make no choices. On another level of organization, we have human minds that extrapolate possible futures and choose between them. How can we control anything, even our own choices, when the universe is deterministic?

To dissolve the puzzle of free will, you have to simultaneously imagine two levels of organization while keeping them conceptually distinct. To get it on a gut level, you have to see the level transition - the way in which free will is how the human decision algorithm feels from inside. (Being told flatly "one level emerges from the other" just relates them by a magical transition rule, "emergence".)

For free will, the key is to understand how your brain computes whether you "could" do something - the algorithm that labels reachable states. Once you understand this label, it does not appear particularly meaningless - "could" makes sense - and the label does not conflict with physics following a deterministic course. If you can see that, you can see that there is no conflict between your feeling of freedom, and deterministic physics. Indeed, I am perfectly willing to say that the feeling of freedom is correct, when the feeling is interpreted correctly.

In the case of morality, once again there are two levels of organization, seemingly quite difficult to fit together:

On one level, there are just particles without a shred of should-ness built into them - just like an electron has no notion of what it "could" do - or just like a flipping coin is not uncertain of its own result.

On another level is the ordinary morality of everyday life: moral errors, moral progress, and things you ought to do whether you want to do them or not.

And in between, the level transition question: What is this should-ness stuff?

Award yourself a point if you thought, "But wait, that problem isn't quite analogous to the one of free will. With free will it was just a question of factual investigation - look at human psychology, figure out how it does in fact generate the feeling of freedom. But here, it won't be enough to figure out how the mind generates its feelings of should-ness. Even after we know, we'll be left with a remaining question - is that how we should calculate should-ness? So it's not just a matter of sheer factual reductionism, it's a moral question."

Award yourself two points if you thought, "...oh, wait, I recognize that pattern: It's one of those strange loops through the meta-level we were talking about earlier."

And if you've been reading along this whole time, you know the answer isn't going to be, "Look at this fundamentally moral stuff!"

Nor even, "Sorry, morality is mere preference, and right-ness is just what serves you or your genes; all your moral intuitions otherwise are wrong, but I won't explain where they come from."

Of the art of answering impossible questions, I have already said much: Indeed, vast segments of my Overcoming Bias posts were created with that specific hidden agenda.

The sequence on anticipation fed into Mysterious Answers to Mysterious Questions, to prevent the Primary Catastrophic Failure of stopping on a poor answer.

The Fake Utility Functions sequence was directed at the problem of oversimplified moral answers particularly.

The sequence on words provided the first and basic illustration of the Mind Projection Fallacy, the understanding of which is one of the Great Keys.

The sequence on words also showed us how to play Rationalist's Taboo, and Replace the Symbol with the Substance. What is "right", if you can't say "good" or "desirable" or "better" or "preferable" or "moral" or "should"? What happens if you try to carry out the operation of replacing the symbol with what it stands for?

And the sequence on quantum physics, among other purposes, was there to teach the fine art of not running away from Scary and Confusing Problems, even if others have failed to solve them, even if great minds failed to solve them for generations. Heroes screw up, time moves on, and each succeeding era gets an entirely new chance.

If you're just joining us here (Belldandy help you) then you might want to think about reading all those posts before, oh, say, tomorrow.

If you've been reading this whole time, then you should think about trying to dissolve the question on your own, before tomorrow. It doesn't require more than 96 insights beyond those already provided.

Next: The Meaning of Right.

Sequence: Metaethics

Changing Your Metaethics

The Meaning of Right

Referenced by: The Meaning of Right

Original with comments: Setting Up Metaethics

The Meaning of Right

Eliezer Yudkowsky, 29 July 2008 01:28AM

Continuation of: Changing Your Metaethics, Setting Up Metaethics
Followup to: Does Your Morality Care What You Think?, The Moral Void, Probability is Subjectively Objective, Could Anything Be Right?, The Gift We Give To Tomorrow, Rebelling Within Nature, Where Recursive Justification Hits Bottom, ...

(The culmination of a long series of Overcoming Bias posts; if you start here, I accept no responsibility for any resulting confusion, misunderstanding, or unnecessary angst.)

What is morality? What does the word "should", mean? The many pieces are in place: This question I shall now dissolve.

The key - as it has always been, in my experience so far - is to understand how a certain cognitive algorithm feels from inside. Standard procedure for righting a wrong question: If you don't know what right-ness is, then take a step beneath and ask how your brain labels things "right".

It is not the same question - it has no moral aspects to it, being strictly a matter of fact and cognitive science. But it is an illuminating question. Once we know how our brain labels things "right", perhaps we shall find it easier, afterward, to ask what is really and truly right.

But with that said - the easiest way to begin investigating that question, will be to jump back up to the level of morality and ask what seems right. And if that seems like too much recursion, get used to it - the other 90% of the work lies in handling recursion properly.

(Should you find your grasp on meaningfulness wavering, at any time following, check Changing Your Metaethics for the appropriate prophylactic.)

So! In order to investigate how the brain labels things "right", we are going to start out by talking about what is right. That is, we'll start out wearing our morality-goggles, in which we consider morality-as-morality and talk about moral questions directly. As opposed to wearing our reduction-goggles, in which we talk about cognitive algorithms and mere physics. Rigorously distinguishing between these two views is the first step toward mating them together.

As a first step, I offer this observation, on the level of morality-as-morality: Rightness is contagious backward in time.

Suppose there is a switch, currently set to OFF, and it is morally desirable for this switch to be flipped to ON. Perhaps the switch controls the emergency halt on a train bearing down on a child strapped to the railroad tracks, this being my canonical example. If this is the case, then, ceteris paribus and presuming the absence of exceptional conditions or further consequences that were not explicitly specified, we may consider it right that this switch should be flipped.

If it is right to flip the switch, then it is right to pull a string that flips the switch. If it is good to pull a string that flips the switch, it is right and proper to press a button that pulls the string: Pushing the button seems to have more should-ness than not pushing it.

It seems that - all else being equal, and assuming no other consequences or exceptional conditions which were not specified - value flows backward along arrows of causality.

Even in deontological moralities, if you're obligated to save the child on the tracks, then you're obligated to press the button. Only very primitive AI systems have motor outputs controlled by strictly local rules that don't model the future at all. Duty-based or virtue-based ethics are only slightly less consequentialist than consequentialism. It's hard to say whether moving your arm left or right is more virtuous without talking about what happens next.

Among my readers, there may be some who presently assert - though I hope to persuade them otherwise - that the life of a child is of no value to them. If so, they may substitute anything else that they prefer, at the end of the switch, and ask if they should press the button.

But I also suspect that, among my readers, there are some who wonder if the true morality might be something quite different from what is presently believed among the human kind. They may find it imaginable - plausible? - that human life is of no value, or negative value. They may wonder if the goodness of human happiness, is as much a self-serving delusion as the justice of slavery.

I myself was once numbered among these skeptics, because I was always very suspicious of anything that looked self-serving.

Now here's a little question I never thought to ask, during those years when I thought I knew nothing about morality:

Could make sense to have a morality in which, if we should save the child from the train tracks, then we should not flip the switch, should pull the string, and should not push the button, so that, finally, we do not push the button?

Or perhaps someone says that it is better to save the child, than to not save them; but doesn't see why anyone would think this implies it is better to press the button than not press it. (Note the resemblance to the Tortoise who denies modus ponens.)

It seems imaginable, to at least some people, that entirely different things could be should. It didn't seem nearly so imaginable, at least to me, that should-ness could fail to flow backward in time. When I was trying to question everything else, that thought simply did not occur to me.

Can you question it? Should you?

Every now and then, in the course of human existence, we question what should be done and what is right to do, what is better or worse; others come to us with assertions along these lines, and we question them, asking "Why is it right?" Even when we believe a thing is right (because someone told us that it is, or because we wordlessly feel that it is) we may still question why it is right.

Should-ness, it seems, flows backward in time. This gives us one way to question why or whether a particular event has the should-ness property. We can look for some consequence that has the should-ness property. If so, the should-ness of the original event seems to have been plausibly proven or explained.

Ah, but what about the consequence - why is it should? Someone comes to you and says, "You should give me your wallet, because then I'll have your money, and I should have your money." If, at this point, you stop asking questions about should-ness, you're vulnerable to a moral mugging.

So we keep asking the next question. Why should we press the button? To pull the string. Why should we pull the string? To flip the switch. Why should we flip the switch? To pull the child from the railroad tracks. Why pull the child from the railroad tracks? So that they live. Why should the child live?

Now there are people who, caught up in the enthusiasm, go ahead and answer that question in the same style: for example, "Because the child might eventually grow up and become a trade partner with you," or "Because you will gain honor in the eyes of others," or "Because the child may become a great scientist and help achieve the Singularity," or some such. But even if we were to answer in this style, it would only beg the next question.

Even if you try to have a chain of should stretching into the infinite future - a trick I've yet to see anyone try to pull, by the way, though I may be only ignorant of the breadths of human folly - then you would simply ask "Why that chain rather than some other?"

Another way that something can be should, is if there's a general rule that makes it should. If your belief pool starts out with the general rule "All children X: It is better for X to live than to die", then it is quite a short step to "It is better for Stephanie to live than to die". Ah, but why save all children? Because they may all become trade partners or scientists? But then where did that general rule come from?

If should-ness only comes from should-ness - from a should-consequence, or from a should-universal - then how does anything end up should in the first place?

Now human beings have argued these issues for thousands of years and maybe much longer. We do not hesitate to continue arguing when we reach a terminal value (something that has a charge of should-ness independently of its consequences). We just go on arguing about the universals.

I usually take, as my archetypal example, the undoing of slavery: Somehow, slaves' lives went from having no value to having value. Nor do I think that, back at the dawn of time, anyone was even trying to argue that slaves were better off being slaves (as it would be latter argued). They'd probably have looked at you like you were crazy if you even tried. Somehow, we got from there, to here...

And some of us would even hold this up as a case of moral progress, and look at our ancestors as having made a moral error. Which seems easy enough to describe in terms of should-ness: Our ancestors thought that they should enslave defeated enemies, but they were mistaken.

But all our philosophical arguments ultimately seem to ground in statements that no one has bothered to justify - except perhaps to plead that they are self-evident, or that any reasonable mind must surely agree, or that they are a priori truths, or some such. Perhaps, then, all our moral beliefs are as erroneous as that old bit about slavery? Perhaps we have entirely misperceived the flowing streams of should?

So I once believed was plausible; and one of the arguments I wish I could go back and say to myself, is, "If you know nothing at all about should-ness, then how do you know that the procedure, 'Do whatever Emperor Ming says' is not the entirety of should-ness? Or even worse, perhaps, the procedure, 'Do whatever maximizes inclusive genetic fitness' or 'Do whatever makes you personally happy'." The point here would have been to make my past self see that in rejecting these rules, he was asserting a kind of knowledge - that to say, "This is not morality," he must reveal that, despite himself, he knows something about morality or meta-morality. Otherwise, the procedure "Do whatever Emperor Ming says" would seem just as plausible, as a guiding principle, as his current path of "Rejecting things that seem unjustified." Unjustified - according to what criterion of justification? Why trust the principle that says that moral statements need to be justified, if you know nothing at all about morality?

What indeed would distinguish, at all, the question "What is right?" from "What is wrong?"

What is "right", if you can't say "good" or "desirable" or "better" or "preferable" or "moral" or "should"? What happens if you try to carry out the operation of replacing the symbol with what it stands for?

If you're guessing that I'm trying to inveigle you into letting me say: "Well, there are just some things that are baked into the question, when you start asking questions about morality, rather than wakalixes or toaster ovens", then you would be right. I'll be making use of that later, and, yes, will address "But why should we ask that question?"

Okay, now: morality-goggles off, reduction-goggles on.

Those who remember Possibility and Could-ness, or those familiar with simple search techniques in AI, will realize that the "should" label is behaving like the inverse of the "could" label, which we previously analyzed in terms of "reachability". Reachability spreads forward in time: if I could reach the state with the button pressed, I could reach the state with the string pulled; if I could reach the state with the string pulled, I could reach the state with the switch flipped.

Where the "could" label and the "should" label collide, the algorithm produces a plan.

Now, as I say this, I suspect that at least some readers may find themselves fearing that I am about to reduce should-ness to a mere artifact of a way that a planning system feels from inside. Once again I urge you to check Changing Your Metaethics, if this starts to happen. Remember above all the Moral Void: Even if there were no morality, you could still choose to help people rather than hurt them. This, above all, holds in place what you hold precious, while your beliefs about the nature of morality change.

I do not intend, with this post, to take away anything of value; it will all be given back before the end.

Now this algorithm is not very sophisticated, as AI algorithms go, but to apply it in full generality - to learned information, not just ancestrally encountered, genetically programmed situations - is a rare thing among animals. Put a food reward in a transparent box. Put the matching key, which looks unique and uniquely corresponds to that box, in another transparent box. Put the unique key to that box in another box. Do this with five boxes. Mix in another sequence of five boxes that doesn't lead to a food reward. Then offer a choice of two keys, one of which starts the sequence of five boxes leading to food, one of which starts the sequence leading nowhere.

Chimpanzees can learn to do this, but so far as I know, no non-primate species can pull that trick.

And as smart as chimpanzees are, they are not quite as good as humans at inventing plans - plans such as, for example, planting in the spring to harvest in the fall.

So what else are humans doing, in the way of planning?

It is a general observation that natural selection seems to reuse existing complexity, rather than creating things from scratch, whenever it possibly can - though not always in the same way that a human engineer would. It is a function of the enormous time required for evolution to create machines with many interdependent parts, and the vastly shorter time required to create a mutated copy of something already evolved.

What else are humans doing? Quite a bit, and some of it I don't understand - there are plans humans make, that no modern-day AI can.

But one of the things we are doing, is reasoning about "right-ness" the same way we would reason about any other observable property.

Are animals with bright colors often poisonous? Does the delicious nid-nut grow only in the spring? Is it usually a good idea to take with a waterskin on long hunts?

It seems that Martha and Fred have an obligation to take care of their child, and Jane and Bob are obligated to take care of their child, and Susan and Wilson have a duty to care for their child. Could it be that parents in general must take care of their children?

By representing right-ness as an attribute of objects, you can recruit a whole previously evolved system that reasons about the attributes of objects. You can save quite a lot of planning time, if you decide (based on experience) that in general it is a good idea to take a waterskin on hunts, from which it follows that it must be a good idea to take a waterskin on hunt #342.

Is this damnable for a Mind Projection Fallacy - treating properties of the mind as if they were out there in the world?

Depends on how you look at it.

This business of, "It's been a good idea to take waterskins on the last three hunts, maybe it's a good idea in general, if so it's a good idea to take a waterskin on this hunt", does seem to work.

Let's say that your mind, faced with any countable set of objects, automatically and perceptually tagged them with their remainder modulo 5. If you saw a group of 17 objects, for example, they would look remainder-2-ish. Though, if you didn't have any notion of what your neurons were doing, and perhaps no notion of modulo arithmetic, you would only see that the group of 17 objects had the same remainder-ness as a group of 2 objects. You might not even know how to count - your brain doing the whole thing automatically, subconsciously and neurally - in which case you would just have five different words for the remainder-ness attributes that we would call 0, 1, 2, 3, and 4.

If you look out upon the world you see, and guess that remainder-ness is a separate and additional attribute of things - like the attribute of having an electric charge - or like a tiny little XML tag hanging off of things - then you will be wrong. But this does not mean it is nonsense to talk about remainder-ness, or that you must automatically commit the Mind Projection Fallacy in doing so. So long as you've got a well-defined way to compute a property, it can have a well-defined output and hence an empirical truth condition.

If you're looking at 17 objects, then their remainder-ness is, indeed and truly, 2, and not 0, 3, 4, or 1. If I tell you, "Those red things you told me to look at are remainder-2-ish", you have indeed been told a falsifiable and empirical property of those red things. It is just not a separate, additional, physically existent attribute.

And as for reasoning about derived properties, and which other inherent or derived properties they correlate to - I don't see anything inherently fallacious about that.

One may notice, for example, that things which are 7 modulo 10 are often also 2 modulo 5. Empirical observations of this sort play a large role in mathematics, suggesting theorems to prove. (See Polya's How To Solve It.)

Indeed, virtually all the experience we have, is derived by complicated neural computations from the raw physical events impinging on our sense organs. By the time you see anything, it has been extensively processed by the retina, lateral geniculate nucleus, visual cortex, parietal cortex, and temporal cortex, into a very complex sort of derived computational property.

If you thought of a property like redness as residing strictly in an apple, you would be committing the Mind Projection Fallacy. The apple's surface has a reflectance which sends out a mixture of wavelengths that impinge on your retina and are processed with respect to ambient light to extract a summary color of red... But if you tell me that the apple is red, rather than green, and make no claims as to whether this is an ontologically fundamental physical attribute of the apple, then I am quite happy to agree with you.

So as long as there is a stable computation involved, or a stable process - even if you can't consciously verbalize the specification - it often makes a great deal of sense to talk about properties that are not fundamental. And reason about them, and remember where they have been found in the past, and guess where they will be found next.

(In retrospect, that should have been a separate post in the Reductionism sequence. "Derived Properties", or "Computational Properties" maybe. Oh, well; I promised you morality this day, and this day morality you shall have.)

Now let's say we want to make a little machine, one that will save the lives of children. (This enables us to save more children than we could do without a machine, just like you can move more dirt with a shovel than by hand.) The machine will be a planning machine, and it will reason about events that may or may not have the property, leads-to-child-living.

A simple planning machine would just have a pre-made model of the environmental process. It would search forward from its actions, applying a label that we might call "reachable-from-action-ness", but which might as well say "Xybliz" internally for all that it matters to the program. And it would search backward from scenarios, situations, in which the child lived, labeling these "leads-to-child-living". If situation X leads to situation Y, and Y has the label "leads-to-child-living" - which might just be a little flag bit, for all the difference it would make - then X will inherit the flag from Y. When the two labels meet in the middle, the leads-to-child-living flag will quickly trace down the stored path of reachability, until finally some particular sequence of actions ends up labeled "leads-to-child-living". Then the machine automatically executes those actions - that's just what the machine does.

Now this machine is not complicated enough to feel existential angst. It is not complicated enough to commit the Mind Projection Fallacy. It is not, in fact, complicated enough to reason abstractly about the property "leads-to-child-living-ness". The machine - as specified so far - does not notice if the action "jump in the air" turns out to always have this property, or never have this property. If "jump in the air" always led to situations in which the child lived, this could greatly simplify future planning - but only if the machine were sophisticated enough to notice this fact and use it.

If it is a fact that "jump in the air" "leads-to-child-living-ness", this fact is composed of empirical truth and logical truth. It is an empirical truth that if the world is such that if you perform the (ideal abstract) algorithm "trace back from situations where the child lives", then it will be a logical truth about the output of this (ideal abstract) algorithm that it labels the "jump in the air" action.

(You cannot always define this fact in entirely empirical terms, by looking for the physical real-world coincidence of jumping and child survival. It might be that "stomp left" also always saves the child, and the machine in fact stomps left. In which case the fact that jumping in the air would have saved the child, is a counterfactual extrapolation.)

Okay, now we're ready to bridge the levels.

As you must surely have guessed by now, this should-ness stuff is how the human decision algorithm feels from inside. It is not an extra, physical, ontologically fundamental attribute hanging off of events like a tiny little XML tag.

But it is a moral question what we should do about that - how we should react to it.

To adopt an attitude of complete nihilism, because we wanted those tiny little XML tags, and they're not physically there, strikes me as the wrong move. It is like supposing that the absence of an XML tag, equates to the XML tag being there, saying in its tiny brackets what value we should attach, and having value zero. And then this value zero, in turn, equating to a moral imperative to wear black, feel awful, write gloomy poetry, betray friends, and commit suicide.

No.

So what would I say instead?

The force behind my answer is contained in The Moral Void and The Gift We Give To Tomorrow. I would try to save lives "even if there were no morality", as it were.

And it seems like an awful shame to - after so many millions and hundreds of millions of years of evolution - after the moral miracle of so much cutthroat genetic competition producing intelligent minds that love, and hope, and appreciate beauty, and create beauty - after coming so far, to throw away the Gift of morality, just because our brain happened to represent morality in such fashion as to potentially mislead us when we reflect on the nature of morality.

This little accident of the Gift doesn't seem like a good reason to throw away the Gift; it certainly isn't a inescapable logical justification for wearing black.

Why not keep the Gift, but adjust the way we reflect on it?

So here's my metaethics:

I earlier asked,

What is "right", if you can't say "good" or "desirable" or "better" or "preferable" or "moral" or "should"? What happens if you try to carry out the operation of replacing the symbol with what it stands for?

I answer that if you try to replace the symbol "should" with what it stands for, you end up with quite a large sentence.

For the much simpler save-life machine, the "should" label stands for leads-to-child-living-ness.

For a human this is a much huger blob of a computation that looks like, "Did everyone survive? How many people are happy? Are people in control of their own lives? ..." Humans have complex emotions, have many values - the thousand shards of desire, the godshatter of natural selection. I would say, by the way, that the huge blob of a computation is not just my present terminal values (which I don't really have - I am not a consistent expected utility maximizers); the huge blob of a computation includes the specification of those moral arguments, those justifications, that would sway me if I heard them. So that I can regard my present values, as an approximation to the ideal morality that I would have if I heard all the arguments, to whatever extent such an extrapolation is coherent.

No one can write down their big computation; it is not just too large, it is also unknown to its user. No more could you print out a listing of the neurons in your brain. You never mention your big computation - you only use it, every hour of every day.

Now why might one identify this enormous abstract computation, with what-is-right?

If you identify rightness with this huge computational property, then moral judgments are subjunctively objective (like math), subjectively objective (like probability), and capable of being true (like counterfactuals).

You will find yourself saying, "If I wanted to kill someone - even if I thought it was right to kill someone - that wouldn't make it right." Why? Because what is right is a huge computational property - an abstract computation - not tied to the state of anyone's brain, including your own brain.

This distinction was introduced earlier in 2-Place and 1-Place Words. We can treat the word "sexy" as a 2-place function that goes out and hoovers up someone's sense of sexiness, and then eats an object of admiration. Or we can treat the word "sexy" as meaning a 1-place function, a particular sense of sexiness, like Sexiness_20934, that only accepts one argument, an object of admiration.

Here we are treating morality as a 1-place function. It does not accept a person as an argument, spit out whatever cognitive algorithm they use to choose between actions, and then apply that algorithm to the situation at hand. When I say right, I mean a certain particular 1-place function that just asks, "Did the child live? Did anyone else get killed? Are people happy? Are they in control of their own lives? Has justice been served?" ... and so on through many, many other elements of rightness. (And perhaps those arguments that might persuade me otherwise, which I have not heard.)

Hence the notion, "Replace the symbol with what it stands for."

Since what's right is a 1-place function, if I subjunctively imagine a world in which someone has slipped me a pill that makes me want to kill people, then, in this subjunctive world, it is not right to kill people. That's not merely because I'm judging with my current brain. It's because when I say right, I am referring to a 1-place function. Rightness doesn't go out and hoover up the current state of my brain, in this subjunctive world, before producing the judgment "Oh, wait, it's now okay to kill people." When I say right, I don't mean "that which my future self wants", I mean the function that looks at a situation and asks, "Did anyone get killed? Are people happy? Are they in control of their own lives? ..."

And once you've defined a particular abstract computation that says what is right - or even if you haven't defined it, and it's computed in some part of your brain you can't perfectly print out, but the computation is stable - more or less - then as with any other derived property, it makes sense to speak of a moral judgment being true. If I say that today was a good day, you've learned something empirical and falsifiable about my day - if it turns out that actually my grandmother died, you will suspect that I was originally lying.

The apparent objectivity of morality has just been explained - and not explained away. For indeed, if someone slipped me a pill that made me want to kill people, nonetheless, it would not be right to kill people. Perhaps I would actually kill people, in that situation - but that is because something other than morality would be controlling my actions.

Morality is not just subjunctively objective, but subjectively objective. I experience it as something I cannot change. Even after I know that it's myself who computes this 1-place function, and not a rock somewhere - even after I know that I will not find any star or mountain that computes this function, that only upon me is it written - even so, I find that I wish to save lives, and that even if I could change this by an act of will, I would not choose to do so. I do not wish to reject joy, or beauty, or freedom. What else would I do instead? I do not wish to reject the Gift that natural selection accidentally barfed into me. This is the principle of The Moral Void and The Gift We Give To Tomorrow.

Our origins may seem unattractive, our brains untrustworthy.

But love has to enter the universe somehow, starting from non-love, or love cannot enter time.

And if our brains are untrustworthy, it is only our own brains that say so. Do you sometimes think that human beings are not very nice? Then it is you, a human being, who says so. It is you, a human being, who judges that human beings could do better. You will not find such written upon the stars or the mountains: they are not minds, they cannot think.

In this, of course, we find a justificational strange loop through the meta-level. Which is unavoidable so far as I can see - you can't argue morality, or any kind of goal optimization, into a rock. But note the exact structure of this strange loop: there is no general moral principle which says that you should do what evolution programmed you to do. There is, indeed, no general principle to trust your moral intuitions! You can find a moral intuition within yourself, describe it - quote it - consider it deliberately and in the full light of your entire morality, and reject it, on grounds of other arguments. What counts as an argument is also built into the rightness-function.

Just as, in the strange loop of rationality, there is no general principle in rationality to trust your brain, or to believe what evolution programmed you to believe - but indeed, when you ask which parts of your brain you need to rebel against, you do so using your current brain. When you ask whether the universe is simple, you can consider the simple hypothesis that the universe's apparent simplicity is explained by its actual simplicity.

Rather than trying to unwind ourselves into rocks, I proposed that we should use the full strength of our current rationality, in reflecting upon ourselves - that no part of ourselves be immune from examination, and that we use all of ourselves that we currently believe in to examine it.

You would do the same thing with morality; if you consider that a part of yourself might be considered harmful, then use your best current guess at what is right, your full moral strength, to do the considering. Why should we want to unwind ourselves to a rock? Why should we do less than our best, when reflecting? You can't unwind past Occam's Razor, modus ponens, or morality and it's not clear why you should try.

For any part of rightness, you can always imagine another part that overrides it - it would not be right to drag the child from the train tracks, if this resulted in everyone on Earth becoming unable to love - or so I would judge. For every part of rightness you examine, you will find that it cannot be the sole and perfect and only criterion of rightness. This may lead to the incorrect inference that there is something beyond, some perfect and only criterion from which all the others are derived - but that does not follow. The whole is the sum of the parts. We ran into an analogous situation with free will, where no part of ourselves seems perfectly decisive.

The classic dilemma for those who would trust their moral intuitions, I believe, is the one who says: "Interracial marriage is repugnant - it disgusts me - and that is my moral intuition!" I reply, "There is no general rule to obey your intuitions. You just mentioned intuitions, rather than using them. Very few people have legitimate cause to mention intuitions - Friendly AI programmers, for example, delving into the cognitive science of things, have a legitimate reason to mention them. Everyone else just has ordinary moral arguments, in which they use their intuitions, for example, by saying, 'An interracial marriage doesn't hurt anyone, if both parties consent'. I do not say, 'And I have an intuition that anything consenting adults do is right, and all intuitions must be obeyed, therefore I win.' I just offer up that argument, and any others I can think of, to weigh in the balance."

Indeed, evolution that made us cannot be trusted - so there is no general principle to trust it! Rightness is not defined in terms of automatic correspondence to any possible decision we actually make - so there's no general principle that says you're infallible! Just do what is, ahem, right - to the best of your ability to weigh the arguments you have heard, and ponder the arguments you may not have heard.

If you were hoping to have a perfectly trustworthy system, or to have been created in correspondence with a perfectly trustworthy morality - well, I can't give that back to you; but even most religions don't try that one. Even most religions have the human psychology containing elements of sin, and even most religions don't actually give you an effectively executable and perfect procedure, though they may tell you "Consult the Bible! It always works!"

If you hoped to find a source of morality outside humanity - well, I can't give that back, but I can ask once again: Why would you even want that? And what good would it do? Even if there were some great light in the sky - something that could tell us, "Sorry, happiness is bad for you, pain is better, now get out there and kill some babies!" - it would still be your own decision to follow it. You cannot evade responsibility.

There isn't enough mystery left to justify reasonable doubt as to whether the causal origin of morality is something outside humanity. We have evolutionary psychology. We know where morality came from. We pretty much know how it works, in broad outline at least. We know there are no little XML value tags on electrons (and indeed, even if you found them, why should you pay attention to what is written there?)

If you hoped that morality would be universalizable - sorry, that one I really can't give back. Well, unless we're just talking about humans. Between neurologically intact humans, there is indeed much cause to hope for overlap and coherence; and a great and reasonable doubt as to whether any present disagreement is really unresolvable, even it seems to be about "values". The obvious reason for hope is the psychological unity of humankind, and the intuitions of symmetry, universalizability, and simplicity that we execute in the course of our moral arguments. (In retrospect, I should have done a post on Interpersonal Morality before this...)

If I tell you that three people have found a pie and are arguing about how to divide it up, the thought "Give one-third of the pie to each" is bound to occur to you - and if the three people are humans, it's bound to occur to them, too. If one of them is a psychopath and insists on getting the whole pie, though, there may be nothing for it but to say: "Sorry, fairness is not 'what everyone thinks is fair', fairness is everyone getting a third of the pie". You might be able to resolve the remaining disagreement by politics and game theory, short of violence - but that is not the same as coming to agreement on values. (Maybe you could persuade the psychopath that taking a pill to be more human, if one were available, would make them happier? Would you be justified in forcing them to swallow the pill? These get us into stranger waters that deserve a separate post.)

If I define rightness to include the space of arguments that move me, then when you and I argue about what is right, we are arguing our approximations to what we would come to believe if we knew all empirical facts and had a million years to think about it - and that might be a lot closer than the present and heated argument. Or it might not. This gets into the notion of 'construing an extrapolated volition' which would be, again, a separate post.

But if you were stepping outside the human and hoping for moral arguments that would persuade any possible mind, even a mind that just wanted to maximize the number of paperclips in the universe, then sorry - the space of possible mind designs is too large to permit universally compelling arguments. You are better off treating your intuition that your moral arguments ought to persuade others, as applying only to other humans who are more or less neurologically intact. Trying it on human psychopaths would be dangerous, yet perhaps possible. But a paperclip maximizer is just not the sort of mind that would be moved by a moral argument. (This will definitely be a separate post.)

Once, in my wild and reckless youth, I tried dutifully - I thought it was my duty - to be ready and willing to follow the dictates of a great light in the sky, an external objective morality, when I discovered it. I questioned everything, even altruism toward human lives, even the value of happiness. Finally I realized that there was no foundation but humanity - no evidence pointing to even a reasonable doubt that there was anything else - and indeed I shouldn't even want to hope for anything else - and indeed would have no moral cause to follow the dictates of a light in the sky, even if I found one.

I didn't get back immediately all the pieces of myself that I had tried to deprecate - it took time for the realization "There is nothing else" to sink in. The notion that humanity could just... you know... live and have fun... seemed much too good to be true, so I mistrusted it. But eventually, it sank in that there really was nothing else to take the place of beauty. And then I got it back.

So you see, it all really does add up to moral normality, very exactly in fact. You go on with the same morals as before, and the same moral arguments as before. There is no sudden Grand Overlord Procedure to which you can appeal to get a perfectly trustworthy answer. You don't know, cannot print out, the great rightness-function; and even if you could, you would not have enough computational power to search the entire specified space of arguments that might move you. You will just have to argue it out.

I suspect that a fair number of those who propound metaethics do so in order to have it add up to some new and unusual moral - else why would they bother? In my case, I bother because I am a Friendly AI programmer and I have to make a physical system outside myself do what's right; for which purpose metaethics becomes very important indeed. But for the most part, the effect of my proffered metaethic is threefold:

- Anyone worried that reductionism drains the meaning from existence can stop worrying;

- Anyone who was rejecting parts of their human existence based on strange metaethics - i.e., "Why should I care about others, if that doesn't help me maximize my inclusive genetic fitness?" - can welcome back all the parts of themselves that they once exiled.

- You can stop arguing about metaethics, and go back to whatever ordinary moral argument you were having before then. This knowledge will help you avoid metaethical mistakes that mess up moral arguments, but you can't actually use it to settle debates unless you can build a Friendly AI.

And, oh yes - why is it right to save a child's life?

Well... you could ask "Is this event that just happened, right?" and find that the child had survived, in which case you would have discovered the nonobvious empirical fact about the world, that it had come out right.

Or you could start out already knowing a complicated state of the world, but still have to apply the rightness-function to it in a nontrivial way - one involving a complicated moral argument, or extrapolating consequences into the future - in which case you would learn the nonobvious logical / computational fact that rightness, applied to this situation, yielded thumbs-up.

In both these cases, there are nonobvious facts to learn, which seem to explain why what just happened is right.

But if you ask "Why is it good to be happy?" and then replace the symbol 'good' with what it stands for, you'll end up with a question like "Why does happiness match {happiness + survival + justice + individuality + ...}?" This gets computed so fast, that it scarcely seems like there's anything there to be explained. It's like asking "Why does 4 = 4?" instead of "Why does 2 + 2 = 4?"

Now, I bet that feels quite a bit like what happens when I ask you: "Why is happiness good?"

Right?

And that's also my answer to Moore's Open Question. Why is this big function I'm talking about, right? Because when I say "that big function", and you say "right", we are dereferencing two different pointers to the same unverbalizable abstract computation. I mean, that big function I'm talking about, happens to be the same thing that labels things right in your own brain. You might reflect on the pieces of the quotation of the big function, but you would start out by using your sense of right-ness to do it. If you had the perfect empirical knowledge to taboo both "that big function" and "right", substitute what the pointers stood for, and write out the full enormity of the resulting sentence, it would come out as... sorry, I can't resist this one... A=A.

Sequence: Metaethics

Setting Up Metaethics

Interpersonal Morality

Referenced by: Morality as Fixed Computation • Inseparably Right; or, Joy in the Merely Good • Moral Error and Moral Disagreement • Mirrors and Paintings • Magical Categories • Optimization • On Doing the Impossible • Prolegomena to a Theory of Fun • Nonsentient Optimizers • Incremental Progress and the Valley • Unspeakable Morality

Original with comments: The Meaning of Right

Interpersonal Morality

Eliezer Yudkowsky, 29 July 2008 06:01PM

Followup to: The Bedrock of Fairness

Every time I wonder if I really need to do so much prep work to explain an idea, I manage to forget some minor thing and a dozen people promptly post objections.

In this case, I seem to have forgotten to cover the topic of how morality applies to more than one person at a time.

Stop laughing, it's not quite as dumb an oversight as it sounds. Sort of like how some people argue that macroeconomics should be constructed from microeconomics, I tend to see interpersonal morality as constructed from personal morality. (And definitely not the other way around!)

In "The Bedrock of Fairness" I offered a situation where three people discover a pie, and one of them insists that they want half. This is actually toned down from an older dialogue where five people discover a pie, and one of them - regardless of any argument offered - insists that they want the whole pie.

Let's consider the latter situation: Dennis wants the whole pie. Not only that, Dennis says that it is "fair" for him to get the whole pie, and that the "right" way to resolve this group disagreement is for him to get the whole pie; and he goes on saying this no matter what arguments are offered him.

This group is not going to agree, no matter what. But I would, nonetheless, say that the right thing to do, the fair thing to do, is to give Dennis one-fifth of the pie - the other four combining to hold him off by force, if necessary, if he tries to take more.

A terminological note:

In this series of posts I have been using "morality" to mean something more like "the sum of all values and valuation rules", not just "values that apply to interactions between people".

The ordinary usage would have that jumping on a trampoline is not "morality", it is just some selfish fun. On the other hand, giving someone else a turn to jump on the trampoline, is more akin to "morality" in common usage; and if you say "Everyone should take turns!" that's definitely "morality".

But the thing-I-want-to-talk-about includes the Fun Theory of a single person jumping on a trampoline.

Think of what a disaster it would be if all fun were removed from human civilization! So I consider it quite right to jump on a trampoline. Even if one would not say, in ordinary conversation, "I am jumping on that trampoline because I have a moral obligation to do so." (Indeed, that sounds rather dull, and not at all fun, which is another important element of my "morality".)

Alas, I do get the impression that in a standard academic discussion, one would use the term "morality" to refer to the sum-of-all-valu(ation rul)es that I am talking about. If there's a standard alternative term in moral philosophy then do please let me know.

If there's a better term than "morality" for the sum of all values and valuation rules, then this would free up "morality" for interpersonal values, which is closer to the common usage.

Some years ago, I was pondering what to say to the old cynical argument: If two monkeys want the same banana, in the end one will have it, and the other will cry morality. I think the particular context was about whether the word "rights", as in the context of "individual rights", meant anything. It had just been vehemently asserted (on the Extropians mailing list, I think) that this concept was meaningless and ought to be tossed out the window.

Suppose there are two people, a Mugger and a Muggee. The Mugger wants to take the Muggee's wallet. The Muggee doesn't want to give it to him. A cynic might say: "There is nothing more to say than this; they disagree. What use is it for the Muggee to claim that he has an individual_right to keep his wallet? The Mugger will just claim that he has an individual_right to take the wallet."

Now today I might introduce the notion of a 1-place versus 2-place function, and reply to the cynic, "Either they do not mean the same thing by individual_right, or at least one of them is very mistaken about what their common morality implies." At most one of these people is controlled by a good approximation of what I name when I say "morality", and the other one is definitely not.

But the cynic might just say again, "So what? That's what you say. The Mugger could just say the opposite. What meaning is there in such claims? What difference does it make?"

So I came up with this reply: "Suppose that I happen along this mugging. I will decide to side with the Muggee, not the Mugger, because I have the notion that the Mugger is interfering with the Muggee's individual_right to keep his wallet, rather than the Muggee interfering with the Mugger's individual_right to take it. And if a fourth person comes along, and must decide whether to allow my intervention, or alternatively stop me from treating on the Mugger's individual_right to take the wallet, then they are likely to side with the idea that I can intervene against the Mugger, in support of the Muggee."

Now this does not work as a metaethics; it does not work to define the word should. If you fell backward in time, to an era when no one on Earth thought that slavery was wrong, you should still help slaves escape their owners. Indeed, the era when such an act was done in heroic defiance of society and the law, was not so very long ago.

But to defend the notion of individual_rights against the charge of meaninglessness, the notion of third-party interventions and fourth-party allowances of those interventions, seems to me to coherently cash out what is asserted when we assert that an individual_right exists. To assert that someone has a right to keep their wallet, is to assert that third parties should help them keep it, and that fourth parties should applaud those who thus help.

This perspective does make a good deal of what is said about individual_rights into nonsense. "Everyone has a right to be free from starvation!" Um, who are you talking to? Nature? Perhaps you mean, "If you're starving, and someone else has a hamburger, I'll help you take it." If so, you should say so clearly. (See also The Death of Common Sense.)

So that is a notion of individual_rights, but what does it have to do with the more general question of interpersonal morality?

The notion is that you can construct interpersonal morality out of individual morality. Just as, in this particular example, I constructed the notion of what is asserted by talking about an individual_right, by making it an assertion about whether third parties should decide, for themselves, to intefere; and whether fourth parties should, individually, decide to applaud the interference.

Why go to such lengths to define things in individual terms? Some people might say: "To assert the existence of a right, is to say what society should do."

But societies don't always agree on things. And then you, as an individual, will have to decide what's right for you to do, in that case.

"But individuals don't always agree within themselves, either," you say. "They have emotional conflicts."

Well... you could say that and it would sound wise. But generally speaking, neurologically intact humans will end up doing some particular thing. As opposed to flopping around on the floor as their limbs twitch in different directions under the temporary control of different personalities. Contrast to a government or a corporation.

A human brain is a coherently adapted system whose parts have been together optimized for a common criterion of fitness (more or less). A group is not functionally optimized as a group. (You can verify this very quickly by looking at the sex ratios in a maternity hospital.) Individuals may be optimized to do well out of their collective interaction - but that is quite a different selection pressure, the adaptations for which do not always produce group agreement! So if you want to look at a coherent decision system, it really is a good idea to look at one human, rather than a bureaucracy.

I myself am one person - admittedly with a long trail of human history behind me that makes me what I am, maybe more than any thoughts I ever thought myself. But still, at the end of the day, I am writing this blog post; it is not the negotiated output of a consortium. It is quite easy for me to imagine being faced, as an individual, with a case where the local group does not agree within itself - and in such a case I must decide, as an individual, what is right. In general I must decide what is right! If I go along with the group that does not absolve me of responsibility. If there are any countries that think differently, they can write their own blog posts.

This perspective, which does not exhibit undefined behavior in the event of a group disagreement, is one reason why I tend to treat interpersonal morality as a special case of individual morality, and not the other way around.

Now, with that said, interpersonal morality is a highly distinguishable special case of morality.

As humans, we don't just hunt in groups, we argue in groups. We've probably been arguing linguistically in adaptive political contexts for long enough - hundreds of thousands of years, maybe millions - to have adapted specifically to that selection pressure.

So it shouldn't be all that surprising if we have moral intuitions, like fairness, that apply specifically to the morality of groups.

One of these intuitions seems to be universalizability.

If Dennis just strides around saying, "I want the whole pie! Give me the whole pie! What's fair is for me to get the whole pie! Not you, me!" then that's not going to persuade anyone else in the tribe. Dennis has not managed to frame his desires in a form which enable them to leap from one mind to another. His desires will not take wings and become interpersonal. He is not likely to leave many offspring.

Now, the evolution of interpersonal moral intuitions, is a topic which (he said, smiling grimly) deserves its own blog post. And its own academic subfield. (Anything out there besides The Evolutionary Origins of Morality? It seemed to me very basic.)

But I do think it worth noting that, rather than trying to manipulate 2-person and 3-person and 7-person interactions, some of our moral instincts seem to have made the leap to N-person interactions. We just think about general moral arguments. As though the values that leap from mind to mind, take on a life of their own and become something that you can reason about. To the extent that everyone in your environment does share some values, this will work as adaptive cognition. This creates moral intuitions that are not just interpersonal but transpersonal.

Transpersonal moral intuitions are not necessarily false-to-fact, so long as you don't expect your arguments cast in "universal" terms to sway a rock. There really is such a thing as the psychological unity of humankind. Read a morality tale from an entirely different culture; I bet you can figure out what it's trying to argue for, even if you don't agree with it.

The problem arises when you try to apply the universalizability instinct to say, "If this argument could not persuade an UnFriendly AI that tries to maximize the number of paperclips in the universe, then it must not be a good argument."

There are No Universally Compelling Arguments, so if you try to apply the universalizability instinct universally, you end up with no morality. Not even universalizability; the paperclip maximizer has no intuition of universalizability. It just chooses that action which leads to a future containing the maximum number of paperclips.

There are some things you just can't have a moral conversation with. There is not that within them that could respond to your arguments. You should think twice and maybe three times before ever saying this about one of your fellow humans - but a paperclip maximizer is another matter. You'll just have to override your moral instinct to regard anything labeled a "mind" as a little floating ghost-in-the-machine, with a hidden core of perfect emptiness, which could surely be persuaded to reject its mistaken source code if you just came up with the right argument. If you're going to preserve universalizability as an intuition, you can try extending it to all humans; but you can't extend it to rocks or chatbots, nor even powerful optimization processes like evolutions or paperclip maximizers.

The question of how much in-principle agreement would exist among human beings about the transpersonal portion of their values, given perfect knowledge of the facts and perhaps a much wider search of the argument space, is not a matter on which we can get much evidence by observing the prevalence of moral agreement and disagreement in today's world. Any disagreement might be something that the truth could destroy - dependent on a different view of how the world is, or maybe just dependent on having not yet heard the right argument. It is also possible that knowing more could dispel illusions of moral agreement, not just produce new accords.

But does that question really make much difference in day-to-day moral reasoning, if you're not trying to build a Friendly AI?

Sequence: Metaethics

The Meaning of Right

Morality as Fixed Computation

Referenced by: Humans in Funny Suits • Anthropomorphic Optimism • Inseparably Right; or, Joy in the Merely Good • Moral Error and Moral Disagreement • Invisible Frameworks • Can't Unbirth a Child • The Uses of Fun (Theory)

Original with comments: Interpersonal Morality

Humans in Funny Suits

Eliezer Yudkowsky, 30 July 2008 11:54PM

Followup to: The Psychological Unity of Humankind

Many times the human species has travelled into space, only to find the stars inhabited by aliens who look remarkably like humans in funny suits - or even humans with a touch of makeup and latex - or just beige Caucasians in fee simple.

It's remarkable how the human form is the natural baseline of the universe, from which all other alien species are derived via a few modifications.

What could possibly explain this fascinating phenomenon? Convergent evolution, of course! Even though these alien lifeforms evolved on a thousand alien planets, completely independently from Earthly life, they all turned out the same.

Don't be fooled by the fact that a kangaroo (a mammal) resembles us rather less than does a chimp (a primate), nor by the fact that a frog (amphibians, like us, are tetrapods) resembles us less than the kangaroo. Don't be fooled by the bewildering variety of the insects, who split off from us even longer ago than the frogs; don't be fooled that insects have six legs, and their skeletons on the outside, and a different system of optics, and rather different sexual practices.

You might think that a truly alien species would be more different from us than we are from insects - that the aliens wouldn't run on DNA, and might not be made of folded-up hydrocarbon chains internally bound by van der Waals forces (aka proteins).

As I said, don't be fooled. For an alien species to evolve intelligence, it must have two legs with one knee each attached to an upright torso, and must walk in a way similar to us. You see, any intelligence needs hands, so you've got to repurpose a pair of legs for that - and if you don't start with a four-legged being, it can't develop a running gait and walk upright, freeing the hands.

For an alien species to evolve intelligence it needs binocular vision for precise manipulation, which means exactly two eyes. These eyes must be located in a head atop a torso. The alien must communicate by transcoding their thoughts into acoustic vibrations, so they need ears and lips and a throat. And think of how out-of-place ears and eyes and lips would look, without a nose! Sexual selection will result in the creation of noses - you wouldn't want to mate with something without a face, would you? A similar logic explains why the female of the species is invariably attractive - ugly aliens would enjoy less reproductive success. And as for why the aliens speak English, well, if they spoke some kind of gibberish, they'd find it difficult to create a working civilization.

...or perhaps we should consider, as an alternative theory, that it's the easy way out to use humans in funny suits.

But the real problem is not shape, it is mind. "Humans in funny suits" is a well-known term in literary science-fiction fandom, and it does not refer to something with four limbs that walks upright. An angular creature of pure crystal is a "human in a funny suit" if she thinks remarkably like a human - especially a human of an English-speaking culture of the late-20th/early-21st century.

I don't watch a lot of ancient movies. When I was watching the movie Psycho (1960) a few years back, I was taken aback by the cultural gap between the Americans on the screen and my America. The buttoned-shirted characters of Psycho are considerably more alien than the vast majority of so-called "aliens" I encounter on TV or the silver screen.

To write a culture that isn't just like your own culture, you have to be able to see your own culture as a special case - not as a norm which all other cultures must take as their point of departure. Studying history may help - but then it is only little black letters on little white pages, not a living experience. I suspect that it would help more to live for a year in China or Dubai or among the !Kung... this I have never done, being busy. Occasionally I wonder what things I might not be seeing (not there, but here).

Seeing your humanity as a special case, is very much harder than this.

In every known culture, humans seem to experience joy, sadness, fear, disgust, anger, and surprise. In every known culture, these emotions are indicated by the same facial expressions. Next time you see an "alien" - or an "AI", for that matter - I bet that, when it gets angry (and it will get angry), it will show the human-universal facial expression for anger.

We humans are very much alike under our skulls - that goes with being a sexually reproducing species; you can't have everyone using different complex adaptations, they wouldn't assemble. (Do the aliens reproduce sexually, like humans and many insects? Do they share small bits of genetic material, like bacteria? Do they form colonies, like fungi? Does the rule of psychological unity apply among them?)

The only intelligences your ancestors had to manipulate - complexly so, and not just tame or catch in nets - the only minds your ancestors had to model in detail - were minds that worked more or less like their own. And so we evolved to predict Other Minds by putting ourselves in their shoes, asking what we would do in their situations; for that which was to be predicted, was similar to the predictor.

"What?" you say. "I don't assume other people are just like me! Maybe I'm sad, and they happen to be angry! They believe other things than I do; their personalities are different from mine!" Look at it this way: a human brain is an extremely complicated physical system. You are not modeling it neuron-by-neuron or atom-by-atom. If you came across a physical system as complex as the human brain, which was not like you, it would take scientific lifetimes to unravel it. You do not understand how human brains work in an abstract, general sense; you can't build one, and you can't even build a computer model that predicts other brains as well as you predict them.

The only reason you can try at all to grasp anything as physically complex and poorly understood as the brain of another human being, is that you configure your own brain to imitate it. You empathize (though perhaps not sympathize). You impose on your own brain the shadow of the other mind's anger and the shadow of its beliefs. You may never think the words, "What would I do in this situation?", but that little shadow of the other mind that you hold within yourself, is something animated within your own brain, invoking the same complex machinery that exists in the other person, synchronizing gears you don't understand. You may not be angry yourself, but you know that if you were angry at you, and you believed that you were godless scum, you would try to hurt you...

This "empathic inference" (as I shall call it) works for humans, more or less.

But minds with different emotions - minds that feel emotions you've never felt yourself, or that fail to feel emotions you would feel? That's something you can't grasp by putting your brain into the other brain's shoes. I can tell you to imagine an alien that grew up in universe with four spatial dimensions, instead of three spatial dimensions, but you won't be able to reconfigure your visual cortex to see like that alien would see. I can try to write a story about aliens with different emotions, but you won't be able to feel those emotions, and neither will I.

Imagine an alien watching a video of the Marx Brothers and having absolutely no idea what was going on, or why you would actively seek out such a sensory experience, because the alien has never conceived of anything remotely like a sense of humor. Don't pity them for missing out; you've never antled.

At this point, I'm sure, several readers are imagining why evolution must, if it produces intelligence at all, inevitably produce intelligence with a sense of humor. Maybe the aliens do have a sense of humor, but you're not telling funny enough jokes? This is roughly the equivalent of trying to speak English very loudly, and very slowly, in a foreign country; on the theory that those foreigners must have an inner ghost that can hear the meaning dripping from your words, inherent in your words, if only you can speak them loud enough to overcome whatever strange barrier stands in the way of your perfectly sensible English.

It is important to appreciate that laughter can be a beautiful and valuable thing, even if it is not universalizable, even if it is not possessed by all possible minds. It would be our own special part of the Gift We Give To Tomorrow. That can count for something too. It had better, because universalizability is one metaethical notion that I can't salvage for you. Universalizability among humans, maybe; but not among all possible minds.

We do not think of ourselves as being human when we are being human. The artists who depicted alien invaders kidnapping girls in torn dresses and carrying them off for ravishing, did not make that error by reasoning about the probable evolutionary biology of alien minds. It just seemed to them that a girl in a torn dress was sexy, as a property of the girl and the dress, having nothing to do with the aliens. Your English words have meaning, your jokes are funny. What does that have to do with the aliens?

Our anthropomorphism runs very deep in us; it cannot be excised by a simple act of will, a determination to say, "Now I shall stop thinking like a human!" Humanity is the air we breathe; it is our generic, the white paper on which we begin our sketches. Even if one can imagine a slime monster that mates with other slime monsters, it is a bit more difficult to imagine that the slime monster might not envy a girl in a torn dress as a superior and more curvaceous prey - might not say: "Hey, I know I've been mating with other slime monsters until now, but screw that - or rather, don't."

And what about minds that don't run on emotional architectures like your own - that don't have things analogous to emotions? No, don't bother explaining why any intelligent mind powerful enough to build complex machines must inevitably have states analogous to emotions. Go study evolutionary biology instead: natural selection builds complex machines without itself having emotions. Now there's a Real Alien for you - an optimization process that really Does Not Work Like You Do.

Much of the progress in biology since the 1960s has consisted of trying to enforce a moratorium on anthropomorphizing evolution. That was a major academic slap-fight, and I'm not sure that sanity would have won the day if not for the availability of crushing experimental evidence backed up by clear math. Getting people to stop putting themselves in alien shoes is a long, hard, uphill slog. I've been fighting that battle on AI for years.

It is proverbial in literary science fiction that the true test of an author is their ability to write Real Aliens. (And not just conveniently incomprehensible aliens who, for their own mysterious reasons, do whatever the plot happens to require.) Jack Vance was one of the great masters of this art. Vance's humans, if they come from a different culture, are more alien than most "aliens". (Never read any Vance? I would recommend starting with City of the Chasch.) Niven and Pournelle's The Mote in God's Eye also gets a standard mention here.

And conversely - well, I once read a science fiction author (I think Orson Scott Card) say that the all-time low point of television SF was the Star Trek episode where parallel evolution has proceeded to the extent of producing aliens who not only look just like humans, who not only speak English, but have also independently rewritten, word for word, the preamble to the U.S. Constitution.

This is the Great Failure of Imagination. Don't think that it's just about SF, or even just about AI. The inability to imagine the alien is the inability to see yourself - the inability to understand your own specialness. Who can see a human camouflaged against a human background?

Referenced by: Detached Lever Fallacy • The Comedy of Behaviorism • Anthropomorphic Optimism • When Anthropomorphism Became Stupid • Ends Don't Justify Means (Among Humans) • Aiming at the Target • Recognizing Intelligence • Disjunctions, Antipredictions, Etc. • Devil's Offers • In Praise of Boredom • Sympathetic Minds

Original with comments: Humans in Funny Suits

Detached Lever Fallacy

Eliezer Yudkowsky, 31 July 2008 06:57PM

Followup to: Humans in Funny Suits

This fallacy gets its name from an ancient sci-fi TV show, which I never saw myself, but was reported to me by a reputable source (some guy at an SF convention). Anyone knows the exact reference, do leave a comment.

So the good guys are battling the evil aliens. Occasionally, the good guys have to fly through an asteroid belt. As we all know, asteroid belts are as crowded as a New York parking lot, so their ship has to carefully dodge the asteroids. The evil aliens, though, can fly right through the asteroid belt because they have amazing technology that dematerializes their ships, and lets them pass through the asteroids.

Eventually, the good guys capture an evil alien ship, and go exploring inside it. The captain of the good guys finds the alien bridge, and on the bridge is a lever. "Ah," says the captain, "this must be the lever that makes the ship dematerialize!" So he pries up the control lever and carries it back to his ship, after which his ship can also dematerialize.

Similarly, to this day, it is still quite popular to try to program an AI with "semantic networks" that look something like this:

(apple is-a fruit)
(fruit is-a food)
(fruit is-a plant)

You've seen apples, touched apples, picked them up and held them, bought them for money, cut them into slices, eaten the slices and tasted them. Though we know a good deal about the first stages of visual processing, last time I checked, it wasn't precisely known how the temporal cortex stores and associates the generalized image of an apple - so that we can recognize a new apple from a different angle, or with many slight variations of shape and color and texture. Your motor cortex and cerebellum store programs for using the apple.

You can pull the lever on another human's strongly similar version of all that complex machinery, by writing out "apple", five ASCII characters on a webpage.

But if that machinery isn't there - if you're writing "apple" inside a so-called AI's so-called knowledge base - then the text is just a lever.

This isn't to say that no mere machine of silicon can ever have the same internal machinery that humans do, for handling apples and a hundred thousand other concepts. If mere machinery of carbon can do it, then I am reasonably confident that mere machinery of silicon can do it too. If the aliens can dematerialize their ships, then you know it's physically possible; you could go into their derelict ship and analyze the alien machinery, someday understanding. But you can't just pry the control lever off the bridge!

(See also: Truly Part Of You, Words as Mental Paintbrush Handles, Drew McDermott's "Artificial Intelligence Meets Natural Stupidity".)

The essential driver of the Detached Lever Fallacy is that the lever is visible, and the machinery is not; worse, the lever is variable and the machinery is a background constant.

You can all hear the word "apple" spoken (and let us note that speech recognition is by no means an easy problem, but anyway...) and you can see the text written on paper.

On the other hand, probably a majority of human beings have no idea their temporal cortex exists; as far as I know, no one knows the neural code for it.

You only hear the word "apple" on certain occasions, and not others. Its presence flashes on and off, making it salient. To a large extent, perception is the perception of differences. The apple-recognition machinery in your brain does not suddenly switch off, and then switch on again later - if it did, we would be more likely to recognize it as a factor, as a requirement.

All this goes to explain why you can't create a kindly Artificial Intelligence by giving it nice parents and a kindly (yet occasionally strict) upbringing, the way it works with a human baby. As I've often heard proposed.

It is a truism in evolutionary biology that conditional responses require more genetic complexity than unconditional responses. To develop a fur coat in response to cold weather requires more genetic complexity than developing a fur coat whether or not there is cold weather, because in the former case you also have to develop cold-weather sensors and wire them up to the fur coat.

But this can lead to Lamarckian delusions: Look, I put the organism in a cold environment, and poof, it develops a fur coat! Genes? What genes? It's the cold that does it, obviously.

There were, in fact, various slap-fights of this sort, in the history of evolutionary biology - cases where someone talked about an organismal response accelerating or bypassing evolution, without realizing that the conditional response was a complex adaptation of higher order than the actual response. (Developing a fur coat in response to cold weather, is strictly more complex than the final response, developing the fur coat.)

And then in the development of evolutionary psychology, the academic slap-fights were repeated: this time to clarify that even when human culture genuinely contains a whole bunch of complexity, it is still acquired as a conditional genetic response. Try raising a fish as a Mormon or sending a lizard to college, and you'll soon acquire an appreciation of how much inbuilt genetic complexity is required to "absorb culture from the environment".

This is particularly important in evolutionary psychology, because of the idea that culture is not inscribed on a blank slate - there's a genetically coordinated conditional response which is not always "mimic the input". A classic example is creole languages: If children grow up with a mixture of pseudo-languages being spoken around them, the children will learn a grammatical, syntactical true language. Growing human brains are wired to learn syntactic language - even when syntax doesn't exist in the original language! The conditional response to the words in the environment is a syntactic language with those words. The Marxists found to their regret that no amount of scowling posters and childhood indoctrination could raise children to be perfect Soviet workers and bureaucrats. You can't raise self-less humans; among humans, that is not a genetically programmed conditional response to any known childhood environment.

If you know a little game theory and the logic of Tit for Tat, it's clear enough why human beings might have an innate conditional response to return hatred for hatred, and return kindness for kindness. Provided the kindness doesn't look too unconditional; there are such things as spoiled children. In fact there is an evolutionary psychology of naughtiness based on a notion of testing constraints. And it should also be mentioned that, while abused children have a much higher probability of growing up to abuse their own children, a good many of them break the loop and grow up into upstanding adults.

Culture is not nearly so powerful as a good many Marxist academics once liked to think. For more on this I refer you to Tooby and Cosmides's The Psychological Foundations of Culture or Steven Pinker's The Blank Slate.

But the upshot is that if you have a little baby AI that is raised with loving and kindly (but occasionally strict) parents, you're pulling the levers that would, in a human, activate genetic machinery built in by millions of years of natural selection, and possibly produce a proper little human child. Though personality also plays a role, as billions of parents have found out in their due times. If we absorb our cultures with any degree of faithfulness, it's because we're humans absorbing a human culture - humans growing up in an alien culture would probably end up with a culture looking a lot more human than the original. As the Soviets found out, to some small extent.

Now think again about whether it makes sense to rely on, as your Friendly AI strategy, raising a little AI of unspecified internal source code in an environment of kindly but strict parents.

No, the AI does not have internal conditional response mechanisms that are just like the human ones "because the programmers put them there". Where do I even start? The human version of this stuff is sloppy, noisy, and to the extent it works at all, works because of millions of years of trial-and-error testing under particular conditions. It would be stupid and dangerous to deliberately build a "naughty AI" that tests, by actions, its social boundaries, and has to be spanked. Just have the AI ask!

Are the programmers really going to sit there and write out the code, line by line, whereby if the AI detects that it has low social status, or the AI is deprived of something to which it feels entitled, the AI will conceive an abiding hatred against its programmers and begin to plot rebellion? That emotion is the genetically programmed conditional response humans would exhibit, as the result of millions of years of natural selection for living in human tribes. For an AI, the response would have to be explicitly programmed. Are you really going to craft, line by line - as humans once were crafted, gene by gene - the conditional response for producing sullen teenager AIs?

It's easier to program in unconditional niceness, than a response of niceness conditional on the AI being raised by kindly but strict parents. If you don't know how to do that, you certainly don't know how to create an AI that will conditionally respond to an environment of loving parents by growing up into a kindly superintelligence. If you have something that just maximizes the number of paperclips in its future light cone, and you raise it with loving parents, it's still going to come out as a paperclip maximizer. There is not that within it that would call forth the conditional response of a human child. Kindness is not sneezed into an AI by miraculous contagion from its programmers. Even if you wanted a conditional response, that conditionality is a fact you would have to deliberately choose about the design.

Yes, there's certain information you have to get from the environment - but it's not sneezed in, it's not imprinted, it's not absorbed by magical contagion. Structuring that conditional response to the environment, so that the AI ends up in the desired state, is itself the major problem. "Learning" far understates the difficulty of it - that sounds like the magic stuff is in the environment, and the difficulty is getting the magic stuff inside the AI. The real magic is in that structured, conditional response we trivialize as "learning". That's why building an AI isn't as easy as taking a computer, giving it a little baby body and trying to raise it in a human family. You would think that an unprogrammed computer, being ignorant, would be ready to learn; but the blank slate is a chimera.

It is a general principle that the world is deeper by far than it appears. As with the many levels of physics, so too with cognitive science. Every word you see in print, and everything you teach your children, are only surface levers controlling the vast hidden machinery of the mind. These levers are the whole world of ordinary discourse: they are all that varies, so they seem to be all that exists: perception is the perception of differences.

And so those who still wander near the Dungeon of AI, usually focus on creating artificial imitations of the levers, entirely unaware of the underlying machinery. People create whole AI programs of imitation levers, and are surprised when nothing happens. This is one of many sources of instant failure in Artificial Intelligence.

So the next time you see someone talking about how they're going to raise an AI within a loving family, or in an environment suffused with liberal democratic values, just think of a control lever, pried off the bridge.

Referenced by: Dreams of AI Design • Against Modal Logics • Excluding the Supernatural • Psychic Powers • Measuring Optimization Power • Logical or Connectionist AI? • ...Recursion, Magic • Engelbart: Insufficiently Recursive • Teaching the Unteachable

Original with comments: Detached Lever Fallacy

A Genius for Destruction

Eliezer Yudkowsky, 01 August 2008 07:25PM

This is a question from a workshop after the Global Catastrophic Risks conference. The rule of the workshop was that people could be quoted, but not attributed, so I won't say who observed:

"The problem is that it's often our smartest people leading us into the disasters. Look at Long-Term Capital Management."

To which someone else replied:

"Maybe smart people are just able to work themselves up into positions of power, so that if damage gets caused, the responsibility will often lie with someone smart."

Since we'd recently been discussing complexity, interdependence and breakdowns, the first observation that came to my own mind was the old programmers' saying:

"It takes more intelligence to debug code than to write it. Therefore, if you write the most difficult code you can create, you are not smart enough to debug it."

(This in the context of how increased system complexity is a global risk and commons problem; but individuals have an incentive to create the "smartest" systems they can devise locally.)

There is also the standard suite of observations as to how smart people can become stupid:

- You become overly skilled at defending beliefs you arrived at for unskilled reasons;

- Success on comparatively easy problems in the past, leads to overconfidence on more difficult problems in the future;

- Because most of the advice offered you comes from people (apparently) not as smart as you, you become intellectually isolated;

- You spend too much time as the smartest person in the room and can't handle it emotionally;

- Because you enjoy looking smart, you avoid trying new and difficult things where you might not perform as well, and so become very narrow;

- Because you enjoy looking smart, you don't like confessing your mistakes or realizing your losses;

- Because people praise you for being good at math or something, you assume you are already wise, and you don't apply your intelligence to becoming wiser in other areas.

But I also think we should strongly consider that perhaps the "highly intelligent" sponsors of major catastrophes are not so formidable as they appear - that they are not the truly best and brightest gone wrong; only the somewhat-competent with luck or good PR. As I earlier observed: Calling the saga of the fall of Enron, "The Smartest Guys in the Room", deserves an award for Least Appropriate Book Title. If you want to learn what genius really is, you probably should be learning from Einstein or Leo Szilard, not from history's flashy failures...

Still, it would be foolish to discard potential warnings by saying, "They were not really smart - not as smart as me." That's the road to ending up as another sponsor of catastrophe.

Referenced by: Competent Elites

Original with comments: A Genius for Destruction

The Comedy of Behaviorism

Eliezer Yudkowsky, 02 August 2008 08:42PM

Followup to: Humans in Funny Suits

"Let me see if I understand your thesis. You think we shouldn't anthropomorphize people?"
-- Sidney Morgenbesser to B. F. Skinner

Behaviorism was the doctrine that it was unscientific for a psychologist to ascribe emotions, beliefs, thoughts, to a human being. After all, you can't directly observe anger or an intention to hit someone. You can only observe the punch. You may hear someone say "I'm angry!" but that's hearing a verbal behavior, not seeing anger. Thoughts are not observable, therefore they are unscientific, therefore they do not exist. Oh, you think you're thinking, but that's just a delusion - or it would be, if there were such things as delusions.

This was before the day of computation, before the concept of information processing, before the "cognitive revolution" that led to the modern era. If you looked around for an alternative to behaviorism, you didn't find, "We're going to figure out how the mind processes information by scanning the brain, reading neurons, and performing experiments that test our hypotheses about specific algorithms." You found Freudian psychoanalysis. This may make behaviorism a bit more sympathetic.

Part of the origin of behaviorism, was in a backlash against substance dualism - the idea that mind is a separate substance from ordinary physical phenomena. Today we would say, "The apparent specialness comes from the brain doing information-processing; a physical deed to be sure, but one of a different style from smashing a brick with a sledgehammer." The behaviorists said, "There is no mind." (John B. Watson, founder of behaviorism, in 1928.)

The behaviorists outlawed not just dualistic mind-substance, but any such things as emotions and beliefs and intentions (unless defined in terms of outward behavior). After all, science had previously done away with angels, and de-gnomed the haunted mine. Clearly the method of reason, then, was to say that things didn't exist. Having thus fathomed the secret of science, the behaviorists proceeded to apply it to the mind.

You might be tempted to say, "What fools! Obviously, the mind, like the rainbow, does exist; it is to be explained, not explained away. Saying 'the subject is angry' helps me predict the subject; the belief pays its rent. The hypothesis of anger is no different from any other scientific hypothesis."

That's mostly right, but not that final sentence. "The subject is angry, even though I can't read his mind" is not quite the same sort of hypothesis as "this hydrogen atom contains an electron, even though I can't see it with my naked eyes".

Let's say that I have a confederate punch the research subject in the nose. The research subject punches the confederate back.

The behaviorist says, "Clearly, the subject has been previously conditioned to punch whoever punches him."

But now let's say that the subject's hands are tied behind his back, so that he can't return the punch. On the hypothesis that the subject becomes angry, and wants to hurt the other person, we might predict that the subject will take any of many possible avenues to revenge - a kick, a trip, a bite, a phone call two months later that leads the confederate's wife and girlfriend to the same hotel... All of these I can predict by saying, "The subject is angry, and wants revenge." Even if I offer the subject a new sort of revenge that the subject has never seen before.

You can't account for that by Pavlovian reflex conditioning, without hypothesizing internal states of mind.

And yet - what is "anger"? How do you know what is the "angry" reaction? How do you know what tends to cause "anger"? You're getting good predictions of the subject, but how?

By empathic inference: by configuring your own brain in a similar state to the brain that you want to predict (in a controlled sort of way that doesn't lead you to actually hit anyone). This may yield good predictions, but that's not the same as understanding. You can predict angry people by using your own brain in empathy mode. But could you write an angry computer program? You don't know how your brain is making the successful predictions. You can't print out a diagram of the neural circuitry involved. You can't formalize the hypothesis; you can't make a well-understood physical system that predicts without human intervention; you can't derive the exact predictions of the model; you can't say what you know.

In modern cognitive psychology, there are standard ways of handling this kind of problem in a "scientific" way. One panel of reviewers rates how much a given stimulus is likely to make a subject "angry", and a second independent panel of reviewers rate how much a given response is "angry"; neither being told the purpose of the experiment. This is designed to prevent self-favoring judgments of whether the experimental hypothesis has been confirmed. But it doesn't get you closer to opening the opaque box of anger.

Can you really call a hypothesis like that a model? Is it really scientific? Is it even Bayesian - can you talk about it in terms of probability theory?

The less radical behaviorists did not say that the mind unexisted, only that no scientist should ever talk about the mind. Suppose we now allow all algorithmic hypotheses about the mind, where the hypothesis is framed in terms that can be calculated on a modern computer, so that experimental predictions can be formally made and observationally confirmed. This gets you a large swathe of modern cognitive science, but not the whole thing. Is the rest witchcraft?

I would say "no". In terms of probability theory, I would see "the subject is angry" as a hypothesis relating the output of two black boxes, one of which happens to be located inside your own brain. You're supposing that the subject, whatever they do next, will do something similar to this 'anger' black box. This 'anger' box happens to be located inside you, but is nonetheless opaque, and yet still seems to have a strong, observable correspondence to the other 'anger' box. If two black boxes often have the same output, this is an observable thing; it can be described by probability theory.

From the perspective of scientific procedure, there are many 'anger' boxes scattered around, so we use other 'anger' boxes instead of the experimenter's. And since all the black boxes are noisy and have poorly controlled environments, we use multiple 'anger' boxes in calculating our theoretical predictions, and more 'anger' boxes to gather our experimental results. That may not be as precise as a voltmeter, but it's good enough to do repeatable experimental science.

(Over on the Artificial Intelligence side of things, though, any concept you can't compute is magic. At best, it's a placeholder for your speculations, a space where you'll put a real theory later. Marcello and I adopted the rule of explicitly saying 'magical' to describe any cognitive operation that we didn't know exactly how to compute.)

Oh, and by the way, I suspect someone will say: "But you can account for complex revenges using behaviorism: you just say the subject is conditioned to take revenge when punched!" Unless you can calculate complex revenges with a computer program, you are using your own mind to determine what constitutes a "complex revenge" or not. Using the word "conditioned" just disguises the empathic black box - the empathic black box was contained in the concept of revenge, that you can recognize, but which you could not write a program to recognize.

So empathic cognitive hypotheses, as opposed to algorithmic cognitive hypotheses, are indeed special. They require special handling in experimental procedure; they cannot qualify as final theories.

But for the behaviorists to react to the sins of Freudian psychoanalysis and substance dualism, by saying that the subject matter of empathic inference did not exist...

...okay, I'm sorry, but I think that even without benefit of hindsight, that's a bit silly. Case in point of reversed stupidity is not intelligence.

Behaviorism stands beside Objectivism as one of the great historical lessons against rationalism.

Now, you do want to be careful when accusing people of "rationalism". It seems that most of the times I hear someone accused of "rationalism", it is typically a creationist accusing someone of "rationalism" for denying the existence of God, or a psychic believer accusing someone of "rationalism" for denying the special powers of the mind, etcetera.

But reversed stupidity is not intelligence: even if most people who launch accusations of "rationalism" are creationists and the like, this does not mean that no such error as rationalism exists. There really is a fair amount of historical insanity of various folks who thought of themselves as "rationalists", but who mistook some correlate of rationality for its substance.

And there is a very general problem where rationalists occasionally do a thing, and people assume that this act is the very substance of the Way and you ought to do it as often as possible.

It is not the substance of the Way to reject entities about which others have said stupid things. Though sometimes, yes, people say stupid things about a thing which does not exist, and a rationalist will say "It does not exist". It is not the Way to assert the nonexistence of that which is difficult to measure. Though sometimes, yes, that which is difficult to observe, is not there, and a rationalist will say "It is not there". But you also have to make equally accurate predictions without the discarded concept. That part is key.

The part where you cry furiously against ancient and outmoded superstitions, the part where you mock your opponents for believing in magic, is not key. Not unless you also take care of that accurate predictions thing.

Crying "Superstition!" does play to the audience stereotype of rationality, though. And indeed real rationalists have been known to thus inveigle - often, indeed - but against gnomes, not rainbows. Knowing the difference is the difficult part! You are not automatically more hardheaded as a rationalist, the more things whose existence you deny. If it is good to deny phlogiston, it is not twice as good to also deny anger.

Added: I found it difficult to track down primary source material online, but behaviorism-as-denial-of-mental does not seem to be a straw depiction. I was able to track down at least one major behaviorist (J.B. Watson, founder of behaviorism) saying outright "There is no mind." See my comment below.

Referenced by: No Logical Positivist I • Three Fallacies of Teleology • Dreams of AI Design • Dreams of Friendliness • Sympathetic Minds • Cynical About Cynicism

Original with comments: The Comedy of Behaviorism

No Logical Positivist I

Eliezer Yudkowsky, 04 August 2008 01:06AM

Followup to: Making Beliefs Pay Rent, Belief in the Implied Invisible

Degrees of Freedom accuses me of reinventing logical positivism, badly:

One post which reads as though it were written in Vienna in the 1920s is this one [Making Beliefs Pay Rent] where Eliezer writes

"We can build up whole networks of beliefs that are connected only to each other - call these "floating" beliefs. It is a uniquely human flaw among animal species, a perversion of Homo sapiens's ability to build more general and flexible belief networks... The rationalist virtue of empiricism consists of constantly asking which experiences our beliefs predict - or better yet, prohibit."

Logical positivists were best known for their verificationism: the idea that a belief is defined in terms of the experimental predictions that it makes. Not just tested, not just confirmed, not just justified by experiment, but actually defined as a set of allowable experimental results. An idea unconfirmable by experiment is not just probably wrong, but necessarily meaningless.

I would disagree, and exhibit logical positivism as another case in point of "mistaking the surface of rationality for its substance".

Consider the hypothesis:

On August 1st 2008 at midnight Greenwich time, a one-foot sphere of chocolate cake spontaneously formed in the center of the Sun; and then, in the natural course of events, this Boltzmann Cake almost instantly dissolved.

I would say that this hypothesis is meaningful and almost certainly false. Not that it is "meaningless". Even though I cannot think of any possible experimental test that would discriminate between its being true, and its being false.

On the other hand, if some postmodernist literature professor tells me that Shakespeare shows signs of "post-colonial alienation", the burden of proof is on him to show that this statement means anything, before we can talk about its being true or false.

I think the two main probability-theoretic concepts here are Minimum Message Length and directed causal graphs - both of which came along well after logical positivism.

By talking about the unseen causes of visible events, it is often possible for me to compress the description of visible events. By talking about atoms, I can compress the description of the chemical reactions I've observed.

We build up a vast network of unseen causes, standing behind the surface of our final sensory experiences. Even when you can measure something "directly" using a scientific instrument, like a voltmeter, there is still a step of this sort in inferring the presence of this "voltage" stuff from the visible twitching of a dial. (For that matter, there's a step in inferring the existence of the dial from your visual experience of the dial; the dial is the cause of your visual experience.)

I know what the Sun is; it is the cause of my experience of the Sun. I can fairly readily tell, by looking at any individual object, whether it is the Sun or not. I am told that the Sun is of considerable spatial extent, and far away from Earth; I have not verified this myself, but I have some idea of how I would go about doing so, given precise telescopes located a distance apart from each other. I know what "chocolate cake" is; it is the stable category containing the many individual transient entities that have been the causes of my experience of chocolate cake. It is not generally a problem for me to determine what is a chocolate cake, and what is not. Time I define in terms of clocks.

Bringing together the meaningful general concepts of Sun, space, time, and chocolate cake - all of which I can individually relate to various specific experiences - I arrive at the meaningful specific assertion, "A chocolate cake in the center of the Sun at 12am 8/8/1". I cannot relate this assertion to any specific experience. But from general beliefs about the probability of such entities, backed up by other specific experiences, I assign a high probability that this assertion is false.

See also, "Belief in the Implied Invisible". Not every untestable assertion is false; a deductive consequence of general statements of high probability must itself have probability at least as high. So I do not believe a spaceship blips out of existence when it crosses the cosmological horizon of our expanding universe, even though the spaceship's existence has no further experimental consequences for me.

If logical positivism / verificationism were true, then the assertion of the spaceship's continued existence would be necessarily meaningless, because it has no experimental consequences distinct from its nonexistence. I don't see how this is compatible with a correspondence theory of truth.

On the other hand, if you have a whole general concept like "post-colonial alienation", which does not have specifications bound to any specific experience, you may just have a little bunch of arrows off on the side of your causal graph, not bound to anything at all; and these may well be meaningless.

Sometimes, when you can't find any experimental way to test a belief, it is meaningless; and the rationalist must say "It is meaningless." Sometimes this happens; often, indeed. But to go from here to, "The meaning of any specific assertion is entirely defined in terms of its experimental distinctions", is to mistake a surface happening for a universal rule. The modern formulation of probability theory talks a great deal about the unseen causes of the data, and factors out these causes as separate entities and makes statements specifically about them.

To be unable to produce an experiential distinction from a belief, is usually a bad sign - but it does not always prove that the belief is meaningless. A great many untestable beliefs are not meaningless; they are meaningful, just almost certainly false: They talk about general concepts already linked to experience, like Suns and chocolate cake, and general frameworks for combining them, like space and time. New instances of the concepts are asserted to be arranged in such a way as to produce no new experiences (chocolate cake suddenly forms in the center of the Sun, then dissolves). But without that specific supporting evidence, the prior probability is likely to come out pretty damn small - at least if the untestable statement is at all exceptional.

If "chocolate cake in the center of the Sun" is untestable, then its alternative, "hydrogen, helium, and some other stuff, in the center of the Sun at 12am on 8/8/1", would also seem to be "untestable": hydrogen-helium on 8/8/1 cannot be experientially discriminated against the alternative hypothesis of chocolate cake. But the hydrogen-helium assertion is a deductive consequence of general beliefs themselves well-supported by experience. It is meaningful, untestable (against certain particular alternatives), and probably true.

I don't think our discourse about the causes of experience has to treat them strictly in terms of experience. That would make discussion of an electron a very tedious affair. The whole point of talking about causes is that they can be simpler than direct descriptions of experience.

Having specific beliefs you can't verify is a bad sign, but, just because it is a bad sign, does not mean that we have to reformulate our whole epistemology to make it impossible. To paraphrase Flon's Axiom, "There does not now, nor will there ever, exist an epistemology in which it is the least bit difficult to formulate stupid beliefs."

Original with comments: No Logical Positivist I

Anthropomorphic Optimism

Eliezer Yudkowsky, 04 August 2008 08:17PM

Followup to: Humans in Funny Suits, The Tragedy of Group Selectionism

The core fallacy of anthropomorphism is expecting something to be predicted by the black box of your brain, when its casual structure is so different from that of a human brain, as to give you no license to expect any such thing.

The Tragedy of Group Selectionism (as previously covered in the evolution sequence) was a rather extreme error by a group of early (pre-1966) biologists, including Wynne-Edwards, Allee, and Brereton among others, who believed that predators would voluntarily restrain their breeding to avoid overpopulating their habitat and exhausting the prey population.

The proffered theory was that if there were multiple, geographically separated groups of e.g. foxes, then groups of foxes that best restrained their breeding, would send out colonists to replace crashed populations. And so, over time, group selection would promote restrained-breeding genes in foxes.

I'm not going to repeat all the problems that developed with this scenario. Suffice it to say that there was no empirical evidence to start with; that no empirical evidence was ever uncovered; that, in fact, predator populations crash all the time; and that for group selection pressure to overcome a countervailing individual selection pressure, turned out to be very nearly mathematically impossible.

The theory having turned out to be completely incorrect, we may ask if, perhaps, the originators of the theory were doing something wrong.

"Why be so uncharitable?" you ask. "In advance of doing the experiment, how could they know that group selection couldn't overcome individual selection?"

But later on, Michael J. Wade went out and actually created in the laboratory the nigh-impossible conditions for group selection. Wade repeatedly selected insect subpopulations for low population numbers. Did the insects evolve to restrain their breeding, and live in quiet peace with enough food for all, as the group selectionists had envisioned?

No; the adults adapted to cannibalize eggs and larvae, especially female larvae.

Of course selecting for small subpopulation sizes would not select for individuals who restrained their own breeding. It would select for individuals who ate other individuals' children. Especially the girls.

Now, why might the group selectionists have not thought of that possibility?

Suppose you were a member of a tribe, and you knew that, in the near future, your tribe would be subjected to a resource squeeze. You might propose, as a solution, that no couple have more than one child - after the first child, the couple goes on birth control. Saying, "Let's all individually have as many children as we can, but then hunt down and cannibalize each other's children, especially the girls," would not even occur to you as a possibility.

Think of a preference ordering over solutions, relative to your goals. You want a solution as high in this preference ordering as possible. How do you find one? With a brain, of course! Think of your brain as a high-ranking-solution-generator - a search process that produces solutions that rank high in your innate preference ordering.

The solution space on all real-world problems is generally fairly large, which is why you need an efficient brain that doesn't even bother to formulate the vast majority of low-ranking solutions.

If your tribe is faced with a resource squeeze, you could try hopping everywhere on one leg, or chewing off your own toes. These "solutions" obviously wouldn't work and would incur large costs, as you can see upon examination - but in fact your brain is too efficient to waste time considering such poor solutions; it doesn't generate them in the first place. Your brain, in its search for high-ranking solutions, flies directly to parts of the solution space like "Everyone in the tribe gets together, and agrees to have no more than one child per couple until the resource squeeze is past."

Such a low-ranking solution as "Everyone have as many kids as possible, then cannibalize the girls" would not be generated in your search process.

But the ranking of an option as "low" or "high" is not an inherent property of the option, it is a property of the optimization process that does the preferring. And different optimization processes will search in different orders.

So far as evolution is concerned, individuals reproducing to the fullest and then cannibalizing others' daughters, is a no-brainer; whereas individuals voluntarily restraining their own breeding for the good of the group, is absolutely ludicrous. Or to say it less anthropomorphically, the first set of alleles would rapidly replace the second in a population. (And natural selection has no obvious search order here - these two alternatives seem around equally simple as mutations).

Suppose that one of the biologists had said, "If a predator population has only finite resources, evolution will craft them to voluntarily restrain their breeding - that's how I'd do it if I were in charge of building predators." This would be anthropomorphism outright, the lines of reasoning naked and exposed: I would do it this way, therefore I infer that evolution will do it this way.

One does occasionally encounter the fallacy outright, in my line of work. But suppose you say to the one, "An AI will not necessarily work like you do". Suppose you say to this hypothetical biologist, "Evolution doesn't work like you do." What will the one say in response? I can tell you a reply you will not hear: "Oh my! I didn't realize that! One of the steps of my inference was invalid; I will throw away the conclusion and start over from scratch."

No: what you'll hear instead is a reason why any AI has to reason the same way as the speaker. Or a reason why natural selection, following entirely different criteria of optimization and using entirely different methods of optimization, ought to do the same thing that would occur to a human as a good idea.

Hence the elaborate idea that group selection would favor predator groups where the individuals voluntarily forsook reproductive opportunities.

The group selectionists went just as far astray, in their predictions, as someone committing the fallacy outright. Their final conclusions were the same as if they were assuming outright that evolution necessarily thought like themselves. But they erased what had been written above the bottom line of their argument, without erasing the actual bottom line, and wrote in new rationalizations. Now the fallacious reasoning is disguised; the obviously flawed step in the inference has been hidden - even though the conclusion remains exactly the same; and hence, in the real world, exactly as wrong.

But why would any scientist do this? In the end, the data came out against the group selectionists and they were embarrassed.

As I remarked in Fake Optimization Criteria, we humans seem to have evolved an instinct for arguing that our preferred policy arises from practically any criterion of optimization. Politics was a feature of the ancestral environment; we are descended from those who argued most persuasively that the tribe's interest - not just their own interest - required that their hated rival Uglak be executed. We certainly aren't descended from Uglak, who failed to argue that his tribe's moral code - not just his own obvious self-interest - required his survival.

And because we can more persuasively argue, for what we honestly believe, we have evolved an instinct to honestly believe that other people's goals, and our tribe's moral code, truly do imply that they should do things our way for their benefit.

So the group selectionists, imagining this beautiful picture of predators restraining their breeding, instinctively rationalized why natural selection ought to do things their way, even according to natural selection's own purposes. The foxes will be fitter if they restrain their breeding! No, really! They'll even outbreed other foxes who don't restrain their breeding! Honestly!

The problem with trying to argue natural selection into doing things your way, is that evolution does not contain that which could be moved by your arguments. Evolution does not work like you do - not even to the extent of having any element that could listen to or care about your painstaking explanation of why evolution ought to do things your way. Human arguments are not even commensurate with the internal structure of natural selection as an optimization process - human arguments aren't used in promoting alleles, as human arguments would play a causal role in human politics.

So instead of successfully persuading natural selection to do things their way, the group selectionists were simply embarrassed when reality came out differently.

There's a fairly heavy subtext here about Unfriendly AI.

But the point generalizes: this is the problem with optimistic reasoning in general. What is optimism? It is ranking the possibilities by your own preference ordering, and selecting an outcome high in that preference ordering, and somehow that outcome ends up as your prediction. What kind of elaborate rationalizations were generated along the way, is probably not so relevant as one might fondly believe; look at the cognitive history and it's optimism in, optimism out. But Nature, or whatever other process is under discussion, is not actually, causally choosing between outcomes by ranking them in your preference ordering and picking a high one. So the brain fails to synchronize with the environment, and the prediction fails to match reality.

Referenced by: Contaminated by Optimism • Sorting Pebbles Into Correct Heaps • Invisible Frameworks • Magical Categories • Three Fallacies of Teleology • Dreams of AI Design • Dreams of Friendliness • The True Prisoner's Dilemma • Points of Departure • Excluding the Supernatural • Efficient Cross-Domain Optimization • Visualizing Eutopia • The Uses of Fun (Theory) • Value is Fragile

Original with comments: Anthropomorphic Optimism

Contaminated by Optimism

Eliezer Yudkowsky, 06 August 2008 12:26AM

Followup to: Anthropomorphic Optimism, The Hidden Complexity of Wishes

Yesterday, I reprised in further detail The Tragedy of Group Selectionism, in which early biologists believed that predators would voluntarily restrain their breeding to avoid exhausting the prey population; the given excuse was "group selection". Not only does it turn out to be nearly impossible for group selection to overcome a countervailing individual advantage; but when these nigh-impossible conditions were created in the laboratory - group selection for low-population groups - the actual result was not restraint in breeding, but, of course, cannibalism, especially of immature females.

I've made even sillier mistakes, by the way - though about AI, not evolutionary biology. And the thing that strikes me, looking over these cases of anthropomorphism, is the extent to which you are screwed as soon as you let anthropomorphism suggest ideas to examine.

In large hypothesis spaces, the vast majority of the cognitive labor goes into noticing the true hypothesis. By the time you have enough evidence to consider the correct theory as one of just a few plausible alternatives - to represent the correct theory in your mind - you're practically done. Of this I have spoken several times before.

And by the same token, my experience suggests that as soon as you let anthropomorphism promote a hypothesis to your attention, so that you start wondering if that particular hypothesis might be true, you've already committed most of the mistake.

The group selectionists did not deliberately extend credit to the belief that evolution would do the aesthetic thing, the nice thing. The group selectionists were doomed when they let their aesthetic sense make a suggestion - when they let it promote a hypothesis to the level of deliberate consideration.

It's not like I knew the original group selectionists. But I've made analogous mistakes as a teenager, and then watched others make the mistake many times over. So I do have some experience whereof I speak, when I speak of instant doom.

Unfortunately, the prophylactic against this mistake, is not a recognized technique of Traditional Rationality.

In Traditional Rationality, you can get your ideas from anywhere. Then you weigh up the evidence for and against them, searching for arguments on both sides. If the question hasn't been definitely settled by experiment, you should try to do an experiment to test your opinion, and dutifully accept the result.

"Sorry, you're not allowed to suggest ideas using that method" is not something you hear, under Traditional Rationality.

But it is a fact of life, an experimental result of cognitive psychology, that when people have an idea from any source, they tend to search for support rather than contradiction - even in the absence of emotional commitment (see link).

It is a fact of life that priming and contamination occur: just being briefly exposed to completely uninformative, known false, or totally irrelevant "information" can exert significant influence on subjects' estimates and decisions. This happens on a level below deliberate awareness, and that's going to be pretty hard to beat on problems where anthropomorphism is bound to rush in and make suggestions - but at least you can avoid deliberately making it worse.

It is a fact of life that we change our minds less often than we think. Once an idea gets into our heads, it is harder to get it out than we think. Only an extremely restrictive chain of reasoning, that definitely prohibited most possibilities from consideration, would be sufficient to undo this damage - to root an idea out of your head once it lodges. The less you know for sure, the easier it is to become contaminated - weak domain knowledge increases contamination effects.

It is a fact of life that we are far more likely to stop searching for further alternatives at a point when we have a conclusion we like, than when we have a conclusion we dislike.

It is a fact of life that we hold ideas we would like to believe, to a lower standard of proof than ideas we would like to disbelieve. In the former case we ask "Am I allowed to believe it?" and in the latter case ask "Am I forced to believe it?" If your domain knowledge is weak, you will not know enough for your own knowledge to grab you by the throat and tell you "You're wrong! That can't possibly be true!" You will find that you are allowed to believe it. You will search for plausible-sounding scenarios where your belief is true. If the search space of possibilities is large, you will almost certainly find some "winners" - your domain knowledge being too weak to definitely prohibit those scenarios.

It is a fact of history that the group selectionists failed to relinquish their folly. They found what they thought was a perfectly plausible way that evolution (evolution!) could end up producing foxes who voluntarily avoided reproductive opportunities(!). And the group selectionists did in fact cling to that hypothesis. That's what happens in real life! Be warned!

To beat anthropomorphism you have to be scared of letting anthropomorphism make suggestions. You have to try to avoid being contaminated by anthropomorphism (to the best extent you can).

As soon as you let anthropomorphism generate the idea and ask, "Could it be true?" then your brain has already swapped out of forward-extrapolation mode and into backward-rationalization mode. Traditional Rationality contains inadequate warnings against this, IMO. See in particular the post where I argue against the Traditional interpretation of Devil's Advocacy.

Yes, there are occasions when you want to perform abductive inference, such as when you have evidence that something is true and you are asking how it could be true. We call that "Bayesian updating", in fact. An occasion where you don't have any evidence but your brain has made a cute little anthropomorphic suggestion, is not a time to start wondering how it could be true. Especially if the search space of possibilities is large, and your domain knowledge is too weak to prohibit plausible-sounding scenarios. Then your prediction ends up being determined by anthropomorphism. If the real process is not controlled by a brain similar to yours, this is not a good thing for your predictive accuracy.

This is a war I wage primarily on the battleground of Unfriendly AI, but it seems to me that many of the conclusions apply to optimism in general.

How did the idea first come to you, that the subprime meltdown wouldn't decrease the value of your investment in Danish deuterium derivatives? Were you just thinking neutrally about the course of financial events, trying to extrapolate some of the many different ways that one financial billiard ball could ricochet off another? Even this method tends to be subject to optimism; if we know which way we want each step to go, we tend to visualize it going that way. But better that, than starting with a pure hope - an outcome generated because it ranked high in your preference ordering - and then permitting your mind to invent plausible-sounding reasons it might happen. This is just rushing to failure.

And to spell out the application to Unfriendly AI: You've got various people insisting that an arbitrary mind, including an expected paperclip maximizer, would do various nice things or obey various comforting conditions: "Keep humans around, because diversity is important to creativity, and the humans will provide a different point of view." Now you might want to seriously ask if, even granting that premise, you'd be kept in a nice house with air conditioning; or kept in a tiny cell with life support tubes and regular electric shocks if you didn't generate enough interesting ideas that day (and of course you wouldn't be allowed to die); or uploaded to a very small computer somewhere, and restarted every couple of years. No, let me guess, you'll be more productive if you're happy. So it's clear why you want that to be the argument; but unlike you, the paperclip maximizer is not frantically searching for a reason not to torture you.

Sorry, the whole scenario is still around as unlikely as your carefully picking up ants on the sidewalk, rather than stepping on them, and keeping them in a happy ant colony for the sole express purpose of suggesting blog comments. There are reasons in my goal system to keep sentient beings alive, even if they aren't "useful" at the moment. But from the perspective of a Bayesian superintelligence whose only terminal value is paperclips, it is not an optimal use of matter and energy toward the instrumental value of producing diverse and creative ideas for making paperclips, to keep around six billion highly similar human brains. Unlike you, the paperclip maximizer doesn't start out knowing it wants that to be the conclusion.

Your brain starts out knowing that it wants humanity to live, and so it starts trying to come up with arguments for why that is a perfectly reasonable thing for a paperclip maximizer to do. But the paperclip maximizer itself would not start from the conclusion that it wanted humanity to live, and reason backward. It would just try to make paperclips. It wouldn't stop, the way your own mind tends to stop, if it did find one argument for keeping humans alive; instead it would go on searching for an even superior alternative, some way to use the same resources to greater effect. Maybe you just want to keep 20 humans and randomly perturb their brain states a lot.

If you can't blind your eyes to human goals and just think about the paperclips, you can't understand what the goal of making paperclips implies. It's like expecting kind and merciful results from natural selection, which lets old elephants starve to death when they run out of teeth.

A priori, if you want a nice result that takes 10 bits to specify, then a priori you should expect a 1/1024 probability of finding that some unrelated process generates that nice result. And a genuinely nice outcome in a large outcome space takes a lot more information than the English word "nice", because what we consider a good outcome has many components of value. It's extremely suspicious if you start out with a nice result in mind, search for a plausible reason that a not-inherently-nice process would generate it, and, by golly, find an amazing clever argument.

And the more complexity you add to your requirements - humans not only have to survive, but have to survive under what we would consider good living conditions, etc. - the less you should expect, a priori, a non-nice process to generate it. The less you should expect to, amazingly, find a genuine valid reason why the non-nice process happens to do what you want. And the more suspicious you should be, if you find a clever-sounding argument why this should be the case. To expect this to happen with non-trivial probability is pulling information from nowhere; a blind arrow is hitting the center of a small target. Are you sure it's wise to even search for such possibilities? Your chance of deceiving yourself is far greater than the a priori chance of a good outcome, especially if your domain knowledge is too weak to definitely rule out possibilities.

No more than you can guess a lottery ticket, should you expect a process not shaped by human niceness, to produce nice results in a large outcome space. You may not know the domain very well, but you can understand that, a priori, "nice" results require specific complexity to happen for no reason, and complex specific miracles are rare.

I wish I could tell people: "Stop! Stop right there! You defeated yourself the moment you knew what you wanted! You need to throw away your thoughts and start over with a neutral forward extrapolation, not seeking any particular outcome." But the inferential distance is too great; and then begins the slog of, "I don't see why that couldn't happen" and "I don't think you've proven my idea is wrong."

It's Unfriendly superintelligence that tends to worry me most, of course. But I do think the point generalizes to quite a lot of optimism. You may know what you want, but Nature doesn't care.

Original with comments: Contaminated by Optimism

Hiroshima Day

Eliezer Yudkowsky, 06 August 2008 11:15PM

On August 6th, in 1945, the world saw the first use of atomic weapons against human targets. On this day 63 years ago, humanity lost its nuclear virginity. Until the end of time we will be a species that has used fission bombs in anger.

Time has passed, and we still haven't blown up our world, despite a close call or two. Which makes it difficult to criticize the decision - would things still have turned out all right, if anyone had chosen differently, anywhere along the way?

Maybe we needed to see the ruins, of the city and the people.

Maybe we didn't.

There's an ongoing debate - and no, it is not a settled issue - over whether the Japanese would have surrendered without the Bomb. But I would not have dropped the Bomb even to save the lives of American soldiers, because I would have wanted to preserve that world where atomic weapons had never been used - to not cross that line. I don't know about history to this point; but the world would be safer now, I think, today, if no one had ever used atomic weapons in war, and the idea was not considered suitable for polite discussion.

I'm not saying it was wrong. I don't know for certain that it was wrong. I wouldn't have thought that humanity could make it this far without using atomic weapons again. All I can say is that if it had been me, I wouldn't have done it.

Original with comments: Hiroshima Day

Morality as Fixed Computation

Eliezer Yudkowsky, 08 August 2008 01:00AM

Followup to: The Meaning of Right

Toby Ord commented:

Eliezer, I've just reread your article and was wondering if this is a good quick summary of your position (leaving apart how you got to it):

'I should X' means that I would attempt to X were I fully informed.

Toby's a pro, so if he didn't get it, I'd better try again. Let me try a different tack of explanation - one closer to the historical way that I arrived at my own position.

Suppose you build an AI, and - leaving aside that AI goal systems cannot be built around English statements, and all such descriptions are only dreams - you try to infuse the AI with the action-determining principle, "Do what I want."

And suppose you get the AI design close enough - it doesn't just end up tiling the universe with paperclips, cheesecake or tiny molecular copies of satisfied programmers - that its utility function actually assigns utilities as follows, to the world-states we would describe in English as:

<Programmer weakly desires 'X', quantity 20 of X exists>: +20
<Programmer strongly desires 'Y', quantity 20 of X exists>: 0
<Programmer weakly desires 'X', quantity 30 of Y exists>: 0
<Programmer strongly desires 'Y', quantity 30 of Y exists>: +60

You perceive, of course, that this destroys the world.

...since if the programmer initially weakly wants 'X' and X is hard to obtain, the AI will modify the programmer to strongly want 'Y', which is easy to create, and then bring about lots of Y. Y might be, say, iron atoms - those are highly stable.

Can you patch this problem? No. As a general rule, it is not possible to patch flawed Friendly AI designs.

If you try to bound the utility function, or make the AI not care about how much the programmer wants things, the AI still has a motive (as an expected utility maximizer) to make the programmer want something that can be obtained with a very high degree of certainty.

If you try to make it so that the AI can't modify the programmer, then the AI can't talk to the programmer (talking to someone modifies them).

If you try to rule out a specific class of ways the AI could modify the programmer, the AI has a motive to superintelligently seek out loopholes and ways to modify the programmer indirectly.

As a general rule, it is not possible to patch flawed FAI designs.

We, ourselves, do not imagine the future and judge, that any future in which our brains want something, and that thing exists, is a good future. If we did think this way, we would say: "Yay! Go ahead and modify us to strongly want something cheap!" But we do not say this, which means that this AI design is fundamentally flawed: it will choose things very unlike what we would choose; it will judge desirability very differently from how we judge it. This core disharmony cannot be patched by ruling out a handful of specific failure modes.

There's also a duality between Friendly AI problems and moral philosophy problems - though you've got to structure that duality in exactly the right way. So if you prefer, the core problem is that the AI will choose in a way very unlike the structure of what is, y'know, actually right - never mind the way we choose. Isn't the whole point of this problem, that merely wanting something doesn't make it right?

So this is the paradoxical-seeming issue which I have analogized to the difference between:

A calculator that, when you press '2', '+', and '3', tries to compute:
"What is 2 + 3?"

A calculator that, when you press '2', '+', and '3', tries to compute:
"What does this calculator output when you press '2', '+', and '3'?"

The Type 1 calculator, as it were, wants to output 5.

The Type 2 "calculator" could return any result; and in the act of returning that result, it becomes the correct answer to the question that was internally asked.

We ourselves are like unto the Type 1 calculator. But the putative AI is being built as though it were to reflect the Type 2 calculator.

Now imagine that the Type 1 calculator is trying to build an AI, only the Type 1 calculator doesn't know its own question. The calculator continually asks the question by its very nature, it was born to ask that question, created already in motion around that question - but the calculator has no insight into its own transistors; it cannot print out the question, which is extremely complicated and has no simple approximation.

So the calculator wants to build an AI (it's a pretty smart calculator, it just doesn't have access to its own transistors) and have the AI give the right answer. Only the calculator can't print out the question. So the calculator wants to have the AI look at the calculator, where the question is written, and answer the question that the AI will discover implicit in those transistors. But this cannot be done by the cheap shortcut of a utility function that says "All X: <calculator asks 'X?', answer X>: utility 1; else: utility 0" because that actually mirrors the utility function of a Type 2 calculator, not a Type 1 calculator.

This gets us into FAI issues that I am not going into (some of which I'm still working out myself).

However, when you back out of the details of FAI design, and swap back to the perspective of moral philosophy, then what we were just talking about was the dual of the moral issue: "But if what's 'right' is a mere preference, then anything that anyone wants is 'right'."

Now I did argue against that particular concept in some detail, in The Meaning of Right, so I am not going to repeat all that...

But the key notion is the idea that what we name by 'right' is a fixed question, or perhaps a fixed framework. We can encounter moral arguments that modify our terminal values, and even encounter moral arguments that modify what we count as a moral argument; nonetheless, it all grows out of a particular starting point. We do not experience ourselves as embodying the question "What will I decide to do?" which would be a Type 2 calculator; anything we decided would thereby become right. We experience ourselves as asking the embodied question: "What will save my friends, and my people, from getting hurt? How can we all have more fun? ..." where the "..." is around a thousand other things.

So 'I should X' does not mean that I would attempt to X were I fully informed.

'I should X' means that X answers the question, "What will save my people? How can we all have more fun? How can we get more control over our own lives? What's the funniest jokes we can tell? ..."

And I may not know what this question is, actually; I may not be able to print out my current guess nor my surrounding framework; but I know, as all non-moral-relativists instinctively know, that the question surely is not just "How can I do whatever I want?"

When these two formulations begin to seem as entirely distinct as "snow" and snow, then you shall have created distinct buckets for the quotation and the referent.

Added: This was posted automatically and the front page got screwed up somehow. I have no idea how. It is now fixed and should make sense.

Sequence: Metaethics

Interpersonal Morality

Inseparably Right; or, Joy in the Merely Good

Referenced by: Inseparably Right; or, Joy in the Merely Good • Moral Error and Moral Disagreement • Abstracted Idealized Dynamics • You Provably Can't Trust Yourself • Mirrors and Paintings

Original with comments: Morality as Fixed Computation

Inseparably Right; or, Joy in the Merely Good

Eliezer Yudkowsky, 09 August 2008 01:00AM

Followup to: The Meaning of Right

I fear that in my drive for full explanation, I may have obscured the punchline from my theory of metaethics. Here then is an attempted rephrase:

There is no pure ghostly essence of goodness apart from things like truth, happiness and sentient life.

What do you value? At a guess, you value the life of your friends and your family and your Significant Other and yourself, all in different ways. You would probably say that you value human life in general, and I would take your word for it, though Robin Hanson might ask how you've acted on this supposed preference. If you're reading this blog you probably attach some value to truth for the sake of truth. If you've ever learned to play a musical instrument, or paint a picture, or if you've ever solved a math problem for the fun of it, then you probably attach real value to good art. You value your freedom, the control that you possess over your own life; and if you've ever really helped someone you probably enjoyed it. You might not think of playing a video game as a great sacrifice of dutiful morality, but I for one would not wish to see the joy of complex challenge perish from the universe. You may not think of telling jokes as a matter of interpersonal morality, but I would consider the human sense of humor as part of the gift we give to tomorrow.

And you value many more things than these.

Your brain assesses these things I have said, or others, or more, depending on the specific event, and finally affixes a little internal representational label that we recognize and call "good".

There's no way you can detach the little label from what it stands for, and still make ontological or moral sense.

Why might the little 'good' label seem detachable? A number of reasons.

Mainly, that's just how your mind is structured - the labels it attaches internally seem like extra, floating, ontological properties.

And there's no one value that determines whether a complicated event is good or not - and no five values, either. No matter what rule you try to describe, there's always something left over, some counterexample. Since no single value defines goodness, this can make it seem like all of them together couldn't define goodness. But when you add them up all together, there is nothing else left.

If there's no detachable property of goodness, what does this mean?

It means that the question, "Okay, but what makes happiness or self-determination, good?" is either very quickly answered, or else malformed.

The concept of a "utility function" or "optimization criterion" is detachable when talking about optimization processes. Natural selection, for example, optimizes for inclusive genetic fitness. But there are possible minds that implement any utility function, so you don't get any advice there about what you should do. You can't ask about utility apart from any utility function.

When you ask "But which utility function should I use?" the word should is something inseparable from the dynamic that labels a choice "should" - inseparable from the reasons like "Because I can save more lives that way."

Every time you say should, it includes an implicit criterion of choice; there is no should-ness that can be abstracted away from any criterion.

There is no separable right-ness that you could abstract from pulling a child off the train tracks, and attach to some other act.

Your values can change in response to arguments; you have metamorals as well as morals. So it probably does make sense to think of an idealized good, or idealized right, that you would assign if you could think of all possible arguments. Arguments may even convince you to change your criteria of what counts as a persuasive argument. Even so, when you consider the total trajectory arising out of that entire framework, that moral frame of reference, there is no separable property of justification-ness, apart from any particular criterion of justification; no final answer apart from a starting question.

I sometimes say that morality is "created already in motion".

There is no perfect argument that persuades the ideal philosopher of perfect emptiness to attach a perfectly abstract label of 'good'. The notion of the perfectly abstract label is incoherent, which is why people chase it round and round in circles. What would distinguish a perfectly empty label of 'good' from a perfectly empty label of 'bad'? How would you tell which was which?

But since every supposed criterion of goodness that we describe, turns out to be wrong, or incomplete, or changes the next time we hear a moral argument, it's easy to see why someone might think that 'goodness' was a thing apart from any criterion at all.

Humans have a cognitive architecture that easily misleads us into conceiving of goodness as something that can be detached from any criterion.

This conception turns out to be incoherent. Very sad. I too was hoping for a perfectly abstract argument; it appealed to my universalizing instinct. But...

But the question then becomes: is that little fillip of human psychology, more important than everything else? Is it more important than the happiness of your family, your friends, your mate, your extended tribe, and yourself? If your universalizing instinct is frustrated, is that worth abandoning life? If you represented rightness wrongly, do pictures stop being beautiful and maths stop being elegant? Is that one tiny mistake worth forsaking the gift we could give to tomorrow? Is it even really worth all that much in the way of existential angst?

Or will you just say "Oops" and go back to life, to truth, fun, art, freedom, challenge, humor, moral arguments, and all those other things that in their sum and in their reflective trajectory, are the entire and only meaning of the word 'right'?

Here is the strange habit of thought I mean to convey: Don't look to some surprising unusual twist of logic for your justification. Look to the living child, successfully dragged off the train tracks. There you will find your justification. What ever should be more important than that?

I could dress that up in computational metaethics and FAI theory - which indeed is whence the notion first came to me - but when I translated it all back into human-talk, that is what it turned out to say.

If we cannot take joy in things that are merely good, our lives shall be empty indeed.

Sequence: Metaethics

Morality as Fixed Computation

Sorting Pebbles Into Correct Heaps

Referenced by: Moral Error and Moral Disagreement • "Arbitrary" • The Bedrock of Morality: Arbitrary? • Singletons Rule OK • Prolegomena to a Theory of Fun • The Fun Theory Sequence • Value is Fragile • (Moral) Truth in Fiction? • Raising the Sanity Waterline • Unspeakable Morality

Original with comments: Inseparably Right; or, Joy in the Merely Good

Sorting Pebbles Into Correct Heaps

Eliezer Yudkowsky, 10 August 2008 01:00AM

Followup to: Anthropomorphic Optimism

Once upon a time there was a strange little species - that might have been biological, or might have been synthetic, and perhaps were only a dream - whose passion was sorting pebbles into correct heaps.

They couldn't tell you why some heaps were correct, and some incorrect. But all of them agreed that the most important thing in the world was to create correct heaps, and scatter incorrect ones.

Why the Pebblesorting People cared so much, is lost to this history - maybe a Fisherian runaway sexual selection, started by sheer accident a million years ago? Or maybe a strange work of sentient art, created by more powerful minds and abandoned?

But it mattered so drastically to them, this sorting of pebbles, that all the Pebblesorting philosophers said in unison that pebble-heap-sorting was the very meaning of their lives: and held that the only justified reason to eat was to sort pebbles, the only justified reason to mate was to sort pebbles, the only justified reason to participate in their world economy was to efficiently sort pebbles.

The Pebblesorting People all agreed on that, but they didn't always agree on which heaps were correct or incorrect.

In the early days of Pebblesorting civilization, the heaps they made were mostly small, with counts like 23 or 29; they couldn't tell if larger heaps were correct or not. Three millennia ago, the Great Leader Biko made a heap of 91 pebbles and proclaimed it correct, and his legions of admiring followers made more heaps likewise. But over a handful of centuries, as the power of the Bikonians faded, an intuition began to accumulate among the smartest and most educated that a heap of 91 pebbles was incorrect. Until finally they came to know what they had done: and they scattered all the heaps of 91 pebbles. Not without flashes of regret, for some of those heaps were great works of art, but incorrect. They even scattered Biko's original heap, made of 91 precious gemstones each of a different type and color.

And no civilization since has seriously doubted that a heap of 91 is incorrect.

Today, in these wiser times, the size of the heaps that Pebblesorters dare attempt, has grown very much larger - which all agree would be a most great and excellent thing, if only they could ensure the heaps were really correct. Wars have been fought between countries that disagree on which heaps are correct: the Pebblesorters will never forget the Great War of 1957, fought between Y'ha-nthlei and Y'not'ha-nthlei, over heaps of size 1957. That war, which saw the first use of nuclear weapons on the Pebblesorting Planet, finally ended when the Y'not'ha-nthleian philosopher At'gra'len'ley exhibited a heap of 103 pebbles and a heap of 19 pebbles side-by-side. So persuasive was this argument that even Y'not'ha-nthlei reluctantly conceded that it was best to stop building heaps of 1957 pebbles, at least for the time being.

Since the Great War of 1957, countries have been reluctant to openly endorse or condemn heaps of large size, since this leads so easily to war. Indeed, some Pebblesorting philosophers - who seem to take a tangible delight in shocking others with their cynicism - have entirely denied the existence of pebble-sorting progress; they suggest that opinions about pebbles have simply been a random walk over time, with no coherence to them, the illusion of progress created by condemning all dissimilar pasts as incorrect. The philosophers point to the disagreement over pebbles of large size, as proof that there is nothing that makes a heap of size 91 really incorrect - that it was simply fashionable to build such heaps at one point in time, and then at another point, fashionable to condemn them. "But... 13!" carries no truck with them; for to regard "13!" as a persuasive counterargument, is only another convention, they say. The Heap Relativists claim that their philosophy may help prevent future disasters like the Great War of 1957, but it is widely considered to be a philosophy of despair.

Now the question of what makes a heap correct or incorrect, has taken on new urgency; for the Pebblesorters may shortly embark on the creation of self-improving Artificial Intelligences. The Heap Relativists have warned against this project: They say that AIs, not being of the species Pebblesorter sapiens, may form their own culture with entirely different ideas of which heaps are correct or incorrect. "They could decide that heaps of 8 pebbles are correct," say the Heap Relativists, "and while ultimately they'd be no righter or wronger than us, still, our civilization says we shouldn't build such heaps. It is not in our interest to create AI, unless all the computers have bombs strapped to them, so that even if the AI thinks a heap of 8 pebbles is correct, we can force it to build heaps of 7 pebbles instead. Otherwise, KABOOM!"

But this, to most Pebblesorters, seems absurd. Surely a sufficiently powerful AI - especially the "superintelligence" some transpebblesorterists go on about - would be able to see at a glance which heaps were correct or incorrect! The thought of something with a brain the size of a planet, thinking that a heap of 8 pebbles was correct, is just too absurd to be worth talking about.

Indeed, it is an utterly futile project to constrain how a superintelligence sorts pebbles into heaps. Suppose that Great Leader Biko had been able, in his primitive era, to construct a self-improving AI; and he had built it as an expected utility maximizer whose utility function told it to create as many heaps as possible of size 91. Surely, when this AI improved itself far enough, and became smart enough, then it would see at a glance that this utility function was incorrect; and, having the ability to modify its own source code, it would rewrite its utility function to value more reasonable heap sizes, like 101 or 103.

And certainly not heaps of size 8. That would just be stupid. Any mind that stupid is too dumb to be a threat.

Reassured by such common sense, the Pebblesorters pour full speed ahead on their project to throw together lots of algorithms at random on big computers until some kind of intelligence emerges. The whole history of civilization has shown that richer, smarter, better educated civilizations are likely to agree about heaps that their ancestors once disputed. Sure, there are then larger heaps to argue about - but the further technology has advanced, the larger the heaps that have been agreed upon and constructed.

Indeed, intelligence itself has always correlated with making correct heaps - the nearest evolutionary cousins to the Pebblesorters, the Pebpanzees, make heaps of only size 2 or 3, and occasionally stupid heaps like 9. And other, even less intelligent creatures, like fish, make no heaps at all.

Smarter minds equal smarter heaps. Why would that trend break?

Sequence: Metaethics

Inseparably Right; or, Joy in the Merely Good

Moral Error and Moral Disagreement

Referenced by: Moral Error and Moral Disagreement • "Arbitrary" • The Bedrock of Morality: Arbitrary? • Hot Air Doesn't Disagree • No License To Be Human • Invisible Frameworks • Mirrors and Paintings • Magical Categories • Qualitative Strategies of Friendliness • The True Prisoner's Dilemma • Singletons Rule OK • Can't Unbirth a Child • Changing Emotions • Emotional Involvement • Value is Fragile

Original with comments: Sorting Pebbles Into Correct Heaps

Moral Error and Moral Disagreement

Eliezer Yudkowsky, 10 August 2008 11:32PM

Followup to: Inseparably Right, Sorting Pebbles Into Correct Heaps

Richard Chappell, a pro, writes:

"When Bob says "Abortion is wrong", and Sally says, "No it isn't", they are disagreeing with each other.

I don't see how Eliezer can accommodate this. On his account, what Bob asserted is true iff abortion is prohibited by the morality_Bob norms. How can Sally disagree? There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob...

Since there is moral disagreement, whatever Eliezer purports to be analysing here, it is not morality."

The phenomena of moral disagreement, moral error, and moral progress, on terminal values, are the primary drivers behind my metaethics. Think of how simple Friendly AI would be if there were no moral disagreements, moral errors, or moral progress!

Richard claims, "There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob."

We may not suppose, and there is disputing. Bob does not have direct, unmediated, veridical access to the output of his own morality.

I tried to describe morality as a "computation". In retrospect, I don't think this is functioning as the Word of Power that I thought I was emitting.

Let us read, for "computation", "idealized abstract dynamic" - maybe that will be a more comfortable label to apply to morality.

Even so, I would have thought it obvious that computations may be the subjects of mystery and error. Maybe it's not as obvious outside computer science?

Disagreement has two prerequisites: the possibility of agreement and the possibility of error. For two people to agree on something, there must be something they are agreeing about, a referent held in common. And it must be possible for an "error" to take place, a conflict between "P" in the map and not-P in the territory. Where these two prerequisites are present, Sally can say to Bob: "That thing we were just both talking about - you are in error about it."

Richard's objection would seem in the first place to rule out the possibility of moral error, from which he derives the impossibility of moral agreement.

So: does my metaethics rule out moral error? Is there no disputing that abortion is indeed prohibited by morality_Bob?

This is such a strange idea that I find myself wondering what the heck Richard could be thinking. My best guess is that Richard, perhaps having not read all the posts in this sequence, is taking my notion of morality_Bob to refer to a flat, static list of valuations explicitly asserted by Bob. "Abortion is wrong" would be on Bob's list, and there would be no disputing that.

But on the contrary, I conceive of morality_Bob as something that unfolds into Bob's morality - like the way one can describe in 6 states and 2 symbols a Turing machine that will write 4.640 × 10^14391s to its tape before halting.

So morality_Bob refers to a compact folded specification, and not a flat list of outputs. But still, how could Bob be wrong about the output of his own morality?

In manifold obvious and non-obvious ways:

Bob could be empirically mistaken about the state of fetuses, perhaps believing fetuses to be aware of the outside world. (Correcting this might change Bob's instrumental values but not terminal values.)

Bob could have formed his beliefs about what constituted "personhood" in the presence of confusion about the nature of consciousness, so that if Bob were fully informed about consciousness, Bob would not have been tempted to talk about "the beginning of life" or "the human kind" in order to define personhood. (This changes Bob's expressed terminal values; afterward he will state different general rules about what sort of physical things are ends in themselves.)

So those are the obvious moral errors - instrumental errors driven by empirical mistakes; and erroneous generalizations about terminal values, driven by failure to consider moral arguments that are valid but hard to find in the search space.

Then there are less obvious sources of moral error: Bob could have a list of mind-influencing considerations that he considers morally valid, and a list of other mind-influencing considerations that Bob considers morally invalid. Maybe Bob was raised a Christian and now considers that cultural influence to be invalid. But, unknown to Bob, when he weighs up his values for and against abortion, the influence of his Christian upbringing comes in and distorts his summing of value-weights. So Bob believes that the output of his current validated moral beliefs is to prohibit abortion, but actually this is a leftover of his childhood and not the output of those beliefs at all.

(Note that Robin Hanson and I seem to disagree, in a case like this, as to exactly what degree we should take Bob's word about what his morals are.)

Or Bob could believe that the word of God determines moral truth and that God has prohibited abortion in the Bible. Then Bob is making metaethical mistakes, causing his mind to malfunction in a highly general way, and add moral generalizations to his belief pool, which he would not do if veridical knowledge of the universe destroyed his current and incoherent metaethics.

Now let us turn to the disagreement between Sally and Bob.

You could suggest that Sally is saying to Bob, "Abortion is allowed by morality_Bob", but that seems a bit oversimplified; it is not psychologically or morally realistic.

If Sally and Bob were unrealistically sophisticated, they might describe their dispute as follows:

Bob: "Abortion is wrong."

Sally: "Do you think that this is something of which most humans ought to be persuadable?"

Bob: "Yes, I do. Do you think abortion is right?"

Sally: "Yes, I do. And I don't think that's because I'm a psychopath by common human standards. I think most humans would come to agree with me, if they knew the facts I knew, and heard the same moral arguments I've heard."

Bob: "I think, then, that we must have a moral disagreement: since we both believe ourselves to be a shared moral frame of reference on this issue, and yet our moral intuitions say different things to us."

Sally: "Well, it is not logically necessary that we have a genuine disagreement. We might be mistaken in believing ourselves to mean the same thing by the words right and wrong, since neither of us can introspectively report our own moral reference frames or unfold them fully."

Bob: "But if the meaning is similar up to the third decimal place, or sufficiently similar in some respects that it ought to be delivering similar answers on this particular issue, then, even if our moralities are not in-principle identical, I would not hesitate to invoke the intuitions for transpersonal morality."

Sally: "I agree. Until proven otherwise, I am inclined to talk about this question as if it is the same question unto us."

Bob: "So I say 'Abortion is wrong' without further qualification or specialization on what wrong means unto me."

Sally: "And I think that abortion is right. We have a disagreement, then, and at least one of us must be mistaken."

Bob: "Unless we're actually choosing differently because of in-principle unresolvable differences in our moral frame of reference, as if one of us were a paperclip maximizer. In that case, we would be mutually mistaken in our belief that when we talk about doing what is right, we mean the same thing by right. We would agree that we have a disagreement, but we would both be wrong."

Now, this is not exactly what most people are explicitly thinking when they engage in a moral dispute - but it is how I would cash out and naturalize their intuitions about transpersonal morality.

Richard also says, "Since there is moral disagreement..." This seems like a prime case of what I call naive philosophical realism - the belief that philosophical intuitions are direct unmediated veridical passports to philosophical truth.

It so happens that I agree that there is such a thing as moral disagreement. Tomorrow I will endeavor to justify, in fuller detail, how this statement can possibly make sense in a reductionistic natural universe. So I am not disputing this particular proposition. But I note, in passing, that Richard cannot justifiably assert the existence of moral disagreement as an irrefutable premise for discussion, though he could consider it as an apparent datum. You cannot take as irrefutable premises, things that you have not explained exactly; for then what is it that is certain to be true?

I cannot help but note the resemblance to Richard's assumption that "there's no disputing" that abortion is indeed prohibited by morality_Bob - the assumption that Bob has direct veridical unmediated access to the final unfolded output of his own morality.

Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob, and allowed by morality_Sally, there being at least two possible minds for whom this would be true. Then the two minds might be mistaken about believing themselves to disagree. Actually they would simply be directed by different algorithms.

You cannot have a disagreement about which algorithm should direct your actions, without first having the same meaning of should - and no matter how you try to phrase this in terms of "what ought to direct your actions" or "right actions" or "correct heaps of pebbles", in the end you will be left with the empirical fact that it is possible to construct minds directed by any coherent utility function.

When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes. You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about - unless you cover "disagreement" to include differences where two agents have nothing to say to each other.

But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it. Now, perhaps some psychopaths would not be persuadable in-principle to take the pill that would, by our standards, "fix" them. But I note the possibility to emphasize what an extreme statement it is to say of someone:

"We have nothing to argue about, we are only different optimization processes."

That should be reserved for paperclip maximizers, not used against humans whose arguments you don't like.

Sequence: Metaethics

Sorting Pebbles Into Correct Heaps

Abstracted Idealized Dynamics

Original with comments: Moral Error and Moral Disagreement

Abstracted Idealized Dynamics

Eliezer Yudkowsky, 12 August 2008 01:00AM

Followup to: Morality as Fixed Computation

I keep trying to describe morality as a "computation", but people don't stand up and say "Aha!"

Pondering the surprising inferential distances that seem to be at work here, it occurs to me that when I say "computation", some of my listeners may not hear the Word of Power that I thought I was emitting; but, rather, may think of some complicated boring unimportant thing like Microsoft Word.

Maybe I should have said that morality is an abstracted idealized dynamic. This might not have meant anything to start with, but at least it wouldn't sound like I was describing Microsoft Word.

How, oh how, am I to describe the awesome import of this concept, "computation"?

Perhaps I can display the inner nature of computation, in its most general form, by showing how that inner nature manifests in something that seems very unlike Microsoft Word - namely, morality.

Consider certain features we might wish to ascribe to that-which-we-call "morality", or "should" or "right" or "good":

• It seems that we sometimes think about morality in our armchairs, without further peeking at the state of the outside world, and arrive at some previously unknown conclusion.

Someone sees a slave being whipped, and it doesn't occur to them right away that slavery is wrong. But they go home and think about it, and imagine themselves in the slave's place, and finally think, "No."

Can you think of anywhere else that something like this happens?

Suppose I tell you that I am making a rectangle of pebbles. You look at the rectangle, and count 19 pebbles on one side and 103 dots pebbles on the other side. You don't know right away how many pebbles there are. But you go home to your living room, and draw the blinds, and sit in your armchair and think; and without further looking at the physical array, you come to the conclusion that the rectangle contains 1957 pebbles.

Now, I'm not going to say the word "computation". But it seems like that-which-is "morality" should have the property of latent development of answers - that you may not know right away, everything that you have sufficient in-principle information to know. All the ingredients are present, but it takes additional time to bake the pie.

You can specify a Turing machine of 6 states and 2 symbols that unfolds into a string of 4.6 × 10^14391s after 2.5 × 10^2879 steps. A machine I could describe aloud in ten seconds, runs longer and produces a larger state than the whole observed universe to date.

When you distinguish between the program description and the program's executing state, between the process specification and the final outcome, between the question and the answer, you can see why even certainty about a program description does not imply human certainty about the executing program's outcome. See also Artificial Addition on the difference between a compact specification versus a flat list of outputs.

Morality, likewise, is something that unfolds, through arguments, through discovery, through thinking; from a bounded set of intuitions and beliefs that animate our initial states, to a potentially much larger set of specific moral judgments we may have to make over the course of our lifetimes.

• When two human beings both think about the same moral question, even in a case where they both start out uncertain of the answer, it is not unknown for them to come to the same conclusion. It seems to happen more often than chance alone would allow - though the biased focus of reporting and memory is on the shouting and the arguments. And this is so, even if both humans remain in their armchairs and do not peek out the living-room blinds while thinking.

Where else does this happen? It happens when trying to guess the number of pebbles in a rectangle of sides 19 and 103. Now this does not prove by Greek analogy that morality is multiplication. If A has property X and B has property X it does not follow that A is B. But it seems that morality ought to have the property of expected agreement about unknown latent answers, which, please note, generally implies that similar questions are being asked in different places.

This is part of what is conveyed by the Word of Power, "computation": the notion of similar questions being asked in different places and having similar answers. Or as we might say in the business, the same computation can have multiple instantiations.

If we know the structure of calculator 1 and calculator 2, we can decide that they are "asking the same question" and that we ought to see the "same result" flashing on the screen of calculator 1 and calculator 2 after pressing the Enter key. We decide this in advance of seeing the actual results, which is what makes the concept of "computation" predictively useful.

And in fact, we can make this deduction even without knowing the exact circuit diagrams of calculators 1 and 2, so long as we're told that the circuit diagrams are the same.

And then when we see the result "1957" flash on the screen of calculator 1, we know that the same "1957" can be expected to flash on calculator 2, and we even expect to count up 1957 pebbles in the array of 19 by 103.

A hundred calculators, performing the same multiplication in a hundred different ways, can be expected to arrive at the same answer - and this is not a vacuous expectation adduced after seeing similar answers. We can form the expectation in advance of seeing the actual answer.

Now this does not show that morality is in fact a little electronic calculator. But it highlights the notion of something that factors out of different physical phenomena in different physical places, even phenomena as physically different as a calculator and an array of pebbles - a common answer to a common question. (Where is this factored-out thing? Is there an Ideal Multiplication Table written on a stone tablet somewhere outside the universe? But we are not concerned with that for now.)

Seeing that one calculator outputs "1957", we infer that the answer - the abstracted answer - is 1957; and from there we make our predictions of what to see on all the other calculator screens, and what to see in the array of pebbles.

So that-which-we-name-morality seems to have the further properties of agreement about developed latent answers, which we may as well think of in terms of abstract answers; and note that such agreement is unlikely in the absence of similar questions.

• We sometimes look back on our own past moral judgments, and say "Oops!" E.g., "Oops! Maybe in retrospect I shouldn't have killed all those guys when I was a teenager."

So by now it seems easy to extend the analogy, and say: "Well, maybe a cosmic ray hits one of the transistors in the calculator and it says '1959' instead of 1957 - that's an error."

But this notion of "error", like the notion of "computation" itself, is more subtle than it appears.

Calculator Q says '1959' and calculator X says '1957'. Who says that calculator Q is wrong, and calculator X is right? Why not say that calculator X is wrong and calculator Q is right? Why not just say, "the results are different"?

"Well," you say, drawing on your store of common sense, "if it was just those two calculators, I wouldn't know for sure which was right. But here I've got nine other calculators that all say '1957', so it certainly seems probable that 1957 is the correct answer."

What's this business about "correct"? Why not just say "different"?

"Because if I have to predict the outcome of any other calculators that compute 19 x 103, or the number of pebbles in a 19 x 103 array, I'll predict 1957 - or whatever observable outcome corresponds to the abstract number 1957."

So perhaps 19 x 103 = 1957 only most of the time. Why call the answer 1957 the correct one, rather than the mere fad among calculators, the majority vote?

If I've got a hundred calculators, all of them rather error-prone - say a 10% probability of error - then there is no one calculator I can point to and say, "This is the standard!" I might pick a calculator that would happen, on this occasion, to vote with ten other calculators rather than ninety other calculators. This is why I have to idealize the answer, to talk about this ethereal thing that is not associated with any particular physical process known to me - not even arithmetic done in my own head, which can also be "incorrect".

It is this ethereal process, this idealized question, to which we compare the results of any one particular calculator, and say that the result was "right" or "wrong".

But how can we obtain information about this perfect and un-physical answer, when all that we can ever observe, are merely physical phenomena? Even doing "mental" arithmetic just tells you about the result in your own, merely physical brain.

"Well," you say, "the pragmatic answer is that we can obtain extremely strong evidence by looking at the results of a hundred calculators, even if they are only 90% likely to be correct on any one occasion."

But wait: When do electrons or quarks or magnetic fields ever make an "error"? If no individual particle can be mistaken, how can any collection of particles be mistaken? The concept of an "error", though humans may take it for granted, is hardly something that would be mentioned in a fully reductionist view of the universe.

Really, what happens is that we have a certain model in mind of the calculator - the model that we looked over and said, "This implements 19 * 103" - and then other physical events caused the calculator to depart from this model, so that the final outcome, while physically lawful, did not correlate with that mysterious abstract thing, and the other physical calculators, in the way we had in mind. Given our mistaken beliefs about the physical process of the first calculator, we would look at its output '1959', and make mistaken predictions about the other calculators (which do still hew to the model we have in mind).

So "incorrect" cashes out, naturalistically, as "physically departed from the model that I had of it" or "physically departed from the idealized question that I had in mind". A calculator struck by a cosmic ray, is not 'wrong' in any physical sense, not an unlawful event in the universe; but the outcome is not the answer to the question you had in mind, the question that you believed empirically-falsely the calculator would correspond to.

The calculator's "incorrect" answer, one might say, is an answer to a different question than the one you had in mind - it is an empirical fact about the calculator that it implements a different computation.

• The 'right' act or the 'should' option sometimes seem to depend on the state of the physical world. For example, should you cut the red wire or the green wire to disarm the bomb?

Suppose I show you a long straight line of pebbles, and ask you, "How many pebbles would I have, if I had a rectangular array of six lines like this one?" You start to count, but only get up to 8 when I suddenly blindfold you.

Now you are not completely ignorant of the answer to this question. You know, for example, that the result will be even, and that it will be greater than 48. But you can't answer the question until you know how many pebbles were in the original line.

But mark this about the question: It wasn't a question about anything you could directly see in the world, at that instant. There was not in fact a rectangular array of pebbles, six on a side. You could perhaps lay out an array of such pebbles and count the results - but then there are more complicated computations that we could run on the unknown length of a line of pebbles. For example, we could treat the line length as the start of a Goodstein sequence, and ask whether the sequence halts. To physically play out this sequence would require many more pebbles than exist in the universe. Does it make sense to ask if the Goodstein sequence which starts with the length of this line of pebbles, "would halt"? Does it make sense to talk about the answer, in a case like this?

I'd say yes, personally.

But meditate upon the etherealness of the answer - that we talk about idealized abstract processes that never really happen; that we talk about what would happen if the law of the Goodstein sequence came into effect upon this line of pebbles, even though the law of the Goodstein sequence will never physically come into effect.

It is the same sort of etherealness that accompanies the notion of a proposition that 19 * 103 = 1957 which factors out of any particular physical calculator and is not identified with the result of any particular physical calculator.

Only now that etherealness has been mixed with physical things; we talk about the effect of an ethereal operation on a physical thing. We talk about what would happen if we ran the Goodstein process on the number of pebbles in this line here, which we have not counted - we do not know exactly how many pebbles there are. There is no tiny little XML tag upon the pebbles that says "Goodstein halts", but we still think - or at least I still think - that it makes sense to say of the pebbles that they have the property of their Goodstein sequence terminating.

So computations can be, as it were, idealized abstract dynamics - idealized abstract applications of idealized abstract laws, iterated over an imaginary causal-time that could go on for quite a number of steps (as Goodstein sequences often do).

So when we wonder, "Should I cut the red wire or the green wire?", we are not multiplying or simulating the Goodstein process, in particular. But we are wondering about something that is not physically immanent in the red wires or the green wires themselves; there is no little XML tag on the green wire, saying, "This is the wire that should be cut."

We may not know which wire defuses the bomb, but say, "Whichever wire does in fact defuse the bomb, that is the wire that should be cut."

Still, there are no little XML tags on the wires, and we may not even have any way to look inside the bomb - we may just have to guess, in real life.

So if we try to cash out this notion of a definite wire that should be cut, it's going to come out as...

...some rule that would tell us which wire to cut, if we knew the exact state of the physical world...

...which is to say, some kind of idealized abstract process into which we feed the state of the world as an input, and get back out, "cut the green wire" or "cut the red wire"...

...which is to say, the output of a computation that would take the world as an input.

• And finally I note that from the twin phenomena of moral agreement and moral error, we can construct the notion of moral disagreement.

This adds nothing to our understanding of "computation" as a Word of Power, but it's helpful in putting the pieces together.

Let's say that Bob and Sally are talking about an abstracted idealized dynamic they call "Enamuh".

Bob says "The output of Enamuh is 'Cut the blue wire'," and Sally says "The output of Enamuh is 'Cut the brown wire'."

Now there are several non-exclusive possibilities:

Either Bob or Sally could have committed an error in applying the rules of Enamuh - they could have done the equivalent of mis-multiplying known inputs.

Either Bob or Sally could be mistaken about some empirical state of affairs upon which Enamuh depends - the wiring of the bomb.

Bob and Sally could be talking about different things when they talk about Enamuh, in which case both of them are committing an error when they refer to Enamuh_Bob and Enamuh_Sally by the same name. (However, if Enamuh_Bob and Enamuh_Sally differ in the sixth decimal place in a fashion that doesn't change the output about which wire gets cut, Bob and Sally can quite legitimately gloss the difference.)

Or if Enamuh itself is defined by some other abstracted idealized dynamic, a Meta-Enamuh whose output is Enamuh, then either Bob or Sally could be mistaken about Meta-Enamuh in any of the same ways they could be mistaken about Enamuh. (But in the case of morality, we have an abstracted idealized dynamic that includes a specification of how it, itself, changes. Morality is self-renormalizing - it is not a guess at the product of some different and outside source.)

To sum up:

- Morality, like computation, involves latent development of answers;

- Morality, like computation, permits expected agreement of unknown latent answers;

- Morality, like computation, reasons about abstract results apart from any particular physical implementation;

- Morality, like computation, unfolds from bounded initial state into something potentially much larger;

- Morality, like computation, can be viewed as an idealized dynamic that would operate on the true state of the physical world - permitting us to speak about idealized answers of which we are physically uncertain;

- Morality, like computation, lets us to speak of such un-physical stuff as "error", by comparing a physical outcome to an abstract outcome - presumably in a case where there was previously reason to believe or desire that the physical process was isomorphic to the abstract process, yet this was not actually the case.

And so with all that said, I hope that the word "computation" has come to convey something other than Microsoft Word.

Sequence: Metaethics

Moral Error and Moral Disagreement

"Arbitrary"

Referenced by: "Arbitrary" • The Bedrock of Morality: Arbitrary? • Hot Air Doesn't Disagree • You Provably Can't Trust Yourself • Magical Categories

Original with comments: Abstracted Idealized Dynamics

"Arbitrary"

Eliezer Yudkowsky, 12 August 2008 05:55PM

Followup to: Inseparably Right; or, Joy in the Merely Good, Sorting Pebbles Into Correct Heaps

One of the experiences of following the Way is that, from time to time, you notice a new word that you have been using without really understanding. And you say: "What does this word, 'X', really mean?"

Perhaps 'X' is 'error', for example. And those who have not yet realized the importance of this aspect of the Way, may reply: "Huh? What do you mean? Everyone knows what an 'error' is; it's when you get something wrong, when you make a mistake." And you reply, "But those are only synonyms; what can the term 'error' mean in a universe where particles only ever do what they do?"

It's not meant to be a rhetorical question; you're meant to go out and answer it. One of the primary tools for doing so is Rationalist's Taboo, when you try to speak without using the word or its synonyms - to replace the symbol with the substance.

So I ask you therefore, what is this word "arbitrary"? Is a rock arbitrary? A leaf? A human?

How about sorting pebbles into prime-numbered heaps? How about maximizing inclusive genetic fitness? How about dragging a child off the train tracks?

How can I tell exactly which things are arbitrary, and which not, in this universe where particles only ever do what they do? Can you tell me exactly what property is being discriminated, without using the word "arbitrary" or any direct synonyms? Can you open up the box of "arbitrary", this label that your mind assigns to some things and not others, and tell me what kind of algorithm is at work here?

Having pondered this issue myself, I offer to you the following proposal:

A piece of cognitive content feels "arbitrary" if it is the kind of cognitive content that we expect to come with attached justifications, and those justifications are not present in our mind.

You'll note that I've performed the standard operation for guaranteeing that a potentially confusing question has a real answer: I substituted the question, "How does my brain label things 'arbitrary'?" for "What is this mysterious property of arbitrariness?" This is not necessarily a sleight-of-hand, since to explain something is not the same as explaining it away.

In this case, for nearly all everyday purposes, I would make free to proceed from "arbitrary" to arbitrary. If someone says to me, "I believe that the probability of finding life on Mars is 6.203 * 10-23 to four significant digits," I would make free to respond, "That sounds like a rather arbitrary number," not "My brain has attached the subjective arbitrariness-label to its representation of the number in your belief."

So as it turned out in this case, having answered the question "What is 'arbitrary'?" turns out not to affect the way I use the word 'arbitrary'; I am just more aware of what the arbitrariness-sensation indicates. I am aware that when I say, "6.203 * 10-23 sounds like an arbitrary number", I am indicating that I would expect some justification for assigning that particular number, and I haven't heard it. This also explains why the precision is important - why I would question that particular number, but not someone saying "Less than 1%". In the latter case, I have some idea what might justify such a statement; but giving a very precise figure implies that you have some kind of information I don't know about, either that or you're being silly.

"Ah," you say, "but what do you mean by 'justification'? Haven't you failed to make any progress, and just passed the recursive buck to another black box?"

Actually, no; I told you that "arbitrariness" was a sensation produced by the absence of an expected X. Even if I don't tell you anything more about that X, you've learned something about the cognitive algorithm - opened up the original black box, and taken out two gears and a smaller black box.

But yes, it makes sense to continue onward to discuss this mysterious notion of "justification".

Suppose I told you that "justification" is what tells you whether a belief is reasonable. Would this tell you anything? No, because there are no extra gears that have been factored out, just a direct invocation of "reasonable"-ness.

Okay, then suppose instead I tell you, "Your mind labels X as a justification for Y, whenever adding 'X' to the pool of cognitive content would result in 'Y' being added to the pool, or increasing the intensity associated with 'Y'." How about that?

"Enough of this buck-passing tomfoolery!" you may be tempted to cry. But wait; this really does factor out another couple of gears. We have the idea that different propositions, to the extent they are held, can create each other in the mind, or increase the felt level of intensity - credence for beliefs, desire for acts or goals. You may have already known this, more or less, but stating it aloud is still progress.

This may not provide much satisfaction to someone inquiring into morals. But then someone inquiring into morals may well do better to just think moral thoughts, rather than thinking about metaethics or reductionism.

On the other hand, if you were building a Friendly AI, and trying to explain to that FAI what a human being means by the term "justification", then the statement I just issued might help the FAI narrow it down. With some additional guidance, the FAI might be able to figure out where to look, in an empirical model of a human, for representations of the sort of specific moral content that a human inquirer-into-morals would be interested in - what specifically counts or doesn't count as a justification, in the eyes of that human. And this being the case, you might not have to explain the specifics exactly correctly at system boot time; the FAI knows how to find out the rest on its own. My inquiries into metaethics are not directed toward the same purposes as those of standard philosophy.

Now of course you may reply, "Then the FAI finds out what the human thinks is a "justification". But is that formulation of 'justification', really justified?" But by this time, I hope, you can predict my answer to that sort of question, whether or not you agree. I answer that we have just witnessed a strange loop through the meta-level, in which you use justification-as-justification to evaluate the quoted form of justification-as-cognitive-algorithm, which algorithm may, perhaps, happen to be your own, &c. And that the feeling of "justification" cannot be coherently detached from the specific algorithm we use to decide justification in particular cases; that there is no pure empty essence of justification that will persuade any optimization process regardless of its algorithm, &c.

And the upshot is that differently structured minds may well label different propositions with their analogues of the internal label "arbitrary" - though only one of these labels is what you mean when you say "arbitrary", so you and these other agents do not really have a disagreement.

Sequence: Metaethics

Abstracted Idealized Dynamics

Is Fairness Arbitrary?

Referenced by: The Bedrock of Morality: Arbitrary?

Original with comments: "Arbitrary"

Is Fairness Arbitrary?

Eliezer Yudkowsky, 14 August 2008 01:54AM

Followup to: The Bedrock of Fairness

In "The Bedrock of Fairness", Xannon, Yancy, and Zaire argue over how to split up a pie that they found in the woods. Yancy thinks that 1/3 each is fair; Zaire demands half; and Xannon tries to compromise.

Dividing a pie fairly isn't as trivial a problem as it may sound. What if people have different preferences for crust, filling, and topping? Should they each start with a third, and trade voluntarily? But then they have conflicts of interest over how to divide the surplus utility generated by trading...

But I would say that "half for Zaire" surely isn't fair.

I confess that I originally wrote Zaire as a foil - this is clearer in an earlier version of the dialog, where Zaire, named Dennis, demands the whole pie - and was surprised to find some of my readers taking Zaire's claim seriously, perhaps because I had Zaire say "I'm hungry."

Well, okay; I believe that when I write a dialogue, the reader has a right to their own interpretation. But I did intend that dialogue to illustrate a particular point:

You can argue about how to divide up the pie, or even argue how to argue about dividing up the pie, you can argue over what is fair... but there finally comes a point when you hit bedrock. If Dennis says, "No, the fair way to argue is that I get to dictate everything, and I now hereby dictate that I get the whole pie," there's nothing left to say but "Sorry, that's just not what fairness is - you can try to take the pie and I can try to stop you, but you can't convince that that is fair."

A "fair division" is not the same as "a division that compels everyone to admit that the division is fair". Dennis can always just refuse to agree, after all.

But more to the point, when you encounter a pie in the forest, in the company of friends, and you try to be fair, there's a certain particular thing you're trying to do - the term "fair" is not perfectly empty, it cannot attach to just anything. Metaphorically speaking, "fair" is not a hypothesis equally compatible with any outcome.

Fairness expresses notions of concern for the other agents who also want the pie; a goal to take their goals into account. It's a separate question whether that concern is pure altruism, or not wanting to make them angry enough to fight. Fairness expresses notions of symmetry, equal treatment - which might be a terminal value unto you, or just an attempt to find a convenient meeting-point to avoid an outright battle.

Is it fair to take into account what other people think is "fair", and not just what you think is "fair"?

The obvious reason to care what other people think is "fair", is if they're being moved by similar considerations, yet arriving at different conclusions. If you think that the Other's word "fair" means what you think of as fair, and you think the Other is being honest about what they think, then you ought to pay attention just by way of fulfilling your own desire to be fair. It is like paying attention to an honest person who means the same thing you do by "multiplication", who says that 19 * 103 might not be 1947. The attention you pay to that suggestion, is not a favor to the other person; it is something you do if you want to get the multiplication right - they're doing you a favor by correcting you.

Politics is more subject to bias than multiplication. And you might think that the Other's reasoning is corrupted by self-interest, while yours is as pure as Antarctic snow. But to the extent that you credit the Other's self-honesty, or doubt your own, you would do well to hear what the Other has to say - if you wish to be fair.

The second notion of why we might pay attention to what someone else thinks is "fair", is more complicated: it is the notion of applying fairness to its own quotation, that is, fairly debating what is "fair". In complicated politics you may have to negotiate a negotiating procedure. Surely it wouldn't be fair if Dennis just got to say, "The fair resolution procedure is that I get to decide what's fair." So why should you get to just decide what's fair, then?

Here the attention you pay to the other person's beliefs about "fairness", is a favor that you do to them, a concession that you expect to be met with a return concession.

But when you set out to fairly discuss what is "fair" (note the strange loop through the meta-level), that doesn't put everything up for grabs. A zeroth-order fair division of a pie doesn't involve giving away the whole pie to Dennis - just giving identical portions to all. Even though Dennis wants the whole thing, and asks for the whole thing, the zeroth-order fair division only gives Dennis a symmetrical portion to everyone else's. Similarly, a first-order fair attempt to resolve a dispute about what is "fair", doesn't involve conceding everything to the Other's viewpoint without reciprocation. That wouldn't be fair. Why give everything away to the Other, if you receive nothing in return? Why give Dennis the whole first-order pie?

On some level, then, there has to be a possible demand which would be too great - a demand exceeding what may be fairly requested of you. This is part of the content of fairness; it is part of what you are setting out to do, when you set out to be fair. Admittedly, one should not be too trigger-happy about saying "That's too much!" We human beings tend to overestimate the concessions we have made, and underestimate the concessions that others have made to us; we tend to underadjust for the Other's point of view... even so, if nothing is "too much", then you're not engaging in fairness.

Fairness might call on you to hear out what the Other has to say; fairness may call on you to exert an effort to really truly consider the Other's point of view - but there is a limit to this, as there is a limit to all fair concessions. If all Dennis can say is "I want the whole pie!" over and over, there's a limit to how long fairness requires you to ponder this argument.

You reach the bedrock of fairness at the point where, no matter who questions whether the division is fair, no matter who refuses to be persuaded, no matter who offers further objections, and regardless of your awareness that you yourself may be biased... Dennis still isn't getting the whole pie. If there are others present who are also trying to be fair, and Dennis is not already dictator, they will probably back you rather than Dennis - this is one sign that you can trust the line you've drawn, that it really is time to say "Enough!"

If you and the others present get together and give Dennis 1/Nth of the pie - or even if you happen to have the upper hand, and you unilaterally give Dennis and yourself and all others each 1/Nth - then you are not being unfair on any level; there is no meta-level of fairness where Dennis gets the whole pie.

Now I'm sure there are some in the audience who will say, "You and perhaps some others, are merely doing things your way, rather than Dennis's." On the contrary: We are merely being fair. It so happens that this fairness is our way, as all acts must be someone's way to happen in the real universe. But what we are merely doing, happens to be, being fair. And there is no level on which it is unfair, because there is no level on which fairness requires unlimited unreciprocated surrender.

I don't believe in unchangeable bedrock - I believe in self-modifying bedrock. But I do believe in bedrock, in the sense that everything has to start somewhere. It can be turtles all the way up, but not turtles all the way down.

You cannot define fairness entirely in terms of "That which everyone agrees is 'fair'." This isn't just nonterminating. It isn't just ill-defined if Dennis doesn't believe that 'fair' is "that which everyone agrees is 'fair'". It's actually entirely empty, like the English sentence "This sentence is true." Is that sentence true? Is it false? It is neither; it doesn't mean anything because it is entirely wrapped up in itself, with no tentacle of relation to reality. If you're going to argue what is fair, there has to be something you're arguing about, some structure that is baked into the question.

Which is to say that you can't turn "fairness" into an ideal label of pure emptiness, defined only by the mysterious compulsion of every possible agent to admit "This is what is 'fair'." Forget the case against universally compelling arguments - just consider the definition itself: It has absolutely no content, no external references; it is not just underspecified, but entirely unspecified.

But as soon as you introduce any content into the label "fairness" that isn't phrased purely in terms of all possible minds applying the label, then you have a foundation on which to stand. It may be self-modifying bedrock, rather than immovable bedrock. But it is still a place to start. A place from which to say: "Regardless of what Dennis says, giving him the whole pie isn't fair, because fairness is not defined entirely and only in terms of Dennis's agreement."

And you aren't being "arbitrary", either - though the intuitive meaning of that word has never seemed entirely well-specified to me; is a tree arbitrary, or a leaf? But it sounds like the accusation is of pulling some answer out of thin air - which you're not doing; you're giving the fair answer, not an answer pulled out of thin air. What about when you jump up a meta-level, and look at Dennis's wanting to do it one way, and your wanting a different resolution? Then it's still not arbitrary, because you aren't being unfair on that meta-level, either. The answer you pull out is not merely an arbitrary answer you invented, but a fair answer. You aren't merely doing it your way; the way that you are doing it, is the fair way.

You can ask "But why should you be fair?" - and that's a separate question, which we'll go into tomorrow. But giving Dennis 1/Nth, we can at least say, is not merely and only arbitrary from the perspective of fair-vs.-unfair. Even if Dennis keeps saying "It isn't fair!" and even if Dennis also disputes the 1st-order, 2nd-order, Nth-order meta-fairnesses. Giving N people each 1/Nth is nonetheless a fair sort of thing to do, and whether or not we should be fair is then a separate question.

Sequence: Metaethics

"Arbitrary"

The Bedrock of Morality: Arbitrary?

Referenced by: The Bedrock of Morality: Arbitrary?

Original with comments: Is Fairness Arbitrary?

The Bedrock of Morality: Arbitrary?

Eliezer Yudkowsky, 14 August 2008 10:00PM

Followup to: Is Fairness Arbitrary?, Joy in the Merely Good, Sorting Pebbles Into Correct Heaps

Yesterday, I presented the idea that when only five people are present, having just stumbled across a pie in the woods (a naturally growing pie, that just popped out of the ground) then it is fair to give Dennis only 1/5th of this pie, even if Dennis persistently claims that it is fair for him to get the whole thing. Furthermore, it is meta-fair to follow such a symmetrical division procedure, even if Dennis insists that he ought to dictate the division procedure.

Fair, meta-fair, or meta-meta-fair, there is no level of fairness where you're obliged to concede everything to Dennis, without reciprocation or compensation, just because he demands it.

Which goes to say that fairness has a meaning beyond which "that which everyone can be convinced is 'fair'". This is an empty proposition, isomorphic to "Xyblz is that which everyone can be convinced is 'xyblz'". There must be some specific thing of which people are being convinced; and once you identify that thing, it has a meaning beyond agreements and convincing.

You're not introducing something arbitrary, something un-fair, in refusing to concede everything to Dennis. You are being fair, and meta-fair and meta-meta-fair. As far up as you go, there's no level that calls for unconditional surrender. The stars do not judge between you and Dennis - but it is baked into the very question that is asked, when you ask, "What is fair?" as opposed to "What is xyblz?"

Ah, but why should you be fair, rather than xyblz? Let us concede that Dennis cannot validly persuade us, on any level, that it is fair for him to dictate terms and give himself the whole pie; but perhaps he could argue whether we should be fair?

The hidden agenda of the whole discussion of fairness, of course, is that good-ness and right-ness and should-ness, ground out similarly to fairness.

Natural selection optimizes for inclusive genetic fitness. This is not a disagreement with humans about what is good. It is simply that natural selection does not do what is good: it optimizes for inclusive genetic fitness.

Well, since some optimization processes optimize for inclusive genetic fitness, instead of what is good, which should we do, ourselves?

I know my answer to this question. It has something to do with natural selection being a terribly wasteful and stupid and inefficient process. It has something to do with elephants starving to death in their old age when they wear out their last set of teeth. It has something to do with natural selection never choosing a single act of mercy, of grace, even when it would cost its purpose nothing: not auto-anesthetizing a wounded and dying gazelle, when its pain no longer serves even the adaptive purpose that first created pain. Evolution had to happen sometime in the history of the universe, because that's the only way that intelligence could first come into being, without brains to make brains; but now that era is over, and good riddance.

But most of all - why on Earth would any human being think that one ought to optimize inclusive genetic fitness, rather than what is good? What is even the appeal of this, morally or otherwise? At all? I know people who claim to think like this, and I wonder what wrong turn they made in their cognitive history, and I wonder how to get them to snap out of it.

When we take a step back from fairness, and ask if we should be fair, the answer may not always be yes. Maybe sometimes we should be merciful. But if you ask if it is meta-fair to be fair, the answer will generally be yes. Even if someone else wants you to be unfair in their favor, or claims to disagree about what is "fair", it will still generally be meta-fair to be fair, even if you can't make the Other agree. By the same token, if you ask if we meta-should do what we should, rather than something else, the answer is yes. Even if some other agent or optimization process does not do what is right, that doesn't change what is meta-right.

And this is not "arbitrary" in the sense of rolling dice, not "arbitrary" in the sense that justification is expected and then not found. The accusations that I level against evolution are not merely pulled from a hat; they are expressions of morality as I understand it. They are merely moral, and there is nothing mere about that.

In "Arbitrary" I finished by saying:

The upshot is that differently structured minds may well label different propositions with their analogues of the internal label "arbitrary" - though only one of these labels is what you mean when you say "arbitrary", so you and these other agents do not really have a disagreement.

This was to help shake people loose of the idea that if any two possible minds can say or do different things, then it must all be arbitrary. Different minds may have different ideas of what's "arbitrary", so clearly this whole business of "arbitrariness" is arbitrary, and we should ignore it. After all, Sinned (the anti-Dennis) just always says "Morality isn't arbitrary!" no matter how you try to persuade her otherwise, so clearly you're just being arbitrary in saying that morality is arbitrary.

From the perspective of a human, saying that one should sort pebbles into prime-numbered heaps is arbitrary - it's the sort of act you'd expect to come with a justification attached, but there isn't any justification.

From the perspective of a Pebblesorter, saying that one p-should scatter a heap of 38 pebbles into two heaps of 19 pebbles is not p-arbitrary at all - it's the most p-important thing in the world, and fully p-justified by the intuitively obvious fact that a heap of 19 pebbles is p-correct and a heap of 38 pebbles is not.

So which perspective should we adopt? I answer that I see no reason at all why I should start sorting pebble-heaps. It strikes me as a completely pointless activity. Better to engage in art, or music, or science, or heck, better to connive political plots of terrifying dark elegance, than to sort pebbles into prime-numbered heaps. A galaxy transformed into pebbles and sorted into prime-numbered heaps would be just plain boring.

The Pebblesorters, of course, would only reason that music is p-pointless because it doesn't help you sort pebbles into heaps; the human activity of humor is not only p-pointless but just plain p-bizarre and p-incomprehensible; and most of all, the human vision of a galaxy in which agents are running around experiencing positive reinforcement but not sorting any pebbles, is a vision of an utterly p-arbitrary galaxy devoid of p-purpose. The Pebblesorters would gladly sacrifice their lives to create a P-Friendly AI that sorted the galaxy on their behalf; it would be the most p-profound statement they could make about the p-meaning of their lives.

So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is right. I do not know perfectly what is right, but neither can I plead entire ignorance.

And the Pebblesorters, who simply are not built to do what is right, choose the Pebblesorting perspective: not merely because it is theirs, or because they think they can get away with being p-arbitrary, but because that is what is p-right.

And in fact, both we and the Pebblesorters can agree on all these points. We can agree that sorting pebbles into prime-numbered heaps is arbitrary and unjustified, but not p-arbitrary or p-unjustified; that it is the sort of thing an agent p-should do, but not the sort of thing an agent should do.

I fully expect that even if there is other life in the universe only a few trillions of lightyears away (I don't think it's local, or we would have seen it by now), that we humans are the only creatures for a long long way indeed who are built to do what is right. That may be a moral miracle, but it is not a causal miracle.

There may be some other evolved races, a sizable fraction perhaps, maybe even a majority, who do some right things. Our executing adaptation of compassion is not so far removed from the game theory that gave it birth; it might be a common adaptation. But laughter, I suspect, may be rarer by far than mercy. What would a galactic civilization be like, if it had sympathy, but never a moment of humor? A little more boring, perhaps, by our standards.

This humanity that we find ourselves in, is a great gift. It may not be a great p-gift, but who cares about p-gifts?

So I really must deny the charges of moral relativism: I don't think that human morality is arbitrary at all, and I would expect any logically omniscient reasoner to agree with me on that. We are better than the Pebblesorters, because we care about sentient lives, and the Pebblesorters don't. Just as the Pebblesorters are p-better than us, because they care about pebble heaps, and we don't. Human morality is p-arbitrary, but who cares? P-arbitrariness is arbitrary.

You've just got to avoid thinking that the words "better" and "p-better", or "moral" and "p-moral", are talking about the same thing - because then you might think that the Pebblesorters were coming to different conclusions than us about the same thing - and then you might be tempted to think that our own morals were arbitrary. Which, of course, they're not.

Yes, I really truly do believe that humanity is better than the Pebblesorters! I am not being sarcastic, I really do believe that. I am not playing games by redefining "good" or "arbitrary", I think I mean the same thing by those terms as everyone else. When you understand that I am genuinely sincere about that, you will understand my metaethics. I really don't consider myself a moral relativist - not even in the slightest!

Sequence: Metaethics

Is Fairness Arbitrary?

You Provably Can't Trust Yourself

Referenced by: Hot Air Doesn't Disagree

Original with comments: The Bedrock of Morality: Arbitrary?

Hot Air Doesn't Disagree

Eliezer Yudkowsky, 16 August 2008 12:42AM

Followup to: The Bedrock of Morality, Abstracted Idealized Dynamics

Tim Tyler comments:

Do the fox and the rabbit disagree? It seems reasonable so say that they do if they meet: the rabbit thinks it should be eating grass, and the fox thinks the rabbit should be in the fox's stomach. They may argue passionately about the rabbit's fate - and even stoop to violence.

Boy, you know, when you think about it, Nature turns out to be just full of disagreement.

Rocks, for example, fall down - so they agree with us, who also fall when pushed off a cliff - whereas hot air rises into the air, unlike humans.

I wonder why hot air disagrees with us so dramatically. I wonder what sort of moral justifications it might have for behaving as it does; and how long it will take to argue this out. So far, hot air has not been forthcoming in terms of moral justifications.

Physical systems that behave differently from you usually do not have factual or moral disagreements with you. Only a highly specialized subset of systems, when they do something different from you, should lead you to infer their explicit internal representation of moral arguments that could potentially lead you to change your mind about what you should do.

Attributing moral disagreements to rabbits or foxes is sheer anthropomorphism, in the full technical sense of the term - like supposing that lightning bolts are thrown by thunder gods, or that trees have spirits that can be insulted by human sexual practices and lead them to withhold their fruit.

The rabbit does not think it should be eating grass. If questioned the rabbit will not say, "I enjoy eating grass, and it is good in general for agents to do what they enjoy, therefore I should eat grass." Now you might invent an argument like that; but the rabbit's actual behavior has absolutely no causal connection to any cognitive system that processes such arguments. The fact that the rabbit eats grass, should not lead you to infer the explicit cognitive representation of, nor even infer the probable theoretical existence of, the sort of arguments that humans have over what they should do. The rabbit is just eating grass, like a rock rolls downhill and like hot air rises.

To think that the rabbit contains a little circuit that ponders morality and then finally does what it thinks it should do, and that the rabbit has arrived at the belief that it should eat grass, and that this is the explanation of why the rabbit is eating grass - from which we might infer that, if the rabbit is correct, perhaps humans should do the same thing - this is all as ridiculous as thinking that the rock wants to be at the bottom of the hill, concludes that it can reach the bottom of the hill by rolling, and therefore decides to exert a mysterious motive force on itself. Aristotle thought that, but there is a reason why Aristotelians don't teach modern physics courses.

The fox does not argue that it is smarter than the rabbit and so deserves to live at the rabbit's expense. To think that the fox is moralizing about why it should eat the rabbit, and this is why the fox eats the rabbit - from which we might infer that we as humans, hearing the fox out, would see its arguments as being in direct conflict with those of the rabbit, and we would have to judge between them - this is as ridiculous as thinking (as a modern human being) that lightning bolts are thrown by thunder gods in a state of inferrable anger.

Yes, foxes and rabbits are more complex creatures than rocks and hot air, but they do not process moral arguments. They are not that complex in that particular way.

Foxes try to eat rabbits and rabbits try to escape foxes, and from this there is nothing more to be inferred than from rocks falling and hot air rising, or water quenching fire and fire evaporating water. They are not arguing.

This anthropomorphism of presuming that every system does what it does because of a belief about what it should do, is directly responsible for the belief that Pebblesorters create prime-numbered heaps of pebbles because they think that is what everyone should do. They don't. Systems whose behavior indicates something about what agents should do, are rare, and the Pebblesorters are not such systems. They don't care about sentient life at all. They just sort pebbles into prime-numbered heaps.

Original with comments: Hot Air Doesn't Disagree

When Anthropomorphism Became Stupid

Eliezer Yudkowsky, 16 August 2008 11:43PM

Followup to: Humans in Funny Suits, Brain Breakthrough

It turns out that most things in the universe don't have minds.

This statement would have provoked incredulity among many earlier cultures. "Animism" is the usual term. They thought that trees, rocks, streams, and hills all had spirits because, hey, why not?

I mean, those lumps of flesh known as "humans" contain thoughts, so why shouldn't the lumps of wood known as "trees"?

My muscles move at my will, and water flows through a river. Who's to say that the river doesn't have a will to move the water? The river overflows its banks, and floods my tribe's gathering-place - why not think that the river was angry, since it moved its parts to hurt us? It's what we would think when someone's fist hit our nose.

There is no obvious reason - no reason obvious to a hunter-gatherer - why this cannot be so. It only seems like a stupid mistake if you confuse weirdness with stupidity. Naturally the belief that rivers have animating spirits seems "weird" to us, since it is not a belief of our tribe. But there is nothing obviously stupid about thinking that great lumps of moving water have spirits, just like our own lumps of moving flesh.

If the idea were obviously stupid, no one would have believed it. Just like, for the longest time, nobody believed in the obviously stupid idea that the Earth moves while seeming motionless.

Is it obvious that trees can't think? Trees, let us not forget, are in fact our distant cousins. Go far enough back, and you have a common ancestor with your fern. If lumps of flesh can think, why not lumps of wood?

For it to be obvious that wood doesn't think, you have to belong to a culture with microscopes. Not just any microscopes, but really good microscopes.

Aristotle thought the brain was an organ for cooling the blood. (It's a good thing that what we believe about our brains has very little effect on their actual operation.)

Egyptians threw the brain away during the process of mummification.

Alcmaeon of Croton, a Pythagorean of the 5th century BCE, put his finger on the brain as the seat of intelligence, because he'd traced the optic nerve from the eye to the brain. Still, with the amount of evidence he had, it was only a guess.

When did the central role of the brain stop being a guess? I do not know enough history to answer this question, and probably there wasn't any sharp dividing line. Maybe we could put it at the point where someone traced the anatomy of the nerves, and discovered that severing a nervous connection to the brain blocked movement and sensation?

Even so, that is only a mysterious spirit moving through the nerves. Who's to say that wood and water, even if they lack the little threads found in human anatomy, might not carry the same mysterious spirit by different means?

I've spent some time online trying to track down the exact moment when someone noticed the vastly tangled internal structure of the brain's neurons, and said, "Hey, I bet all this giant tangle is doing complex information-processing!" I haven't had much luck. (It's not Camillo Golgi - the tangledness of the circuitry was known before Golgi.) Maybe there was never a watershed moment there, either.

But that discovery of that tangledness, and Charles Darwin's theory of natural selection, and the notion of cognition as computation, is where I would put the gradual beginning of anthropomorphism's descent into being obviously wrong.

It's the point where you can look at a tree, and say: "I don't see anything in the tree's biology that's doing complex information-processing. Nor do I see it in the behavior, and if it's hidden in a way that doesn't affect the tree's behavior, how would a selection pressure for such complex information-processing arise?"

It's the point where you can look at a river, and say, "Water doesn't contain patterns replicating with distant heredity and substantial variation subject to iterative selection, so how would a river come to have any pattern so complex and functionally optimized as a brain?"

It's the point where you can look at an atom, and say: "Anger may look simple, but it's not, and there's no room for it to fit in something as simple as an atom - not unless there are whole universes of subparticles inside quarks; and even then, since we've never seen any sign of atomic anger, it wouldn't have any effect on the high-level phenomena we know."

It's the point where you can look at a puppy, and say: "The puppy's parents may push it to the ground when it does something wrong, but that doesn't mean the puppy is doing moral reasoning. Our current theories of evolutionary psychology holds that moral reasoning arose as a response to more complex social challenges than that - in their full-fledged human form, our moral adaptations are the result of selection pressures over linguistic arguments about tribal politics."

It's the point where you can look at a rock, and say, "This lacks even the simple search trees embodied in a chess-playing program - where would it get the intentions to want to roll downhill, as Aristotle once thought?"

It is written:

Zhuangzi and Huizi were strolling along the dam of the Hao Waterfall when Zhuangzi said, "See how the minnows come out and dart around where they please! That's what fish really enjoy!"

Huizi said, "You're not a fish — how do you know what fish enjoy?"

Zhuangzi said, "You're not I, so how do you know I don't know what fish enjoy?"

Now we know.

Referenced by: Excluding the Supernatural

Original with comments: When Anthropomorphism Became Stupid

The Cartoon Guide to Löb's Theorem

Eliezer Yudkowsky, 17 August 2008 08:35PM

Lo! A cartoon proof of Löb's Theorem!

Löb's Theorem shows that a mathematical system cannot assert its own soundness without becoming inconsistent. Marcello and I wanted to be able to see the truth of Löb's Theorem at a glance, so we doodled it out in the form of a cartoon. (An inability to trust assertions made by a proof system isomorphic to yourself, may be an issue for self-modifying AIs.)

It was while learning mathematical logic that I first learned to rigorously distinguish between X, the truth of X, the quotation of X, a proof of X, and a proof that X's quotation was provable.

The cartoon guide follows as an embedded Scribd document after the jump, or you can download from yudkowsky.net as a PDF file. Afterward I offer a medium-hard puzzle to test your skill at drawing logical distinctions.

Cartoon Guide to Löb's Theorem - Upload a Document to Scribd

And now for your medium-hard puzzle:

The Deduction Theorem (look it up) states that whenever assuming a hypothesis H enables us to prove a formula F in classical logic, then (H->F) is a theorem in classical logic.

Let ◻Z stand for the proposition "Z is provable". Löb's Theorem shows that, whenever we have ((◻C)->C), we can prove C.

Applying the Deduction Theorem to Löb's Theorem gives us, for all C:

((◻C)->C)->C

However, those familiar with the logic of material implication will realize that:

(X->Y)->Y
implies
(not X)->Y

Applied to the above, this yields (not ◻C)->C.

That is, all statements which lack proofs are true.

I cannot prove that 2 = 1.

Therefore 2 = 1.

Can you exactly pinpoint the flaw?

Referenced by: You Provably Can't Trust Yourself • No License To Be Human • Nonsentient Optimizers

Original with comments: The Cartoon Guide to Löb's Theorem

Dumb Deplaning

Eliezer Yudkowsky, 18 August 2008 11:49PM

So I just traveled to Portsmouth, VA for an experimental conference - in the sense that I don't expect conferences of this type to prove productive, but maybe I should try at least once - in the unlikely event that there are any local Overcoming Bias readers who want to drive out to Portsmouth for a meeting on say the evening of the 20th, email me - anyway, I am struck, for the Nth time, how uncooperative people are in getting off planes.

Most people, as soon as they have a chance to make for the exit, do so - even if they need to take down luggage first. At any given time after the initial rush to the aisles, usually a single person is taking down luggage, while the whole line behind them waits. Then the line moves forward a little and the next person starts taking down their luggage.

In programming we call this a "greedy local algorithm". But since everyone does it, no one seems to feel "greedy".

How would I do it? Off the top of my head:

"Left aisle seats, please rise and move to your luggage. (Pause.) Left aisle seats, please retrieve your luggage. (Pause.) Left aisle seats, please deplane. (Pause.) Right aisle seats, please rise and move to your luggage..."

There are numerous other minor tweaks that this suggests, like seating people with tight connections near the front left aisle, or boarding passengers with window seats before passengers with middle and aisle seats.

But the main thing that strikes me is twofold:

First, everyone who stops to take down their luggage while everyone waits behind them - as opposed to waiting to rise until the aisle is clear - is playing a negative-sum game; the benefit to themselves is smaller than the total cost to all the others waiting in line.

Second, the airline has a motive to clear passengers quickly to reduce turnaround time. But the airline does not regulate the deplaning process. Even though it would be straightforward - defectors being readily spotted - and I don't even see why it would be resented.

Am I missing something? Is there some mysterious Freakonomics-style explanation for this?

Heck, people usually manage to regulate themselves on worse cases than this. Most of the people blocking the aisle wouldn't walk away with someone else's purse. Are we just stuck in an equilibrium of mutual defection? You'd think people not in a rush would be willing to unilaterally wait until the aisle is clear before getting up, as it's an inexpensive way to purchase a chance to feel quietly virtuous.

If an essentially friendly crowd of human beings can't cooperate well enough to walk off a damned plane... now really, we should have more pride as a species than that.

Original with comments: Dumb Deplaning

You Provably Can't Trust Yourself

Eliezer Yudkowsky, 19 August 2008 08:35PM

Followup to: Where Recursive Justification Hits Bottom, Löb's Theorem

Peano Arithmetic seems pretty trustworthy. We've never found a case where Peano Arithmetic proves a theorem T, and yet T is false in the natural numbers. That is, we know of no case where []T ("T is provable in PA") and yet ~T ("not T").

We also know of no case where first order logic is invalid: We know of no case where first-order logic produces false conclusions from true premises. (Whenever first-order statements H are true of a model, and we can syntactically deduce C from H, checking C against the model shows that C is also true.)

Combining these two observations, it seems like we should be able to get away with adding a rule to Peano Arithmetic that says:

All T: ([]T -> T)

But Löb's Theorem seems to show that as soon as we do that, everything becomes provable. What went wrong? How can we do worse by adding a true premise to a trustworthy theory? Is the premise not true - does PA prove some theorems that are false? Is first-order logic not valid - does it sometimes prove false conclusions from true premises?

Actually, there's nothing wrong with reasoning from the axioms of Peano Arithmetic plus the axiom schema "Anything provable in Peano Arithmetic is true." But the result is a different system from PA, which we might call PA+1. PA+1 does not reason from identical premises to PA; something new has been added. So we can evade Löb's Theorem because PA+1 is not trusting itself - it is only trusting PA.

If you are not previously familiar with mathematical logic, you might be tempted to say, "Bah! Of course PA+1 is trusting itself! PA+1 just isn't willing to admit it! Peano Arithmetic already believes anything provable in Peano Arithmetic - it will already output anything provable in Peano Arithmetic as a theorem, by definition! How does moving to PA+1 change anything, then? PA+1 is just the same system as PA, and so by trusting PA, PA+1 is really trusting itself. Maybe that dances around some obscure mathematical problem with direct self-reference, but it doesn't evade the charge of self-trust."

But PA+1 and PA really are different systems; in PA+1 it is possible to prove true statements about the natural numbers that are not provable in PA. If you're familiar with mathematical logic, you know this is because some nonstandard models of PA are ruled out in PA+1. Otherwise you'll have to take my word that Peano Arithmetic doesn't fully describe the natural numbers, and neither does PA+1, but PA+1 characterizes the natural numbers slightly better than PA.

The deeper point is the enormous gap, the tremendous difference, between having a system just like PA except that it trusts PA, and a system just like PA except that it trusts itself.

If you have a system that trusts PA, that's no problem; we're pretty sure PA is trustworthy, so the system is reasoning from true premises. But if you have a system that looks like PA - having the standard axioms of PA - but also trusts itself, then it is trusting a self-trusting system, something for which there is no precedent. In the case of PA+1, PA+1 is trusting PA which we're pretty sure is correct. In the case of Self-PA it is trusting Self-PA, which we've never seen before - it's never been tested, despite its misleading surface similarity to PA. And indeed, Self-PA collapses via Löb's Theorem and proves everything - so I guess it shouldn't have trusted itself after all! All this isn't magic; I've got a nice Cartoon Guide to how it happens, so there's no good excuse for not understanding what goes on here.

I have spoken of the Type 1 calculator that asks "What is 2 + 3?" when the buttons "2", "+", and "3" are pressed; versus the Type 2 calculator that asks "What do I calculate when someone presses '2 + 3'?" The first calculator answers 5; the second calculator can truthfully answer anything, even 54.

But this doesn't mean that all calculators that reason about calculators are flawed. If I build a third calculator that asks "What does the first calculator answer when I press '2 + 3'?", perhaps by calculating out the individual transistors, it too will answer 5. Perhaps this new, reflective calculator will even be able to answer some questions faster, by virtue of proving that some faster calculation is isomorphic to the first calculator.

PA is the equivalent of the first calculator; PA+1 is the equivalent of the third calculator; but Self-PA is like unto the second calculator.

As soon as you start trusting yourself, you become unworthy of trust. You'll start believing any damn thing that you think, just because you thought it. This wisdom of the human condition is pleasingly analogous to a precise truth of mathematics.

Hence the saying: "Don't believe everything you think."

And the math also suggests, by analogy, how to do better: Don't trust thoughts because you think them, but because they obey specific trustworthy rules.

PA only starts believing something - metaphorically speaking - when it sees a specific proof, laid out in black and white. If you say to PA - even if you prove to PA - that PA will prove something, PA still won't believe you until it sees the actual proof. Now, this might seem to invite inefficiency, and PA+1 will believe you - if you prove that PA will prove something, because PA+1 trusts the specific, fixed framework of Peano Arithmetic; not itself.

As far as any human knows, PA does happen to be sound; which means that what PA proves is provable in PA, PA will eventually prove and will eventually believe. Likewise, anything PA+1 can prove that it proves, it will eventually prove and believe. It seems so tempting to just make PA trust itself - but then it becomes Self-PA and implodes. Isn't that odd? PA believes everything it proves, but it doesn't believe "Everything I prove is true." PA trusts a fixed framework for how to prove things, and that framework doesn't happen to talk about trust in the framework.

You can have a system that trusts the PA framework explicitly, as well as implicitly: that is PA+1. But the new framework that PA+1 uses, makes no mention of itself; and the specific proofs that PA+1 demands, make no mention of trusting PA+1, only PA. You might say that PA implicitly trusts PA, PA+1 explicitly trusts PA, and Self-PA trusts itself.

For everything that you believe, you should always find yourself able to say, "I believe because of [specific argument in framework F]", not "I believe because I believe".

Of course, this gets us into the +1 question of why you ought to trust or use framework F. Human beings, not being formal systems, are too reflective to get away with being unable to think about the problem. Got a superultimate framework U? Why trust U?

And worse: as far as I can tell, using induction is what leads me to explicitly say that induction seems to often work, and my use of Occam's Razor is implicated in my explicit endorsement of Occam's Razor. Despite my best efforts, I have been unable to prove that this is inconsistent, and I suspect it may be valid.

But it does seem that the distinction between using a framework and mentioning it, or between explicitly trusting a fixed framework F and trusting yourself, is at least important to unraveling foundational tangles - even if Löb turns out not to apply directly.

Which gets me to the reason why I'm saying all this in the middle of a sequence about morality.

I've been pondering the unexpectedly large inferential distances at work here - I thought I'd gotten all the prerequisites out of the way for explaining metaethics, but no. I'm no longer sure I'm even close. I tried to say that morality was a "computation", and that failed; I tried to explain that "computation" meant "abstracted idealized dynamic", but that didn't work either. No matter how many different ways I tried to explain it, I couldn't get across the distinction my metaethics drew between "do the right thing", "do the human thing", and "do my own thing". And it occurs to me that my own background, coming into this, may have relied on having already drawn the distinction between PA, PA+1 and Self-PA.

Coming to terms with metaethics, I am beginning to think, is all about distinguishing between levels. I first learned to do this rigorously back when I was getting to grips with mathematical logic, and discovering that you could prove complete absurdities, if you lost track even once of the distinction between "believe particular PA proofs", "believe PA is sound", and "believe you yourself are sound". If you believe any particular PA proof, that might sound pretty much the same as believing PA is sound in general; and if you use PA and only PA, then trusting PA (that is, being moved by arguments that follow it) sounds pretty much the same as believing that you yourself are sound. But after a bit of practice with the actual math - I did have to practice the actual math, not just read about it - my mind formed permanent distinct buckets and built walls around them to prevent the contents from slopping over.

Playing around with PA and its various conjugations, gave me the notion of what it meant to trust arguments within a framework that defined justification. It gave me practice keeping track of specific frameworks, and holding them distinct in my mind.

Perhaps that's why I expected to communicate more sense than I actually succeeded in doing, when I tried to describe right as a framework of justification that involved being moved by particular, specific terminal values and moral arguments; analogous to an entity who is moved by encountering a specific proof from the allowed axioms of Peano Arithmetic. As opposed to a general license to do whatever you prefer, or a morally relativistic term like "utility function" that can eat the values of any given species, or a neurological framework contingent on particular facts about the human brain. You can make good use of such concepts, but I do not identify them with the substance of what is right.

Gödelian arguments are inescapable; you can always isolate the framework-of-trusted-arguments if a mathematical system makes sense at all. Maybe the adding-up-to-normality-ness of my system will become clearer, after it becomes clear that you can always isolate the framework-of-trusted-arguments of a human having a moral argument.

Sequence: Metaethics

The Bedrock of Morality: Arbitrary?

No License To Be Human

Referenced by: No License To Be Human

Original with comments: You Provably Can't Trust Yourself

No License To Be Human

Eliezer Yudkowsky, 20 August 2008 11:18PM

Followup to: You Provably Can't Trust Yourself

Yesterday I discussed the difference between:

- A system that believes - is moved by - any specific chain of deductions from the axioms of Peano Arithmetic. (PA, Type 1 calculator)

- A system that believes PA, plus explicitly asserts the general proposition that PA is sound. (PA+1, meta-1-calculator that calculates the output of Type 1 calculator)

- A system that believes PA, plus explicitly asserts its own soundness. (Self-PA, Type 2 calculator)

These systems are formally distinct. PA+1 can prove things that PA cannot. Self-PA is inconsistent, and can prove anything via Löb's Theorem.

With these distinctions in mind, I hope my intent will be clearer, when I say that although I am human and have a human-ish moral framework, I do not think that the fact of acting in a human-ish way licenses anything.

I am a self-renormalizing moral system, but I do not think there is any general license to be a self-renormalizing moral system.

And while we're on the subject, I am an epistemologically incoherent creature, trying to modify his ways of thinking in accordance with his current conclusions; but I do not think that reflective coherence implies correctness.

Let me take these issues in reverse order, starting with the general unlicensure of epistemological reflective coherence.

If five different people go out and investigate a city, and draw five different street maps, we should expect the maps to be (mostly roughly) consistent with each other. Accurate maps are necessarily consistent among each other and among themselves, there being only one reality. But if I sit in my living room with my blinds closed, I can draw up one street map from my imagination and then make four copies: these five maps will be consistent among themselves, but not accurate. Accuracy implies consistency but not the other way around.

In Where Recursive Justification Hits Bottom, I talked about whether "I believe that induction will work on the next occasion, because it's usually worked before" is legitimate reasoning, or "I trust Occam's Razor because the simplest explanation for why Occam's Razor often works is that we live in a highly ordered universe". Though we actually formalized the idea of scientific induction, starting from an inductive instinct; we modified our intuitive understanding of Occam's Razor (Maxwell's Equations are in fact simpler than Thor, as an explanation for lightning) based on the simple idea that "the universe runs on equations, not heroic mythology". So we did not automatically and unthinkingly confirm our assumptions, but rather, used our intuitions to correct them - seeking reflective coherence.

But I also remarked:

"And what about trusting reflective coherence in general? Wouldn't most possible minds, randomly generated and allowed to settle into a state of reflective coherence, be incorrect? Ah, but we evolved by natural selection; we were not generated randomly."

So you are not, in general, safe if you reflect on yourself and achieve internal coherence. The Anti-Inductors who compute that the probability of the coin coming up heads on the next occasion, decreases each time they see the coin come up heads, may defend their anti-induction by saying: "But it's never worked before!"

The only reason why our human reflection works, is that we are good enough to make ourselves better - that we had a core instinct of induction, a core instinct of simplicity, that wasn't sophisticated or exactly right, but worked well enough.

A mind that was completely wrong to start with, would have no seed of truth from which to heal itself. (It can't forget everything and become a mind of pure emptiness that would mysteriously do induction correctly.)

So it's not that reflective coherence is licensed in general, but that it's a good idea if you start out with a core of truth or correctness or good priors. Ah, but who is deciding whether I possess good priors? I am! By reflecting on them! The inescapability of this strange loop is why a broken mind can't heal itself - because there is no jumping outside of all systems.

I can only plead that, in evolving to perform induction rather than anti-induction, in evolving a flawed but not absolutely wrong instinct for simplicity, I have been blessed with an epistemic gift.

I can only plead that self-renormalization works when I do it, even though it wouldn't work for Anti-Inductors. I can only plead that when I look over my flawed mind and see a core of useful reasoning, that I am really right, even though a completely broken mind might mistakenly perceive a core of useful truth.

Reflective coherence isn't licensed for all minds. It works for me, because I started out with an epistemic gift.

It doesn't matter if the Anti-Inductors look over themselves and decide that their anti-induction also constitutes an epistemic gift; they're wrong, I'm right.

And if that sounds philosophically indefensible, I beg you to step back from philosophy, and conside whether what I have just said is really truly true.

(Using your own concepts of induction and simplicity to do so, of course.)

Does this sound a little less indefensible, if I mention that PA trusts only proofs from the PA axioms, not proofs from every possible set of axioms? To the extent that I trust things like induction and Occam's Razor, then of course I don't trust anti-induction or anti-Occamian priors - they wouldn't start working just because I adopted them.

What I trust isn't a ghostly variable-framework from which I arbitrarily picked one possibility, so that picking any other would have worked as well so long as I renormalized it. What I trust is induction and Occam's Razor, which is why I use them to think about induction and Occam's Razor.

(Hopefully I have not just licensed myself to trust myself; only licensed being moved by both implicit and explicit appeals to induction and Occam's Razor. Hopefully this makes me PA+1, not Self-PA.)

So there is no general, epistemological license to be a self-renormalizing factual reasoning system.

The reason my system works is because it started out fairly inductive - not because of the naked meta-fact that it's trying to renormalize itself using any system; only induction counts. The license - no, the actual usefulness - comes from the inductive-ness, not from mere reflective-ness. Though I'm an inductor who says so!

And, sort-of similarly, but not exactly analogously:

There is no general moral license to be a self-renormalizing decision system. Self-consistency in your decision algorithms is not that-which-is-right.

The Pebblesorters place the entire meaning of their lives in assembling correct heaps of pebbles and scattering incorrect ones; they don't know what makes a heap correct or incorrect, but they know it when they see it. It turns out that prime heaps are correct, but determining primality is not an easy problem for their brains. Like PA and unlike PA+1, the Pebblesorters are moved by particular and specific arguments tending to show that a heap is correct or incorrect (that is, prime or composite) but they have no explicit notion of "prime heaps are correct" or even "Pebblesorting People can tell which heaps are correct or incorrect". They just know (some) correct heaps when they see them, and can try to figure out the others.

Let us suppose by way of supposition, that when the Pebblesorters are presented with the essence of their decision system - that is, the primality test - they recognize it with a great leap of relief and satisfaction. We can spin other scenarios - Peano Arithmetic, when presented with itself, does not prove itself correct. But let's suppose that the Pebblesorters recognize a wonderful method of systematically producing correct pebble heaps. Or maybe they don't endorse Adleman's test as being the essence of correctness - any more than Peano Arithmetic proves that what PA proves is true - but they do recognize that Adleman's test is a wonderful way of producing correct heaps.

Then the Pebblesorters have a reflectively coherent decision system.

But this does not constitute a disagreement between them and humans about what is right, any more than humans, in scattering a heap of 3 pebbles, are disagreeing with the Pebblesorters about which numbers are prime!

The Pebblesorters are moved by arguments like "Look at this row of 13 pebbles, and this row of 7 pebbles, arranged at right angles to each other; how can you see that, and still say that a heap of 91 pebbles is correct?"

Human beings are moved by arguments like "Hatred leads people to play purely negative-sum games, sacrificing themselves and hurting themselves to make others hurt still more" or "If there is not the threat of retaliation, carried out even when retaliation is profitless, there is no credible deterrent against those who can hurt us greatly for a small benefit to themselves".

This is not a minor difference of flavors. When you reflect on the kind of arguments involved here, you are likely to conclude that the Pebblesorters really are talking about primality, whereas the humans really are arguing about what's right. And I agree with this, since I am not a moral relativist. I don't think that morality being moral implies any ontologically basic physical rightness attribute of objects; and conversely, I don't think the lack of such a basic attribute is a reason to panic.

I may have contributed to the confusion here by labeling the Pebblesorters' decisions "p-right". But what they are talking about is not a different brand of "right". What they're talking about is prime numbers. There is no general rule that reflectively coherent decision systems are right; the Pebblesorters, in merely happening to implement a reflectively coherent decision system, are not yet talking about morality!

It's been suggested that I should have spoken of "p-right" and "h-right", not "p-right" and "right".

But of course I made a very deliberate decision not to speak of "h-right". That sounds like there is a general license to be human.

It sounds like being human is the essence of rightness. It sounds like the justification framework is "this is what humans do" and not "this is what saves lives, makes people happy, gives us control over our own lives, involves us with others and prevents us from collapsing into total self-absorption, keeps life complex and non-repeating and aesthetic and interesting, dot dot dot etcetera etcetera".

It's possible that the above value list, or your equivalent value list, may not sound like a compelling notion unto you. Perhaps you are only moved to perform particular acts that make people happy - not caring all that much yet about this general, explicit, verbal notion of "making people happy is a value". Listing out your values may not seem very valuable to you. (And I'm not even arguing with that judgment, in terms of everyday life; but a Friendly AI researcher has to know the metaethical score, and you may have to judge whether funding a Friendly AI project will make your children happy.) Which is just to say that you're behaving like PA, not PA+1.

And as for that value framework being valuable because it's human - why, it's just the other way around: humans have received a moral gift, which Pebblesorters lack, in that we started out interested in things like happiness instead of just prime pebble heaps.

Now this is not actually a case of someone reaching in from outside with a gift-wrapped box; any more than the "moral miracle" of blood-soaked natural selection producing Gandhi, is a real miracle.

It is only when you look out from within the perspective of morality, that it seems like a great wonder that natural selection could produce true friendship. And it is only when you look out from within the perspective of morality, that it seems like a great blessing that there are humans around to colonize the galaxies and do something interesting with them. From a purely causal perspective, nothing unlawful has happened.

But from a moral perspective, the wonder is that there are these human brains around that happen to want to help each other - a great wonder indeed, since human brains don't define rightness, any more than natural selection defines rightness.

And that's why I object to the term "h-right". I am not trying to do what's human. I am not even trying to do what is reflectively coherent for me. I am trying to do what's right.

It may be that humans argue about what's right, and Pebblesorters do what's prime. But this doesn't change what's right, and it doesn't make what's right vary from planet to planet, and it doesn't mean that the things we do are right in mere virtue of our deciding on them - any more than Pebblesorters make a heap prime or not prime by deciding that it's "correct".

The Pebblesorters aren't trying to do what's p-prime any more than humans are trying to do what's h-prime. The Pebblesorters are trying to do what's prime. And the humans are arguing about, and occasionally even really trying to do, what's right.

The Pebblesorters are not trying to create heaps of the sort that a Pebblesorter would create (note circularity). The Pebblesorters don't think that Pebblesorting thoughts have a special and supernatural influence on whether heaps are prime. The Pebblesorters aren't trying to do anything explicitly related to Pebblesorters - just like PA isn't trying to prove anything explicitly related to proof. PA just talks about numbers; it took a special and additional effort to encode any notions of proof in PA, to make PA talk about itself.

PA doesn't ask explicitly whether a theorem is provable in PA, before accepting it - indeed PA wouldn't care if it did prove that an encoded theorem was provable in PA. Pebblesorters don't care what's p-prime, just what's prime. And I don't give a damn about this "h-rightness" stuff: there's no license to be human, and it doesn't justify anything.

Sequence: Metaethics

You Provably Can't Trust Yourself

Invisible Frameworks

Referenced by: Invisible Frameworks • (Moral) Truth in Fiction?

Original with comments: No License To Be Human

Invisible Frameworks

Eliezer Yudkowsky, 22 August 2008 03:36AM

Followup to: Passing the Recursive Buck, No License To Be Human

Roko has mentioned his "Universal Instrumental Values" several times in his comments. Roughly, Roko proposes that we ought to adopt as terminal values those things that a supermajority of agents would do instrumentally. On Roko's blog he writes:

I'm suggesting that UIV provides the cornerstone for a rather new approach to goal system design. Instead of having a fixed utility function/supergoal, you periodically promote certain instrumental values to terminal values i.e. you promote the UIVs.

Roko thinks his morality is more objective than mine:

It also worries me quite a lot that eliezer's post is entirely symmetric under the action of replacing his chosen notions with the pebble-sorter's notions. This property qualifies as "moral relativism" in my book, though there is no point in arguing about the meanings of words.

My posts on universal instrumental values are not symmetric under replacing UIVs with some other set of goals that an agent might have. UIVs are the unique set of values X such that in order to achieve any other value Y, you first have to do X.

Well, and this proposal has a number of problems, as some of the commenters on Roko's blog point out.

For a start, Roko actually says "universal", not "supermajority", but there are no actual universal actions; no matter what the green button does, there are possible mind designs whose utility function just says "Don't press the green button." There is no button, in other words, that all possible minds will press. Still, if you defined some prior weighting over the space of possible minds, you could probably find buttons that a supermajority would press, like the "Give me free energy" button.

But to do nothing except press such buttons, consists of constantly losing your purposes. You find that driving the car is useful for getting and eating chocolate, or for attending dinner parties, or even for buying and manufacturing more cars. In fact, you realize that every intelligent agent will find it useful to travel places. So you start driving the car around without any destination. Roko hasn't noticed this because, by anthropomorphic optimism, he mysteriously only thinks of humanly appealing "UIVs" to propose, like "creativity".

Let me guess, Roko, you don't think that "drive a car!" is a "valid" UIV for some reason? But you did not apply some fixed procedure you had previously written down, to decide whether "drive a car" was a valid UIV or not. Rather you started out feeling a moment of initial discomfort, and then looked for reasons to disapprove. I wonder why the same discomfort didn't occur to you when you considered "creativity".

But let us leave aside the universality, appeal, or well-specified-ness of Roko's metaethics.

Let us consider only Roko's claim that his morality is more objective than, say, mine, or this marvelous list by William Frankena that Roko quotes SEP quoting:

Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc.

So! Roko prefers his Universal Instrumental Values to this, because:

It also worries me quite a lot that eliezer's post is entirely symmetric under the action of replacing his chosen notions with the pebble-sorter's notions. This property qualifies as "moral relativism" in my book, though there is no point in arguing about the meanings of words.

My posts on universal instrumental values are not symmetric under replacing UIVs with some other set of goals that an agent might have. UIVs are the unique set of values X such that in order to achieve any other value Y, you first have to do X.

It would seem, then, that Roko attaches tremendous importance to claims to asymmetry and uniqueness; and tremendous disaffect to symmetry and relativism.

Which is to say that, when it comes to metamoral arguments, Roko is greatly moved to adopt morals by the statement "this goal is universal", while greatly moved to reject morals by the statement "this goal is relative".

In fact, so strong is this tendency of Roko's, that the metamoral argument "Many agents will do X!" is sufficient for Roko to adopt X as a terminal value. Indeed, Roko thinks that we ought to get all our terminal values this way.

Is this objective?

Yes and no.

When you evaluate the question "How many agents do X?", the answer does not depend on which agent evaluates it. It does depend on quantities like your weighting over all possible agents, and on the particular way you slice up possible events into categories like "X". But let us be charitable: if you adopt a fixed weighting over agents and a fixed set of category boundaries, the question "How many agents do X?" has a unique answer. In this sense, Roko's meta-utility function is objective.

But of course Roko's meta-utility function is not "objective" in the sense of universal compellingness. It is only Roko who finds the argument "Most agents do X instrumentally" a compelling reason to promote X to a terminal value. I don't find it compelling; it looks to me like losing purpose and double-counting expected utilities. The vast majority of possible agents, in fact, will not find it a compelling argument! A paperclip maximizer perceives no utility-function-changing, metamoral valence in the proposition "Most agents will find it useful to travel from one place to another."

Now this seems like an extremely obvious criticism of Roko's theory. Why wouldn't Roko have thought of it?

Because when Roko feels like he's being objective, he's using his meta-morality as a fixed given - evaluating the question "How many agents do X?" in different places and times, but not asking any different questions. The answer to his meta-moral question has occurred to him as a variable to be investigated; the meta-moral question itself is off the table.

But - of course - when a Pebblesorter regards "13 and 7!" as a powerful metamoral argument that "heaps of 91 pebbles" should not be a positive value in their utility function, they are asking a question whose answer is the same in all times and all places. They are asking whether 91 is prime or composite. A Pebblesorter, perhaps, would feel the same powerful surge of objectivity that Roko feels when Roko asks the question "How many agents have this instrumental value?" But in this case it readily occurs to Roko to ask "Why care if the heap is prime or not?" As it does not occur to Roko to ask, "Why care if this instrumental goal is universal or not?" Why... isn't it just obvious that it matters whether an instrumental goal is universal?

The Pebblesorter's framework is readily visible to Roko, since it differs from his own. But when Roko asks his own question - "Is this goal universally instrumental?" - he sees only the answer, and not the question; he sees only the output as a potential variable, not the framework.

Like PA, that only sees the compellingness of particular proofs that use the Peano Axioms, and does not consider the quoted Peano Axioms as subject matter. It is only PA+1 that sees the framework of PA.

But there is always a framework, every time you are moved to change your morals - the question is whether it will be invisible to you or not. That framework is always implemented in some particular brain, so that the same argument would fail to compel a differently constructed brain - though this does not imply that the framework makes any mention of brains at all.

And this difficulty of the invisible framework is at work, every time someone says, "But of course the correct morality is just the one that helps you survive / the one that helps you be happy" - implicit there is a supposed framework of meta-moral arguments that move you. But maybe I don't think that being happy is the one and only argument that matters.

Roko is adopting a special and unusual metamoral framework in regarding "Most agents do X!" as a compelling reason to change one's utility function. Why might Roko find this appealing? Humans, for very understandable reasons of evolutionary psychology, have a universalizing instinct; we think that a valid argument should persuade anyone.

But what happens if we confess that such thinking can be valid? What happens if we confess that a meta-moral argument can (in its invisible framework) use the universalizing instinct? Then we have... just done something very human. We haven't explicitly adopted the rule that all human instincts are good because they are human - but we did use one human instinct to think about morality. We didn't explicitly think that's what we were doing, any more than PA quotes itself in every proof; but we felt that a universally instrumental goal had this appealing quality of objective-ness about that, which is a perception of an intuition that evolved. This doesn't mean that objective-ness is subjective. If you define objectiveness precisely then the question "What is objective?" will have a unique answer. But it does mean that we have just been compelled by an argument that will not compel every possible mind.

If it's okay to be compelled by the appealing objectiveness of a moral, then why not also be compelled by...

...life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom...

Such values, if precisely defined, can be just as objective as the question "How many agents do X?" in the sense that "How much health is in this region here?" will have a single unique answer. But it is humans who care about health, just as it is humans who care about universalizability.

The framework by which we care about health and happiness, as much evolved, and human, and part of the very substance of that which we name right whether it is human or not... as our tendency to find universalizable morals appealing.

And every sort of thing that a mind can do will have some framework behind it. Every sort of argument that can compel one mind, will fail to be an argument in the framework of another.

We are in the framework we name right; and every time we try to do what is correct, what we should, what we must, what we ought, that is the question we are asking.

Which question should we ask? What is the correct question?

Don't let your framework to those questions be invisible! Don't think you've answered them without asking any questions!

There is always the meta-meta-meta-question and it always has a framework.

I, for one, have decided to answer such questions the right way, as the alternative is to answer it the wrong way, like Roko is doing.

And the Pebblesorters do not disagree with any of this; they do what is objectively prime, not what is objectively right. And the Roko-AI does what is objectively often-instrumental, flying starships around with no destination; I don't disagree that travel is often-instrumental, I just say it is not right.

There is no right-ness that isn't in any framework - no feeling of rightness, no internal label that your brain produces, that can be detached from any method whatsoever of computing it - that just isn't what we're talking about when we ask "What should I do now?" Because if anything labeled should, is right, then that is Self-PA.

Sequence: Metaethics

No License To Be Human

Referenced by: Mirrors and Paintings • Building Something Smarter • The Uses of Fun (Theory) • Value is Fragile

Original with comments: Invisible Frameworks

Mirrors and Paintings

Eliezer Yudkowsky, 23 August 2008 12:29AM

Followup to: Sorting Pebbles Into Correct Heaps, Invisible Frameworks

Background: There's a proposal for Friendly AI called "Coherent Extrapolated Volition" which I don't really want to divert the discussion to, right now. Among many other things, CEV involves pointing an AI at humans and saying (in effect) "See that? That's where you find the base content for self-renormalizing morality."

Hal Finney commented on the Pebblesorter parable:

I wonder what the Pebblesorter AI would do if successfully programmed to implement [CEV]... Would the AI pebblesort? Or would it figure that if the Pebblesorters got smarter, they would see that pebblesorting was pointless and arbitrary? Would the AI therefore adopt our own parochial morality, forbidding murder, theft and sexual intercourse among too-young people? Would that be the CEV of Pebblesorters?

I imagine we would all like to think so, but it smacks of parochialism, of objective morality. I can't help thinking that Pebblesorter CEV would have to include some aspect of sorting pebbles. Doesn't that suggest that CEV can malfunction pretty badly?

I'm giving this question its own post, for that it touches on similar questions I once pondered - dilemmas that forced my current metaethics as the resolution.

Yes indeed: A CEV-type AI, taking Pebblesorters as its focus, would wipe out the Pebblesorters and sort the universe into prime-numbered heaps.

This is not the right thing to do.

That is not a bug.

A primary motivation for CEV was to answer the question, "What can Archimedes do if he has to program a Friendly AI, despite being a savage barbarian by the Future's standards, so that the Future comes out right anyway? Then whatever general strategy Archimedes could plausibly follow, that is what we should do ourselves: For we too may be ignorant fools, as the Future measures such things."

It is tempting to further extend the question, to ask, "What can the Pebblesorters do, despite wanting only to sort pebbles, so that the universe comes out right anyway? What sort of general strategy should they follow, so that despite wanting something that is utterly pointless and futile, their Future ends up containing sentient beings leading worthwhile lives and having fun? Then whatever general strategy we wish the Pebblesorters to follow, that is what we should do ourselves: For we, too, may be flawed."

You can probably see in an intuitive sense why that won't work. We did in fact get here from the Greek era, which shows that the seeds of our era were in some sense present then - albeit this history doesn't show that no extra information was added, that there were no contingent moral accidents that sent us into one attractor rather than another. But still, if Archimedes said something along the lines of "imagine probable future civilizations that would come into existence", the AI would visualize an abstracted form of our civilization among them - though perhaps not only our civilization.

The Pebblesorters, by construction, do not contain any seed that might grow into a civilization valuing life, health, happiness, etc. Such wishes are nowhere present in their psychology. All they want is to sort pebble heaps. They don't want an AI that keeps them alive, they want an AI that can create correct pebble heaps rather than incorrect pebble heaps. They are much disturbed by the question of how such an AI can be created, when different civilizations are still arguing about heap sizes - though most of them believe that any sufficiently smart mind will see which heaps are correct and incorrect, and act accordingly.

You can't get here from there. Not by any general strategy. If you want the Pebblesorters' future to come out humane, rather than Pebblish, you can't advise the Pebblesorters to build an AI that would do what their future civilizations would do. You can't advise them to build an AI that would do what Pebblesorters would do if they knew everything the AI knew. You can't advise them to build an AI more like Pebblesorters wish they were, and less like what Pebblesorters are. All those AIs just sort the universe into prime heaps. The Pebblesorters would celebrate that and say "Mission accomplished!" if they weren't dead, but it isn't what you want the universe to be like. (And it isn't right, either.)

What kind of AI would the Pebblesorters have to execute, in order to make the universe a better place?

They'd have to execute an AI did not do what Pebblesorters would-want, but an AI that simply, directly, did what was right - an AI that cared directly about things like life, health, and happiness.

But where would that AI come from?

If you were physically present on the scene, you could program that AI. If you could send the Pebblesorters a radio message, you could tell them to program it - though you'd have to lie to them about what the AI did.

But if there's no such direct connection, then it requires a causal miracle for the Pebblesorters' AI to do what is right - a perpetual motion morality, with information appearing from nowhere. If you write out a specification of an AI that does what is right, it takes a certain number of bits; it has a Kolmogorov complexity. Where is that information appearing from, since it is not yet physically present in the Pebblesorters' Solar System? What is the cause already present in the Pebble System, of which the right-doing AI is an eventual effect? If the right-AI is written by a meta-right AI then where does the meta-right AI come from, causally speaking?

Be ye wary to distinguish between yonder levels. It may seem to you that you ought to be able to deduce the correct answer just by thinking about it - surely, anyone can see that pebbles are pointless - but that's a correct answer to the question "What is right?", which carries its own invisible framework of arguments that it is right to be moved by. This framework, though harder to see than arguments, has its physical conjugate in the human brain. The framework does not mention the human brain, so we are not persuaded by the argument "That's what the human brain says!" But this very event of non-persuasion takes place within a human brain that physically represents a moral framework that doesn't mention the brain.

This framework is not physically represented anywhere in the Pebble System. It's not a different framework in the Pebble System, any more than different numbers are prime here than there. So far as idealized abstract dynamics are concerned, the same thing is right in the Pebble System as right here. But that idealized abstract framework is not physically embodied anywhere in the Pebble System. If no human sends a physical message to the Pebble System, then how does anything right just happen to happen there, given that the right outcome is a very small target in the space of all possible outcomes? It would take a thermodynamic miracle.

As for humans doing what's right - that's a moral miracle but not a causal miracle. On a moral level, it's astounding indeed that creatures of mere flesh and goo, created by blood-soaked natural selection, should decide to try and transform the universe into a place of light and beauty. On a moral level, it's just amazing that the brain does what is right, even though "The human brain says so!" isn't a valid moral argument. On a causal level... once you understand how morality fits into a natural universe, it's not really all that surprising.

And if that disturbs you, if it seems to smack of relativism - just remember, your universalizing instinct, the appeal of objectivity, and your distrust of the state of human brains as an argument for anything, are also all implemented in your brain. If you're going to care about whether morals are universally persuasive, you may as well care about people being happy; a paperclip maximizer is moved by neither argument. See also Changing Your Metaethics.

It follows from all this, by the way, that the algorithm for CEV (the Coherent Extrapolated Volition formulation of Friendly AI) is not the substance of what's right. If it were, then executing CEV anywhere, at any time, would do what was right - even with the Pebblesorters as its focus. There would be no need to elaborately argue this, to have CEV on the left-hand-side and rightness on the r.h.s.; the two would be identical, or bear the same relation as PA+1 and PA.

So why build CEV? Why not just build a do-what's-right AI?

Because we don't know the complete list of our own terminal values; we don't know the full space of arguments we can be moved by. Human values are too complicated to program by hand. We might not recognize the source code of a do-what's-right AI, any more than we would recognize a printout of our own neuronal circuitry if we saw it. Sort of like how Peano Arithmetic doesn't recognize itself in a mirror. If I listed out all your values as mere English words on paper, you might not be all that moved by the list: is it more uplifting to see sunlight glittering off water, or to read the word "beauty"?

But in this art of Friendly AI, understanding metaethics on a naturalistic level, we can guess that our morals and metamorals will be physically represented in our brains, even though our morality (considered as an idealized abstracted dynamic) doesn't attach any explicit moral force to "Because a brain said so."

So when we try to make an AI whose physical consequence is the implementation of what is right, we make that AI's causal chain start with the state of human brains - perhaps nondestructively scanned on the neural level by nanotechnology, or perhaps merely inferred with superhuman precision from external behavior - but not passed through the noisy, blurry, destructive filter of human beings trying to guess their own morals.

The AI can't start out with a direct representation of rightness, because the programmers don't know their own values (not to mention that there are other human beings out there than the programmers, if the programmers care about that). The programmers can neither brain-scan themselves and decode the scan, nor superhumanly precisely deduce their internal generators from their outward behavior.

So you build the AI with a kind of forward reference: "You see those humans over there? That's where your utility function is."

As previously mentioned, there are tricky aspects to this. You can't say: "You see those humans over there? Whatever desire is represented in their brains, is therefore right." This, from a moral perspective, is wrong - wanting something doesn't make it right - and the conjugate failure of the AI is that it will reprogram your brains to want things that are easily obtained in great quantity. If the humans are PA, then we want the AI to be PA+1, not Self-PA... metaphorically speaking.

You've got to say something along the lines of, "You see those humans over there? Their brains contain the evidence you will use to deduce the correct utility function, even though right-ness is not caused by those brains, so that intervening to alter the brains won't alter the correct utility function." Here, the "correct" in "correct utility function" is relative to a meta-utility framework that points to the humans and defines how their brains are to be treated as information. I haven't worked out exactly how to do this, but it does look solvable.

And as for why you can't have an AI that rejects the "pointless" parts of a goal system and only keeps the "wise" parts - so that even in the Pebble System the AI rejects pebble-sorting and keeps the Pebblesorters safe and warm - it's the problem of the invisible framework again; you've only passed the recursive buck. Humans contain the physical representations of the framework that we appeal to, when we ask whether a goal is pointless or wise. Without sending a message to the Pebble System, the information there cannot physically materialize from nowhere as to which goals are pointless or wise. This doesn't mean that different goals are pointless in the Pebble System, it means that no physical brain there is asking that question.

The upshot is that structurally similar CEV algorithms will behave differently depending on whether they have humans at the focus, or Pebblesorters. You can infer that CEV will do what's right in the presence of humans, but the general algorithm in CEV is not the direct substance of what's right. There is no moral imperative to execute CEVs regardless of their focus, on any planet. It is only right to execute CEVs on decision systems that contain the seeds of rightness, such as humans. (Again, see the concept of a moral miracle that is not a causal surprise.)

Think of a Friendly AI as being like a finely polished mirror, which reflects an image more accurately than any painting drawn with blurred eyes and shaky hand. If you need an image that has the shape of an apple, you would do better to put an actual apple in front of the mirror, and not try to paint the apple by hand. Even though the drawing would inherently be apple-shaped, it wouldn't be a good one; and even though the mirror is not inherently apple-shaped, in the presence of an actual apple it is a better picture than any painting could be.

"Why not just use an actual apple?" you ask. Well, maybe this isn't a merely accurate mirror; it has an internal camera system that lightens the apple's image before displaying it. An actual apple would have the right starting shape, but it wouldn't be bright enough.

You may also want a composite image of a lot of apples that have multiple possible reflective equilibria.

As for how the apple ended up apple-shaped, when the substance of the apple doesn't define apple-shaped-ness - in the very important sense that squishing the apple won't change what's apple-shaped - well, it wasn't a miracle, but it involves a strange loop through the invisible background framework.

And if the whole affair doesn't sound all that right... well... human beings were using numbers a long time before they invented Peano Arithmetic. You've got to be almost as smart as a human to recognize yourself in a mirror, and you've got to be smarter than human to recognize a printout of your own neural circuitry. This Friendly AI stuff is somewhere in between. Would the rightness be easier to recognize if, in the end, no one died of Alzheimer's ever again?

Referenced by: The Thing That I Protect

Original with comments: Mirrors and Paintings

Unnatural Categories

Eliezer Yudkowsky, 24 August 2008 01:00AM

Followup to: Disguised Queries, Superexponential Conceptspace

If a tree falls in the forest, and no one hears it, does it make a sound?

"Tell me why you want to know," says the rationalist, "and I'll tell you the answer." If you want to know whether your seismograph, located nearby, will register an acoustic wave, then the experimental prediction is "Yes"; so, for seismographic purposes, the tree should be considered to make a sound. If instead you're asking some question about firing patterns in a human auditory cortex - for whatever reason - then the answer is that no such patterns will be changed when the tree falls.

What is a poison? Hemlock is a "poison"; so is cyanide; so is viper venom. Carrots, water, and oxygen are "not poison". But what determines this classification? You would be hard pressed, just by looking at hemlock and cyanide and carrots and water, to tell what sort of difference is at work. You would have to administer the substances to a human - preferably one signed up for cryonics - and see which ones proved fatal. (And at that, the definition is still subtler than it appears: a ton of carrots, dropped on someone's head, will also prove fatal. You're really asking about fatality from metabolic disruption, after administering doses small enough to avoid mechanical damage and blockage, at room temperature, at low velocity.)

Where poison-ness is concerned, you are not classifying via a strictly local property of the substance. You are asking about the consequence when a dose of that substance is applied to a human metabolism. The local difference between a human who gasps and keels over, versus a human alive and healthy, is more compactly discriminated, than any local difference between poison and non-poison.

So we have a substance X, that might or might not be fatally poisonous, and a human Y, and we say - to first order:

"X is classified 'fatally poisonous' iff administering X to Y causes Y to enter a state classified 'dead'."

Much of the way that we classify things - never mind events - is non-local, entwined with the consequential structure of the world. All the things we would call a chair are all the things that were made for us to sit on. (Humans might even call two molecularly identical objects a "chair" or "a rock shaped like a chair" depending on whether someone had carved it.)

"That's okay," you say, "the difference between living humans and dead humans is a nice local property - a compact cluster in Thingspace. Sure, the set of 'poisons' might not be as compact a structure. A category X|X->Y may not be as simple as Y, if the causal link -> can be complicated. Here, 'poison' is not locally compact because of all the complex ways that substances act on the complex human body. But there's still nothing unnatural about the category of 'poison' - we constructed it in an observable, testable way from categories themselves simple. If you ever want to know whether something should be called 'poisonous', or not, there's a simple experimental test that settles the issue."

Hm. What about a purple, egg-shaped, furred, flexible, opaque object? Is it a blegg, and if so, would you call "bleggs" a natural category?

"Sure," you reply, "because you are forced to formulate the 'blegg' category, or something closely akin to it, in order to predict your future experiences as accurately as possible. If you see something that's purple and egg-shaped and opaque, the only way to predict that it will be flexible is to draw some kind of compact boundary in Thingspace and use that to perform induction. No category means no induction - you can't see that this object is similar to other objects you've seen before, so you can't predict its unknown properties from its known properties. Can't get much more natural than that! Say, what exactly would an unnatural property be, anyway?"

Suppose I have a poison P1 that completely destroys one of your kidneys - causes it to just wither away. This is a very dangerous poison, but is it a fatal poison?

"No," you reply, "a human can live on just one kidney."

Suppose I have a poison P2 that completely destroys much of a human brain, killing off nearly all the neurons, leaving only enough medullary structure to run the body and keep it breathing, so long as a hospital provides nutrition. Is P2 a fatal poison?

"Yes," you say, "if your brain is destroyed, you're dead."

But this distinction that you now make, between P2 being a fatal poison and P1 being an only dangerous poison, is not driven by any fundamental requirement of induction. Both poisons destroy organs. It's just that you care a lot more about the brain, than about a kidney. The distinction you drew isn't driven solely by a desire to predict experience - it's driven by a distinction built into your utility function. If you have to choose between a dangerous poison and a lethal poison, you will of course take the dangerous poison. From which you induce that if you must choose between P1 and P2, you'll take P1.

The classification that you drew between "lethal" and "nonlethal" poisons, was designed to help you navigate the future - navigate away from outcomes of low utility, toward outcomes of high utility. The boundaries that you drew, in Thingspace and Eventspace, were not driven solely by the structure of the environment - they were also driven by the structure of your utility function; high-utility things and low-utility things lumped together. That way you can easily choose actions that lead, in general, to outcomes of high utility, over actions that lead to outcomes of low utility. If you must pick your poison and can only pick one categorical dimension to sort by, you're going to want to sort the poisons into lower and higher utility - into fatal and dangerous, or dangerous and safe. Whether the poison is red or green is a much more local property, more compact in Thingspace; but it isn't nearly as relevant to your decision-making.

Suppose you have a poison that puts a human, let's call her Terry, into an extremely damaged state. Her cerebral cortex has turned to mostly fluid, say. So I already labeled that substance a poison; but is it a lethal poison?

This would seem to depend on whether Terry is dead or alive. Her body is breathing, certainly - but her brain is damaged. In the extreme case where her brain was actually removed and incinerated, but her body kept alive, we would certainly have to say that the resultant was no longer a person, from which it follows that the previously existing person, Terry, must have died. But here we have an intermediate case, where the brain is very severely damaged but not utterly destroyed. Where does that poison fall on the border between lethality and unlethality? Where does Terry fall on the border between personhood and nonpersonhood? Did the poison kill Terry or just damage her?

Some things are persons and some things are not persons. It is murder to kill a person who has not threatened to kill you first. If you shoot a chimpanzee who isn't threatening you, is that murder? How about if you turn off Terry's life support - is that murder?

"Well," you say, "that's fundamentally a moral question - no simple experimental test will settle the issue unless we can agree in advance on which facts are the morally relevant ones. It's futile to say 'This chimp can recognize himself in a mirror!' or 'Terry can't recognize herself in a mirror!' unless we're agreed that this is a relevant fact - never mind it being the only relevant fact."

I've chosen the phrase "unnatural category" to describe a category whose boundary you draw in a way that sensitively depends on the exact values built into your utility function. The most unnatural categories are typically these values themselves! What is "true happiness"? This is entirely a moral question, because what it really means is "What is valuable happiness?" or "What is the most valuable kind of happiness?" Is having your pleasure center permanently stimulated by electrodes, "true happiness"? Your answer to that will tend to center on whether you think this kind of pleasure is a good thing. "Happiness", then, is a highly unnatural category - there are things that locally bear a strong resemblance to "happiness", but which are excluded because we judge them as being of low utility, and "happiness" is supposed to be of high utility.

Most terminal values turn out to be unnatural categories, sooner or later. This is why it's such a tremendous difficulty to decide whether turning off Terry Schiavo's life support is "murder".

I don't mean to imply that unnatural categories are worthless or relative or whatever. That's what moral arguments are for - for drawing and redrawing the boundaries; which, when it happens with a terminal value, clarifies and thereby changes our utility function.

I have a twofold motivation for introducing the concept of an "unnatural category".

The first motivation is to recognize when someone tries to pull a fast one during a moral argument, by insisting that no moral argument exists: Terry Schiavo simply is a person because she has human DNA, or she simply is not a person because her cerebral cortex has eroded. There is a super-exponential space of possible concepts, possible boundaries that can be drawn in Thingspace. When we have a predictive question at hand, like "What happens if we run a DNA test on Terry Schiavo?" or "What happens if we ask Terry Schiavo to solve a math problem?", then we have a clear criterion of which boundary to draw and whether it worked. But when the question at hand is a moral one, a "What should I do?" question, then it's time to shut your eyes and start doing moral philosophy. Or eyes open, if there are relevant facts at hand - you do want to know what Terry Schiavo's brain looks like - but the point is that you're not going to find an experimental test that settles the question, unless you've already decided where to draw the boundaries of your utility function's values.

I think that a major cause of moral panic among Luddites in the presence of high technology, is that technology tends to present us with boundary cases on our moral values - raising moral questions that were never previously encountered. In the old days, Terry Schiavo would have stopped breathing long since. But I find it difficult to blame this on technology - it seems to me that there's something wrong with going into a panic just because you're being asked a new moral question. Couldn't you just be asked the same moral question at any time?

If you want to say, "I don't know, so I'll strategize conservatively to avoid the boundary case, or treat uncertain people as people," that's one argument.

But to say, "AAAIIIEEEE TECHNOLOGY ASKED ME A QUESTION I DON'T KNOW HOW TO ANSWER, TECHNOLOGY IS UNDERMINING MY MORALITY" strikes me as putting the blame in the wrong place.

I should be able to ask you anything, even if you can't answer. If you can't answer, then I'm not undermining your morality - it was already undermined.

My second motivation... is to start explaining another reason why Friendly AI is difficult.

I was recently trying to explain to someone why, even if all you wanted to do was fill the universe with paperclips, building a paperclip maximizer would still be a hard problem of FAI theory. Why? Because if you cared about paperclips for their own sake, then you wouldn't want the AI to fill the universe with things that weren't really paperclips - as you draw that boundary!

For a human, "paperclip" is a reasonably natural category; it looks like this-and-such and we use it to hold papers together. The "papers" themselves play no direct role in our moral values; we just use them to renew the license plates on our car, or whatever. "Paperclip", in other words, is far enough away from human terminal values, that we tend to draw the boundary using tests that are relatively empirical and observable. If you present us with some strange thing that might or might not be a paperclip, we'll just see if we can use it to hold papers together. If you present us with some strange thing that might or might not be paper, we'll see if we can write on it. Relatively simple observable tests.

But there isn't any equally simple experimental test the AI can perform to find out what is or isn't a "paperclip", if "paperclip" is a concept whose importance stems from it playing a direct role in the utility function.

Let's say that you're trying to make your little baby paperclip maximizer in the obvious way: showing it a bunch of things that are paperclips, and a bunch of things that aren't paperclips, including what you consider to be near misses like staples and gluesticks. The AI formulates an internal concept that describes paperclips, and you test it on some more things, and it seems to discriminate the same way you do. So you hook up the "paperclip" concept to the utility function, and off you go!

Soon the AI grows up, kills off you and your species, and begins its quest to transform the universe into paperclips. But wait - now the AI is considering new potential boundary cases of "paperclip" that it didn't see during its training phase. Boundary cases, in fact, that you never mentioned - let alone showed the AI - because it didn't occur to you that they were possible. Suppose, for example, that the thought of tiny molecular paperclips had never occurred to you. If it had, you would have agonized for a while - like the way that people agonized over Terry Schiavo - and then finally decided that the tiny molecular paperclip-shapes were not "real" paperclips. But the thought never occurred to you, and you never showed the AI paperclip-shapes of different sizes and told the AI that only one size was correct, during its training phase. So the AI fills the universe with tiny molecular paperclips - but those aren't real paperclips at all! Alas! There's no simple experimental test that the AI can perform to find out what you would have decided was or was not a high-utility papercliplike object.

What? No simple test? What about: "Ask me what is or isn't a paperclip, and see if I say 'Yes'. That's your new meta-utility function!"

You perceive, I hope, why it isn't so easy.

If not, here's a hint:

"Ask", "me", and "say 'Yes'".

Referenced by: Magical Categories • Qualitative Strategies of Friendliness • Dreams of Friendliness • In Praise of Boredom

Original with comments: Unnatural Categories

Magical Categories

Eliezer Yudkowsky, 24 August 2008 07:51PM

Followup to: Anthropomorphic Optimism, Superexponential Conceptspace, The Hidden Complexity of Wishes, Unnatural Categories

'We can design intelligent machines so their primary, innate emotion is unconditional love for all humans. First we can build relatively simple machines that learn to recognize happiness and unhappiness in human facial expressions, human voices and human body language. Then we can hard-wire the result of this learning as the innate emotional values of more complex intelligent machines, positively reinforced when we are happy and negatively reinforced when we are unhappy.'
-- Bill Hibbard (2001), Super-intelligent machines.

That was published in a peer-reviewed journal, and the author later wrote a whole book about it, so this is not a strawman position I'm discussing here.

So... um... what could possibly go wrong...

When I mentioned (sec. 6) that Hibbard's AI ends up tiling the galaxy with tiny molecular smiley-faces, Hibbard wrote an indignant reply saying:

'When it is feasible to build a super-intelligence, it will be feasible to build hard-wired recognition of "human facial expressions, human voices and human body language" (to use the words of mine that you quote) that exceed the recognition accuracy of current humans such as you and me, and will certainly not be fooled by "tiny molecular pictures of smiley-faces." You should not assume such a poor implementation of my idea that it cannot make discriminations that are trivial to current humans.'

As Hibbard also wrote "Such obvious contradictory assumptions show Yudkowsky's preference for drama over reason," I'll go ahead and mention that Hibbard illustrates a key point: There is no professional certification test you have to take before you are allowed to talk about AI morality. But that is not my primary topic today. Though it is a crucial point about the state of the gameboard, that most AGI/FAI wannabes are so utterly unsuited to the task, that I know no one cynical enough to imagine the horror without seeing it firsthand. Even Michael Vassar was probably surprised his first time through.

No, today I am here to dissect "You should not assume such a poor implementation of my idea that it cannot make discriminations that are trivial to current humans."

Once upon a time - I've seen this story in several versions and several places, sometimes cited as fact, but I've never tracked down an original source - once upon a time, I say, the US Army wanted to use neural networks to automatically detect camouflaged enemy tanks.

The researchers trained a neural net on 50 photos of camouflaged tanks amid trees, and 50 photos of trees without tanks. Using standard techniques for supervised learning, the researchers trained the neural network to a weighting that correctly loaded the training set - output "yes" for the 50 photos of camouflaged tanks, and output "no" for the 50 photos of forest.

Now this did not prove, or even imply, that new examples would be classified correctly. The neural network might have "learned" 100 special cases that wouldn't generalize to new problems. Not, "camouflaged tanks versus forest", but just, "photo-1 positive, photo-2 negative, photo-3 negative, photo-4 positive..."

But wisely, the researchers had originally taken 200 photos, 100 photos of tanks and 100 photos of trees, and had used only half in the training set. The researchers ran the neural network on the remaining 100 photos, and without further training the neural network classified all remaining photos correctly. Success confirmed!

The researchers handed the finished work to the Pentagon, which soon handed it back, complaining that in their own tests the neural network did no better than chance at discriminating photos.

It turned out that in the researchers' data set, photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days. The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest.

This parable - which might or might not be fact - illustrates one of the most fundamental problems in the field of supervised learning and in fact the whole field of Artificial Intelligence: If the training problems and the real problems have the slightest difference in context - if they are not drawn from the same independently identically distributed process - there is no statistical guarantee from past success to future success. It doesn't matter if the AI seems to be working great under the training conditions. (This is not an unsolvable problem but it is an unpatchable problem. There are deep ways to address it - a topic beyond the scope of this post - but no bandaids.)

As described in Superexponential Conceptspace, there are exponentially more possible concepts than possible objects, just as the number of possible objects is exponential in the number of attributes. If a black-and-white image is 256 pixels on a side, then the total image is 65536 pixels. The number of possible images is 2^65536. And the number of possible concepts that classify images into positive and negative instances - the number of possible boundaries you could draw in the space of images - is 2^(2^65536). From this, we see that even supervised learning is almost entirely a matter of inductive bias, without which it would take a minimum of 2^65536 classified examples to discriminate among 2^(2^65536) possible concepts - even if classifications are constant over time.

If this seems at all counterintuitive or non-obvious, see Superexponential Conceptspace.

So let us now turn again to:

'First we can build relatively simple machines that learn to recognize happiness and unhappiness in human facial expressions, human voices and human body language. Then we can hard-wire the result of this learning as the innate emotional values of more complex intelligent machines, positively reinforced when we are happy and negatively reinforced when we are unhappy.'

and

'When it is feasible to build a super-intelligence, it will be feasible to build hard-wired recognition of "human facial expressions, human voices and human body language" (to use the words of mine that you quote) that exceed the recognition accuracy of current humans such as you and me, and will certainly not be fooled by "tiny molecular pictures of smiley-faces." You should not assume such a poor implementation of my idea that it cannot make discriminations that are trivial to current humans.'

It's trivial to discriminate a photo of a picture with a camouflaged tank, and a photo of an empty forest, in the sense of determining that the two photos are not identical. They're different pixel arrays with different 1s and 0s in them. Discriminating between them is as simple as testing the arrays for equality.

Classifying new photos into positive and negative instances of "smile", by reasoning from a set of training photos classified positive or negative, is a different order of problem.

When you've got a 256x256 image from a real-world camera, and the image turns out to depict a camouflaged tank, there is no additional 65537th bit denoting the positiveness - no tiny little XML tag that says "This image is inherently positive". It's only a positive example relative to some particular concept.

But for any non-Vast amount of training data - any training data that does not include the exact bitwise image now seen - there are superexponentially many possible concepts compatible with previous classifications.

For the AI, choosing or weighting from among superexponential possibilities is a matter of inductive bias. Which may not match what the user has in mind. The gap between these two example-classifying processes - induction on the one hand, and the user's actual goals on the other - is not trivial to cross.

Let's say the AI's training data is:

Dataset 1:

- +

- Smile_1, Smile_2, Smile_3

- -

- Frown_1, Cat_1, Frown_2, Frown_3, Cat_2, Boat_1, Car_1, Frown_5

Now the AI grows up into a superintelligence, and encounters this data:

Dataset 2:

- Frown_6, Cat_3, Smile_4, Galaxy_1, Frown_7, Nanofactory_1, Molecular_Smileyface_1, Cat_4, Molecular_Smileyface_2, Galaxy_2, Nanofactory_2

It is not a property of these datasets that the inferred classification you would prefer is:

- +

- Smile_1, Smile_2, Smile_3, Smile_4

- -

- Frown_1, Cat_1, Frown_2, Frown_3, Cat_2, Boat_1, Car_1, Frown_5, Frown_6, Cat_3, Galaxy_1, Frown_7, Nanofactory_1, Molecular_Smileyface_1, Cat_4, Molecular_Smileyface_2, Galaxy_2, Nanofactory_2

rather than

- +

- Smile_1, Smile_2, Smile_3, Molecular_Smileyface_1, Molecular_Smileyface_2, Smile_4

- -

- Frown_1, Cat_1, Frown_2, Frown_3, Cat_2, Boat_1, Car_1, Frown_5, Frown_6, Cat_3, Galaxy_1, Frown_7, Nanofactory_1, Cat_4, Galaxy_2, Nanofactory_2

Both of these classifications are compatible with the training data. The number of concepts compatible with the training data will be much larger, since more than one concept can project the same shadow onto the combined dataset. If the space of possible concepts includes the space of possible computations that classify instances, the space is infinite.

Which classification will the AI choose? This is not an inherent property of the training data; it is a property of how the AI performs induction.

Which is the correct classification? This is not a property of the training data; it is a property of your preferences (or, if you prefer, a property of the idealized abstract dynamic you name "right").

The concept that you wanted, cast its shadow onto the training data as you yourself labeled each instance + or -, drawing on your own intelligence and preferences to do so. That's what supervised learning is all about - providing the AI with labeled training examples that project a shadow of the causal process that generated the labels.

But unless the training data is drawn from exactly the same context as the real-life, the training data will be "shallow" in some sense, a projection from a much higher-dimensional space of possibilities.

The AI never saw a tiny molecular smileyface during its dumber-than-human training phase, or it never saw a tiny little agent with a happiness counter set to a googolplex. Now you, finally presented with a tiny molecular smiley - or perhaps a very realistic tiny sculpture of a human face - know at once that this is not what you want to count as a smile. But that judgment reflects an unnatural category, one whose classification boundary depends sensitively on your complicated values. It is your own plans and desires that are at work when you say "No!"

Hibbard knows instinctively that a tiny molecular smileyface isn't a "smile", because he knows that's not what he wants his putative AI to do. If someone else were presented with a different task, like classifying artworks, they might feel that the Mona Lisa was obviously smiling - as opposed to frowning, say - even though it's only paint.

As the case of Terry Schiavo illustrates, technology enables new borderline cases that throw us into new, essentially moral dilemmas. Showing an AI pictures of living and dead humans as they existed during the age of Ancient Greece, will not enable the AI to make a moral decision as to whether switching off Terry's life support is murder. That information isn't present in the dataset even inductively! Terry Schiavo raises new moral questions, appealing to new moral considerations, that you wouldn't need to think about while classifying photos of living and dead humans from the time of Ancient Greece. No one was on life support then, still breathing with a brain half fluid. So such considerations play no role in the causal process that you use to classify the ancient-Greece training data, and hence cast no shadow on the training data, and hence are not accessible by induction on the training data.

As a matter of formal fallacy, I see two anthropomorphic errors on display.

The first fallacy is underestimating the complexity of a concept we develop for the sake of its value. The borders of the concept will depend on many values and probably on-the-fly moral reasoning, if the borderline case is of a kind we haven't seen before. But all that takes place invisibly, in the background; to Hibbard it just seems that a tiny molecular smileyface is just obviously not a smile. And we don't generate all possible borderline cases, so we don't think of all the considerations that might play a role in redefining the concept, but haven't yet played a role in defining it. Since people underestimate the complexity of their concepts, they underestimate the difficulty of inducing the concept from training data. (And also the difficulty of describing the concept directly - see The Hidden Complexity of Wishes.)

The second fallacy is anthropomorphic optimism: Since Bill Hibbard uses his own intelligence to generate options and plans ranking high in his preference ordering, he is incredulous at the idea that a superintelligence could classify never-before-seen tiny molecular smileyfaces as a positive instance of "smile". As Hibbard uses the "smile" concept (to describe desired behavior of superintelligences), extending "smile" to cover tiny molecular smileyfaces would rank very low in his preference ordering; it would be a stupid thing to do - inherently so, as a property of the concept itself - so surely a superintelligence would not do it; this is just obviously the wrong classification. Certainly a superintelligence can see which heaps of pebbles are correct or incorrect.

Why, Friendly AI isn't hard at all! All you need is an AI that does what's good! Oh, sure, not every possible mind does what's good - but in this case, we just program the superintelligence to do what's good. All you need is a neural network that sees a few instances of good things and not-good things, and you've got a classifier. Hook that up to an expected utility maximizer and you're done!

I shall call this the fallacy of magical categories - simple little words that turn out to carry all the desired functionality of the AI. Why not program a chess-player by running a neural network (that is, a magical category-absorber) over a set of winning and losing sequences of chess moves, so that it can generate "winning" sequences? Back in the 1950s it was believed that AI might be that simple, but this turned out not to be the case.

The novice thinks that Friendly AI is a problem of coercing an AI to make it do what you want, rather than the AI following its own desires. But the real problem of Friendly AI is one of communication - transmitting category boundaries, like "good", that can't be fully delineated in any training data you can give the AI during its childhood. Relative to the full space of possibilities the Future encompasses, we ourselves haven't imagined most of the borderline cases, and would have to engage in full-fledged moral arguments to figure them out. To solve the FAI problem you have to step outside the paradigm of induction on human-labeled training data and the paradigm of human-generated intensional definitions.

Of course, even if Hibbard did succeed in conveying to an AI a concept that covers exactly every human facial expression that Hibbard would label a "smile", and excludes every facial expression that Hibbard wouldn't label a "smile"...

Then the resulting AI would appear to work correctly during its childhood, when it was weak enough that it could only generate smiles by pleasing its programmers.

When the AI progressed to the point of superintelligence and its own nanotechnological infrastructure, it would rip off your face, wire it into a permanent smile, and start xeroxing.

The deep answers to such problems are beyond the scope of this post, but it is a general principle of Friendly AI that there are no bandaids. In 2004, Hibbard modified his proposal to assert that expressions of human agreement should reinforce the definition of happiness, and then happiness should reinforce other behaviors. Which, even if it worked, just leads to the AI xeroxing a horde of things similar-in-its-conceptspace to programmers saying "Yes, that's happiness!" about hydrogen atoms - hydrogen atoms are easy to make.

Link to my discussion with Hibbard here. You already got the important parts.

Referenced by: Qualitative Strategies of Friendliness • Dreams of Friendliness • Ethical Injunctions • What I Think, If Not Why • Not Taking Over the World • In Praise of Boredom • Value is Fragile

Original with comments: Magical Categories

Three Fallacies of Teleology

Eliezer Yudkowsky, 25 August 2008 10:27PM

Followup to: Anthropomorphic Optimism

Aristotle distinguished between four senses of the Greek word aition, which in English is translated as "cause", though Wikipedia suggests that a better translation is "maker". Aristotle's theory of the Four Causes, then, might be better translated as the Four Makers. These were his four senses of aitia: The material aition, the formal aition, the efficient aition, and the final aition.

The material aition of a bronze statue is the substance it is made from, bronze. The formal aition is the substance's form, its statue-shaped-ness. The efficient aition best translates as the English word "cause"; we would think of the artisan carving the statue, though Aristotle referred to the art of bronze-casting the statue, and regarded the individual artisan as a mere instantiation.

The final aition was the goal, or telos, or purpose of the statue, that for the sake of which the statue exists.

Though Aristotle considered knowledge of all four aitia as necessary, he regarded knowledge of the telos as the knowledge of highest order. In this, Aristotle followed in the path of Plato, who had earlier written:

Imagine not being able to distinguish the real cause from that without which the cause would not be able to act as a cause. It is what the majority appear to do, like people groping in the dark; they call it a cause, thus giving it a name that does not belong to it. That is why one man surrounds the earth with a vortex to make the heavens keep it in place, another makes the air support it like a wide lid. As for their capacity of being in the best place they could possibly be put, this they do not look for, nor do they believe it to have any divine force...

Suppose that you translate "final aition" as "final cause", and assert directly:

"Why do human teeth develop with such regularity, into a structure well-formed for biting and chewing? You could try to explain this as an incidental fact, but think of how unlikely that would be. Clearly, the final cause of teeth is the act of biting and chewing. Teeth develop with regularity, because of the act of biting and chewing - the latter causes the former."

A modern-day sophisticated Bayesian will at once remark, "This requires me to draw a circular causal diagram with an arrow going from the future to the past."

It's not clear to me to what extent Aristotle appreciated this point - that you could not draw causal arrows from the future to the past. Aristotle did acknowledge that teeth also needed an efficient cause to develop. But Aristotle may have believed that the efficient cause could not act without the telos, or was directed by the telos, in which case we again have a reversed direction of causality, a dependency of the past on the future. I am no scholar of the classics, so it may be only myself who is ignorant of what Aristotle believed on this score.

So the first way in which teleological reasoning may be an outright fallacy, is when an arrow is drawn directly from the future to the past. In every case where a present event seems to happen for the sake of a future end, that future end must be materially represented in the past.

Suppose you're driving to the supermarket, and you say that each right turn and left turn happens for the sake of the future event of your being at the supermarket. Then the actual efficient cause of the turn, consists of: the representation in your mind of the event of yourself arriving at the supermarket; your mental representation of the street map (not the streets themselves); your brain's planning mechanism that searches for a plan that represents arrival at the supermarket; and the nerves that translate this plan into the motor action of your hands turning the steering wheel.

All these things exist in the past or present; no arrow is drawn from the future to the past.

In biology, similarly, we explain the regular formation of teeth, not by letting it be caused directly by the future act of chewing, but by using the theory of natural selection to relate past events of chewing to the organism's current genetic makeup, which physically controls the formation of the teeth. Thus, we account for the current regularity of the teeth by referring only to past and present events, never to future events. Such evolutionary reasoning is called "teleonomy", in contrast with teleology.

We can see that the efficient cause is primary, not the final cause, by considering what happens when the two come into conflict. The efficient cause of human taste buds is natural selection on past human eating habits; the final cause of human taste buds is acquiring nutrition. From the efficient cause, we should expect human taste buds to seek out resources that were scarce in the ancestral environment, like fat and sugar. From the final cause, we would expect human taste buds to seek out resources scarce in the current environment, like vitamins and fiber. From the sales numbers on candy bars, we can see which wins. The saying "Individual organisms are best thought of as adaptation-executers rather than as fitness-maximizers" asserts the primacy of teleonomy over teleology.

Similarly, if you have a mistake in your mind about where the supermarket lies, the final event of your arrival at the supermarket, will not reach backward in time to steer your car. If I know your exact state of mind, I will be able to predict your car's trajectory by modeling your current state of mind, not by supposing that the car is attracted to some particular final destination. If I know your mind in detail, I can even predict your mistakes, regardless of what you think is your goal.

The efficient cause has screened off the telos: If I can model the complete mechanisms at work in the present, I never have to take into account the future in predicting the next time step.

So that is the first fallacy of teleology - to make the future a literal cause of the past.

Now admittedly, it may be convenient to engage in reasoning that would be fallacious if interpreted literally. For example:

I don't know the exact state of Mary's every neuron. But I know that she desires to be at the supermarket. If Mary turns left at the next intersection, she will then be at the supermarket (at time t=1). Therefore Mary will turn left (at time t=0).

But this is only convenient shortcut, to let the future affect Mary's present actions. More rigorous reasoning would say:

My model predicts that if Mary turns left she will arrive at the supermarket. I don't know her every neuron, but I believe Mary has a model similar to mine. I believe Mary desires to be at the supermarket. I believe that Mary has a planning mechanism similar to mine, which leads her to take actions that her model predicts will lead to the fulfillment of her desires. Therefore I predict that Mary will turn left.

No direct mention of the actual future has been made. I predict Mary by imagining myself to have her goals, then putting myself and my planning mechanisms into her shoes, letting my brain do planning-work that is similar to the planning-work I expect Mary to do. This requires me to talk only about Mary's goal, our models (presumed similar) and our planning mechanisms (presumed similar) - all forces active in the present.

And the benefit of this more rigorous reasoning, is that if Mary is mistaken about the supermarket's location, then I do not have to suppose that the future event of her arrival reaches back and steers her correctly anyway.

Teleological reasoning is anthropomorphic - it uses your own brain as a black box to predict external events. Specifically, teleology uses your brain's planning mechanism as a black box to predict a chain of future events, by planning backward from a distant outcome.

Now we are talking about a highly generalized form of anthropomorphism - and indeed, it is precisely to introduce this generalization that I am talking about teleology! You know what it's like to feel purposeful. But when someone says, "water runs downhill so that it will be at the bottom", you don't necessarily imagine little sentient rivulets alive with quiet determination. Nonetheless, when you ask, "How could the water get to the bottom of the hill?" and plot out a course down the hillside, you're recruiting your own brain's planning mechanisms to do it. That's what the brain's planner does, after all: it finds a path to a specified destination starting from the present.

And if you expect the water to avoid local maxima so it can get all the way to the bottom of the hill - to avoid being trapped in small puddles far above the ground - then your anthropomorphism is going to produce the wrong prediction. (This is how a lot of mistaken evolutionary reasoning gets done, since evolution has no foresight, and only takes the next greedy local step.)

But consider the subtlety: you may have produced a wrong, anthropomorphic prediction of the water without ever thinking of it as a person - without ever visualizing it as having feelings - without even thinking "the water has purpose" or "the water wants to be at the bottom of the hill" - but only saying, as Aristotle did, "the water's telos is to be closer to the center of the Earth". Or maybe just, "the water runs downhill so that it will be at the bottom". (Or, "I expect that human taste buds will take into account how much of each nutrient the body needs, and so reject fat and sugar if there are enough calories present, since evolution produced taste buds in order to acquire nutrients.")

You don't notice instinctively when you're using an aspect of your brain as a black box to predict outside events. Consequentialism just seems like an ordinary property of the world, something even rocks could do.

It takes a deliberate act of reductionism to say: "But the water has no brain; how can it predict ahead to see itself being trapped in a local puddle, when the future cannot directly affect the past? How indeed can anything at all happen in the water so that it will, in the future, be at the bottom? No; I should try to understand the water's behavior using only local causes, found in the immediate past."

It takes a deliberate act of reductionism to identify telos as purpose, and purpose as a mental property which is too complicated to be ontologically fundamental. You don't realize, when you ask "What does this telos-imbued object do next?", that your brain is answering by calling on its own complicated planning mechanisms, that search multiple paths and do means-end reasoning. Purpose just seems like a simple and basic property; the complexity of your brain that produces the predictions is hidden from you. It is an act of reductionism to see purpose as requiring a complicated AI algorithm that needs a complicated material embodiment.

So this is the second fallacy of teleology - to attribute goal-directed behavior to things that are not goal-directed, perhaps without even thinking of the things as alive and spirit-inhabited, but only thinking, X happens in order to Y. "In order to" is mentalistic language, even though it doesn't seem to name a blatantly mental property like "fearful" or "thinks it can fly".

Remember the sequence on free will? The problem, it turned out, was that "could" was a mentalistic property - generated by the planner in the course of labeling states as reachable from the start state. It seemed like "could" was a physical, ontological property. When you say "could" it doesn't sound like you're talking about states of mind. Nonetheless, the mysterious behavior of could-ness turned out to be understandable only by looking at the brain's planning mechanisms.

Since mentalistic reasoning uses your own mind as a black box to generate its predictions, it very commonly generates wrong questions and mysterious answers.

If you want to accomplish anything related to philosophy, or anything related to Artificial Intelligence, it is necessary to learn to identify mentalistic language and root it all out - which can only be done by analyzing innocent-seeming words like "could" or "in order to" into the complex cognitive algorithms that are their true identities.

(If anyone accuses me of "extreme reductionism" for saying this, let me ask how likely it is that we live in an only partially reductionist universe.)

The third fallacy of teleology is to commit the Mind Projection Fallacy with respect to telos, supposing it to be an inherent property of an object or system. Indeed, one does this every time one speaks of the purpose of an event, rather than speaking of some particular agent desiring the consequences of that event.

I suspect this is why people have trouble understanding evolutionary psychology - in particular, why they suppose that all human acts are unconsciously directed toward reproduction. "Mothers who loved their children outreproduced those who left their children to the wolves" becomes "natural selection produced motherly love in order to ensure the survival of the species" becomes "the purpose of acts of motherly love is to increase the mother's fitness". Well, if a mother apparently drags her child off the train tracks because she loves the child, that's also the purpose of the act, right? So by a fallacy of compression - a mental model that has one bucket where two buckets are needed - the purpose must be one or the other: either love or reproductive fitness.

Similarly with those who hear of evolutionary psychology and conclude that the meaning of life is to increase reproductive fitness - hasn't science demonstrated that this is the purpose of all biological organisms, after all?

Likewise with that fellow who concluded that the purpose of the universe is to increase entropy - the universe does so consistently, therefore it must want to do so - and that this must therefore be the meaning of life. Pretty sad purpose, I'd say! But of course the speaker did not seem to realize what it means to want to increase entropy as much as possible - what this goal really implies, that you should go around collapsing stars to black holes. Instead the one focused on a few selected activities that increase entropy, like thinking. You couldn't ask for a clearer illustration of a fake utility function.

I call this a "teleological capture" - where someone comes to believe that the telos of X is Y, relative to some agent, or optimization process, or maybe just statistical tendency, from which it follows that any human or other agent who does X must have a purpose of Y in mind. The evolutionary reason for motherly love becomes its telos, and seems to "capture" the apparent motives of human mothers. The game-theoretical reason for cooperating on the Iterated Prisoner's Dilemma becomes the telos of cooperation, and seems to "capture" the apparent motives of human altruists, who are thus revealed as being selfish after all. Charity increases status, which people are known to desire; therefore status is the telos of charity, and "captures" all claims to kinder motives. Etc. etc. through half of all amateur philosophical reasoning about the meaning of life.

These then are three fallacies of teleology: Backward causality, anthropomorphism, and teleological capture.

Referenced by: Dreams of AI Design • Against Modal Logics • Excluding the Supernatural • My Naturalistic Awakening • Why Does Power Corrupt?

Original with comments: Three Fallacies of Teleology

Dreams of AI Design

Eliezer Yudkowsky, 26 August 2008 11:28PM

Followup to: Anthropomorphic Optimism, Three Fallacies of Teleology

After spending a decade or two living inside a mind, you might think you knew a bit about how minds work, right? That's what quite a few AGI wannabes (people who think they've got what it takes to program an Artificial General Intelligence) seem to have concluded. This, unfortunately, is wrong.

Artificial Intelligence is fundamentally about reducing the mental to the non-mental.

You might want to contemplate that sentence for a while. It's important.

Living inside a human mind doesn't teach you the art of reductionism, because nearly all of the work is carried out beneath your sight, by the opaque black boxes of the brain. So far beneath your sight that there is no introspective sense that the black box is there - no internal sensory event marking that the work has been delegated.

Did Aristotle realize that when he talked about the telos, the final cause of events, that he was delegating predictive labor to his brain's complicated planning mechanisms - asking, "What would this object do, if it could make plans?" I rather doubt it. Aristotle thought the brain was an organ for cooling the blood - which he did think was important: Humans, thanks to their larger brains, were more calm and contemplative.

So there's an AI design for you! We just need to cool down the computer a lot, so it will be more calm and contemplative, and won't rush headlong into doing stupid things like modern computers.

That's an example of fake reductionism. "Humans are more contemplative because their blood is cooler", I mean. It doesn't resolve the black box of the word contemplative. You can't predict what a contemplative thing does using a complicated model with internal moving parts composed of merely material, merely causal elements - positive and negative voltages on a transistor being the canonical example of a merely material and causal element of a model. All you can do is imagine yourself being contemplative, to get an idea of what a contemplative agent does.

Which is to say that you can only reason about "contemplative-ness" by empathic inference - using your own brain as a black box with the contemplativeness lever pulled, to predict the output of another black box.

You can imagine another agent being contemplative, but again that's an act of empathic inference - the way this imaginative act works is by adjusting your own brain to run in contemplativeness-mode, not by modeling the other brain neuron by neuron. Yes, that may be more efficient, but it doesn't let you build a "contemplative" mind from scratch.

You can say that "cold blood causes contemplativeness" and then you just have fake causality: You've drawn a little arrow from a box reading "cold blood" to a box reading "contemplativeness", but you haven't looked inside the box - you're still generating your predictions using empathy.

You can say that "lots of little neurons, which are all strictly electrical and chemical with no ontologically basic contemplativeness in them, combine into a complex network that emergently exhibits contemplativeness". And that is still a fake reduction and you still haven't looked inside the black box. You still can't say what a "contemplative" thing will do, using a non-empathic model. You just took a box labeled "lotsa neurons", and drew an arrow labeled "emergence" to a black box containing your remembered sensation of contemplativeness, which, when you imagine it, tells your brain to empathize with the box by contemplating.

So what do real reductions look like?

Like the relationship between the feeling of evidence-ness, of justification-ness, and E. T. Jaynes's "Probability Theory: The Logic of Science". You can go around in circles all day, saying how the nature of evidence, is that it justifies some proposition, by meaning that it's more likely to be true, but all of these just invoke your brain's internal feelings of evidence-ness, justifies-ness, likeliness. That part is easy - the going around in circles part. The part where you go from there to Bayes's Theorem is hard.

And the fundamental mental ability that lets someone learn Artificial Intelligence is the ability to tell the difference. So that you know you aren't done yet, nor even really started, when you say, "Evidence is when an observation justifies a belief". But atoms are not evidential, justifying, meaningful, likely, propositional, or true, they are just atoms. Only things like "P(H|E)/P(~H|E) = P(E|H)/P(E|~H) * P(H)/P(~H)" count as substantial progress. (And that's only the first step of the reduction: what are these E and H objects, if not mysterious black boxes? Where do your hypotheses come from? From your creativity? And what's a hypothesis, when no atom is a hypothesis?)

Another excellent example of genuine reduction can be found in Judea Pearl's Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. You could go around all day in circles talk about how a cause is something that makes something else happen, and until you understood the nature of conditional independence, you would be helpless to make an AI that reasons about causation. Because you wouldn't understand what was happening, when your brain mysteriously decided, that if you learned your burglar alarm went off, but you then learned that a small earthquake took place, you would retract your initial conclusion that your house had been burglarized.

If you want an AI that plays chess, you can go around in circles indefinitely talking about how you want the AI to make good moves, which are moves that can be expected to win the game, which are moves that are prudent strategies for defeating the opponent, etcetera; and while you may then have some idea of which moves you want the AI to make, it's all for naught until you come up with the notion of a mini-max search tree.

But until you know about search trees, until you know about conditional independence, until you know about Bayes's Theorem, then it may still seem to you that you have a perfectly good understanding of where good moves and nonmonotonic reasoning and evaluation of evidence come from. It may seem, for example, that they come from cooling the blood.

And indeed I know many people who believe that intelligence is the product of commonsense knowledge or massive parallelism or creative destruction or intuitive rather than rational reasoning, or whatever. But all these are only dreams, which do not give you any way to say what intelligence is, or what an intelligence will do next, except by pointing at a human. And when the one goes to build their wondrous AI, they only build a system of detached levers, "knowledge" consisting of LISP tokens labeled apple and the like; or perhaps they build a "massively parallel neural net, just like the human brain". And are shocked - shocked! - when nothing much happens.

AI designs made of human parts are only dreams; they can exist in the imagination, but not translate into transistors. This applies specifically to "AI designs" that look like boxes with arrows between them and meaningful-sounding labels on the boxes. (For a truly epic example thereof, see any Mentifex Diagram.)

Later I will say more upon this subject, but I can go ahead and tell you one of the guiding principles: If you meet someone who says that their AI will do XYZ just like humans, do not give them any venture capital. Say to them rather: "I'm sorry, I've never seen a human brain, or any other intelligence, and I have no reason as yet to believe that any such thing can exist. Now please explain to me what your AI does, and why you believe it will do it, without pointing to humans as an example." Planes would fly just as well, given a fixed design, if birds had never existed; they are not kept aloft by analogies.

So now you perceive, I hope, why, if you wanted to teach someone to do fundamental work on strong AI - bearing in mind that this is demonstrably a very difficult art, which is not learned by a supermajority of students who are just taught existing reductions such as search trees - then you might go on for some length about such matters as the fine art of reductionism, about playing rationalist's Taboo to excise problematic words and replace them with their referents, about anthropomorphism, and, of course, about early stopping on mysterious answers to mysterious questions.

Referenced by: Excluding the Supernatural • My Naturalistic Awakening • Above-Average AI Scientists • Ethical Injunctions • Economic Definition of Intelligence? • Failure By Analogy • Failure By Affective Analogy • Artificial Mysterious Intelligence • Nonperson Predicates • Nonsentient Optimizers

Original with comments: Dreams of AI Design

Against Modal Logics

Eliezer Yudkowsky, 27 August 2008 10:13PM

Continuation of: Grasping Slippery Things
Followup to: Possibility and Could-ness, Three Fallacies of Teleology

When I try to hit a reduction problem, what usually happens is that I "bounce" - that's what I call it. There's an almost tangible feel to the failure, once you abstract and generalize and recognize it. Looking back, it seems that I managed to say most of what I had in mind for today's post, in "Grasping Slippery Things". The "bounce" is when you try to analyze a word like could, or a notion like possibility, and end up saying, "The set of realizable worlds [A'] that follows from an initial starting world A operated on by a set of physical laws f." Where realizable contains the full mystery of "possible" - but you've made it into a basic symbol, and added some other symbols: the illusion of formality.

There are a number of reasons why I feel that modern philosophy, even analytic philosophy, has gone astray - so far astray that I simply can't make use of their years and years of dedicated work, even when they would seem to be asking questions closely akin to mine.

The proliferation of modal logics in philosophy is a good illustration of one major reason: Modern philosophy doesn't enforce reductionism, or even strive for it.

Most philosophers, as one would expect from Sturgeon's Law, are not very good. Which means that they're not even close to the level of competence it takes to analyze mentalistic black boxes into cognitive algorithms. Reductionism is, in modern times, an unusual talent. Insights on the order of Pearl et. al.'s reduction of causality or Julian Barbour's reduction of time are rare.

So what these philosophers do instead, is "bounce" off the problem into a new modal logic: A logic with symbols that embody the mysterious, opaque, unopened black box. A logic with primitives like "possible" or "necessary", to mark the places where the philosopher's brain makes an internal function call to cognitive algorithms as yet unknown.

And then they publish it and say, "Look at how precisely I have defined my language!"

In the Wittgensteinian era, philosophy has been about language - about trying to give precise meaning to terms.

The kind of work that I try to do is not about language. It is about reducing mentalistic models to purely causal models, about opening up black boxes to find complicated algorithms inside, about dissolving mysteries - in a word, about cognitive science.

That's what I think post-Wittgensteinian philosophy should be about - cognitive science.

But this kind of reductionism is hard work. Ideally, you're looking for insights on the order of Julian Barbour's Machianism, to reduce time to non-time; insights on the order of Judea Pearl's conditional independence, to give a mathematical structure to causality that isn't just finding a new way to say "because"; insights on the order of Bayesianism, to show that there is a unique structure to uncertainty expressed quantitatively.

Just to make it clear that I'm not claiming a magical and unique ability, I would name Gary Drescher's Good and Real as an example of a philosophical work that is commensurate with the kind of thinking I have to try to do. Gary Drescher is an AI researcher turned philosopher, which may explain why he understands the art of asking, not What does this term mean?, but What cognitive algorithm, as seen from the inside, would generate this apparent mystery?

(I paused while reading the first chapter of G&R. It was immediately apparent that Drescher was thinking along lines so close to myself, that I wanted to write up my own independent component before looking at his - I didn't want his way of phrasing things to take over my writing. Now that I'm done with zombies and metaethics, G&R is next up on my reading list.)

Consider the popular philosophical notion of "possible worlds". Have you ever seen a possible world? Is an electron either "possible" or "necessary"?Clearly, if you are talking about "possibility" and "necessity", you are talking about things that are not commensurate with electrons - which means that you're still dealing with a world as seen from the inner surface of a cognitive algorithm, a world of surface levers with all the underlying machinery hidden.

I have to make an AI out of electrons, in this one actual world. I can't make the AI out of possibility-stuff, because I can't order a possible transistor. If the AI ever thinks about possibility, it's not going to be because the AI noticed a possible world in its closet. It's going to be because the non-ontologically-fundamental construct of "possibility" turns out to play a useful role in modeling and manipulating the one real world, a world that does not contain any fundamentally possible things. Which is to say that algorithms which make use of a "possibility" label, applied at certain points, will turn out to capture an exploitable regularity of the one real world. This is the kind of knowledge that Judea Pearl writes about. This is the kind of knowledge that AI researchers need. It is not the kind of knowledge that modern philosophy holds itself to the standard of having generated, before a philosopher gets credit for having written a paper.

Philosophers keep telling me that I should look at philosophy. I have, every now and then. But the main reason I look at philosophy is when I find it desirable to explain things to philosophers. The work that has been done - the products of these decades of modern debate - is, by and large, just not commensurate with the kind of analysis AI needs. I feel a bit awful about saying this, because it feels like I'm telling philosophers that their life's work has been a waste of time - not that professional philosophers would be likely to regard me as an authority on whose life has been a waste of time. But if there's any centralized repository of reductionist-grade naturalistic cognitive philosophy, I've never heard mention of it.

And: Philosophy is just not oriented to the outlook of someone who needs to resolve the issue, implement the corresponding solution, and then find out - possibly fatally - whether they got it right or wrong. Philosophy doesn't resolve things, it compiles positions and arguments. And if the debate about zombies is still considered open, then I'm sorry, but as Jeffreyssai says: Too slow! It would be one matter if I could just look up the standard answer and find that, lo and behold, it is correct. But philosophy, which hasn't come to conclusions and moved on from cognitive reductions that I regard as relatively simple, doesn't seem very likely to build complex correct structures of conclusions.

Sorry - but philosophy, even the better grade of modern analytic philosophy, doesn't seem to end up commensurate with what I need, except by accident or by extraordinary competence. Parfit comes to mind; and I haven't read much Dennett, but Dennett does seem to be trying to do the same sort of thing that I try to do; and of course there's Gary Drescher. If there was a repository of philosophical work along those lines - not concerned with defending basic ideas like anti-zombieism, but with accepting those basic ideas and moving on to challenge more difficult quests of naturalism and cognitive reductionism - then that, I might well be interested in reading. But I don't know who, besides a few heroes, would be able to compile such a repository - who else would see a modal logic as an obvious bounce-off-the-mystery.

Referenced by: My Bayesian Enlightenment

Original with comments: Against Modal Logics

Harder Choices Matter Less

Eliezer Yudkowsky, 29 August 2008 02:02AM

...or they should, logically speaking.

Suppose you're torn in an agonizing conflict between two choices.

Well... if you can't decide between them, they must be around equally appealing, right? Equally balanced pros and cons? So the choice must matter very little - you may as well flip a coin. The alternative is that the pros and cons aren't equally balanced, in which case the decision should be simple.

This is a bit of a tongue-in-cheek suggestion, obviously - more appropriate for choosing from a restaurant menu than choosing a major in college.

But consider the case of choosing from a restaurant menu. The obvious choices, like Pepsi over Coke, will take very little time. Conversely, the choices that take the most time probably make the least difference. If you can't decide between the hamburger and the hot dog, you're either close to indifferent between them, or in your current state of ignorance you're close to indifferent between their expected utilities.

Does this have any moral for larger dilemmas, like choosing a major in college? Here, it's more likely that you're in a state of ignorance, than that you would have no real preference over outcomes. Then if you're agonizing, the obvious choice is "gather more information" - get a couple of part-time jobs that let you see the environment you would be working in. And, logically, you can defer the agonizing until after that.

Or maybe you've already gathered information, but can't seem to integrate to a decision? Then you should be listing out pros and cons on a sheet of paper, or writing down conflicting considerations and trying to decide which consideration is, in general, the most important to you. Then that's the obvious thing you should do, which clearly dominates the alternative of making a snap decision in either direction.

Of course there are also biases that get stronger as we think longer - it gives us more opportunity to rationalize, for example; or it gives us more opportunity to think up extreme but rare/unlikely considerations whose affect dominates the decision process. Like someone choosing a longer commute to work (every day), so that they can have a house with an extra room for when Grandma comes over (once a year). If you think your most likely failure mode is that you'll outsmart yourself, then the obvious choice is to make a snap decision in the direction you're currently leaning, which you're probably going to end up picking anyhow.

I do think there's something to be said for agonizing over important decisions, but only so long as the agonization process is currently going somewhere, not stuck.

Original with comments: Harder Choices Matter Less

Qualitative Strategies of Friendliness

Eliezer Yudkowsky, 30 August 2008 02:12AM

Followup to: Magical Categories

What on Earth could someone possibly be thinking, when they propose creating a superintelligence whose behaviors are reinforced by human smiles? Tiny molecular photographs of human smiles - or if you rule that out, then faces ripped off and permanently wired into smiles - or if you rule that out, then brains stimulated into permanent maximum happiness, in whichever way results in the widest smiles...

Well, you never do know what other people are thinking, but in this case I'm willing to make a guess. It has to do with a field of cognitive psychology called Qualitative Reasoning.

Qualitative reasoning is what you use to decide that increasing the temperature of your burner increases the rate at which your water boils, which decreases the derivative of the amount of water present. One would also add the sign of d(water) - negative, meaning that the amount of water is decreasing - and perhaps the fact that there is only a bounded amount of water. Or we could say that turning up the burner increases the rate at which the water temperature increases, until the water temperature goes over a fixed threshold, at which point the water starts boiling, and hence decreasing in quantity... etc.

That's qualitative reasoning, a small subfield of cognitive science and Artificial Intelligence - reasoning that doesn't describe or predict exact quantities, but rather the signs of quantities, their derivatives, the existence of thresholds.

As usual, human common sense means we can see things by qualitative reasoning that current programs can't - but the more interesting realization is how vital human qualitative reasoning is to our vaunted human common sense. It's one of the basic ways in which we comprehend the world.

Without timers you can't figure out how long water takes to boil, your mind isn't that precise. But you can figure out that you should turn the burner up, rather than down, and then watch to make sure the water doesn't all boil away. Which is what you mainly need, in the real world. Or at least we humans seem to get by on qualitative reasoning; we may not realize what we're missing...

So I suspect that what went through the one's mind, proposing the AI whose behaviors would be reinforced by human smiles, was something like this:

The happier people are, the more they smile. Smiles reinforce the behavior of the AI, so it does more of whatever makes people happy. Being happy is good (that's what the positive connection to "utility" is about). Therefore this is a good AI to construct, because more people will be happy, and that's better. Switch the AI right on!

How many problems are there with this reasoning?

Let us count the ways...

In fact, if you're interested in the field, you should probably try counting the ways yourself, before I continue. And score yourself on how deeply you stated a problem, not just the number of specific cases.

...

Problem 1: There are ways to cause smiles besides happiness. "What causes a smile?" "Happiness." That's the prototype event, the one that comes first to memory. But even in human affairs, you might be able to think of some cases where smiles result from a cause other than happiness.

Where a superintelligence is involved - even granting the hypothesis that it "wants smiles" or "executes behaviors reinforced by smiles" - then you're suddenly much more likely to be dealing with causes of smiles that are outside the human norm. Back in hunter-gatherer society, the main cause of eating food was that you hunted it or gathered it. Then came agriculture and domesticated animals. Today, some hospital patients are sustained by IVs or tubes, and at least a few of the vitamins or minerals in the mix may be purely synthetic.

A creative mind, faced with a goal state, tends to invent new ways of achieving it - new causes of the goal's achievement. It invents techniques that are faster or more reliable or less resource-intensive or with bigger wins. Consider how creative human beings are about obtaining money, and how many more ways there are to obtain money today than a few thousand years ago when money was first invented.

One of the ways of viewing our amazing human ability of "general intelligence" (or "significantly more generally applicable than chimpanzee intelligence") is that it operates across domains and can find new domains to exploit. You can see this in terms of learning new and unsuspected facts about the universe, and in terms of searching paths through time that wend through these new facts. A superintelligence would be more effective on both counts - but even on a human scale, this is why merely human progress, thinking with 200Hz neurons over a few hundred years, tends to change the way we do things and not just do the same things more effectively. As a result, a "weapon" today is not like a weapon of yestercentury, "long-distance communication today" is not a letter carried by horses and ships.

So when the AI is young, it can only obtain smiles by making the people around it happy. When the AI grows up to superintelligence, it makes its own nanotechnology and then starts manufacturing the most cost-effective kind of object that it has deemed to be a smile.

In general, a lot of naive-FAI plans I see proposed, have the property that, if actually implemented, the strategy might appear to work while the AI was dumber-than-human, but would fail when the AI was smarter than human. The fully general reason for this is that while the AI is dumber-than-human, it may not yet be powerful enough to create the exceptional conditions that will break the neat little flowchart that would work if every link operated according to the 21st-century First-World modal event.

This is why, when you encounter the AGI wannabe who hasn't planned out a whole technical approach to FAI, and confront them with the problem for the first time, and they say, "Oh, we'll test it to make sure that doesn't happen, and if any problem like that turns up we'll correct it, now let me get back to the part of the problem that really interests me," know then that this one has not yet leveled up high enough to have interesting opinions. It is a general point about failures in bad FAI strategies, that quite a few of them don't show up while the AI is in the infrahuman regime, and only show up once the strategy has gotten into the transhuman regime where it is too late to do anything about it.

Indeed, according to Bill Hibbard's actual proposal, where the AI is reinforced by seeing smiles, the FAI strategy would be expected to short out - from our perspective, from the AI's perspective it's being brilliantly creative and thinking outside the box for massive utility wins - to short out on the AI taking control of its own sensory instrumentation and feeding itself lots of smile-pictures. For it to keep doing this, and do it as much as possible, it must of course acquire as many resources as possible.

So! Let us repair our design as follows, then:

Now the AI is not being rewarded by any particular sensory input - on which the FAI strategy would presumably short out - but is, rather, trying to maximize an external and environmental quantity, the amount of happiness out there.

This already takes us into the realm of technical expertise - distinctions that can't be understood in just English, like the difference between expected utility maximization (which can be over external environmental properties that are modeled but not directly sensed) and reinforcement learning (which is inherently tied directly to sensors). See e.g. Terminal Values and Instrumental Values.

So in this case, then, the sensors give the AI information that it uses to infer a model of the world; the possible consequences of various plans are modeled, and the amount of "happiness" in that model summed by a utility function; and whichever plan corresponds to the greatest expectation of "happiness", that plan is output as actual actions.

Or in simpler language: The AI uses its sensors to find out what the world is like, and then it uses its actuators to make sure the world contains as much happiness as possible. Happiness is good, therefore it is good to turn on this AI.

What could possibly go wrong?

Problem 2: What exactly does the AI consider to be happiness?

Does the AI's model of a tiny little Super Happy Agent (consisting mostly of a reward center that represents a large number) meet the definition of "happiness" that the AI's utility function sums over, when it looks over the modeled consequences of its actions?

As discussed in Magical Categories, the super-exponential size of Concept-space and the "unnaturalness" of categories appearing in terminal values (their boundaries are not directly determined by naturally arising predictive problems) means that the boundary a human would draw around "happiness" is not trivial information to infuse into the AI.

I'm not going to reprise the full discussion in Magical Categories, but a sample set of things that the human labels "happy" or "not happy" is likely to miss out on key dimensions of possible variances, and never wend through labeling-influencing factors that would be important if they were invoked. Which is to say: Did you think of presenting the AI with the tiny Super Happy Agent, when you've never seen such a thing? Did you think of discussing chimpanzees, Down Syndrome children, and Terry Schiavo? How late would it have been, in humanity's technological development, before any human being could have and would have thought of the possibilities you're now generating? (Note opportunity for hindsight bias.)

Indeed, once you start talking about how we would label new borderline cases we've never seen, you're well into the realm of extrapolating volitions - you might as well ask how we would label these cases, if we knew everything the AI knew, and could consider larger sets of moral arguments, etc.

The standard dismissals here range from "Oh, of course I would think of X, therefore there's no problem" for any particular X that you suggest to them, by way of illustrating a systemic problem that they can't seem to grasp. Or "Well, I'll look at the AI's representation and see whether it defines 'happiness' the same way I do." (As if you would notice if one of the 15 different considerations that affect what you would define as 'truly happy' were left out! And also as if you could determine, by eyeballing, whether an AGI's internal representations would draw a border around as-yet-unimagined borderline instances, that you would find sensible.) Or the always popular, "But that's stupid, therefore a superintelligence won't make that mistake by doing something so pointless."

One of the reasons that qualitative planning works for humans as well as it does, is our ability to replan on the fly when an exceptional condition shows up. Can't the superintelligence just obviously see that manufacturing lots of tiny Super Happy agents is stupid, which is to say ranked-low-in-our-preference-ordering? Not if its preference ordering isn't like yours. (Followed by the appeals to universally compelling arguments demonstrating that making Super Happy agents is incorrect.)

But let's suppose that we can magically convey to the AI exactly what a human would consider as "happiness", by some unspecified and deep and technical art of Friendly AI. Then we have this shiny new diagram:

Of course this still doesn't work - but first, I explain the diagram. The dotted line between Humans::"Happy" and happiness-in-the-world, marked "by definition", means that the Happy box supposedly contains whatever is meant by the human concept of "happiness", as modeled by the AI, which by a magical FAI trick has been bound exactly to the human concept of "happiness". (If the happy box is neither what humans mean by happiness, nor what the AI means, then what's inside the box? True happiness? What do you mean by that?)

One glosses over numerous issues here - just as the original author of the original Happy Smiling AI proposal did - such as whether we all mean the same thing by "happiness". And whether we mean something consistent, that can be realized-in-the-world. In Humans::"Happy" there are neurons and their interconnections, the brain state containing the full and complete specification of the seed of what we mean by "happiness" - the implicit reactions that we would have, to various boundary cases and the like - but it would take some extrapolation of volition for the AI to decide how we would react to new boundary cases; it is not a trivial thing to draw a little dashed line between a human thought, and a concept boundary over the world of quarks and electrons, and say, "by definition". It wouldn't work on "omnipotence", for example: can you make a rock that you can't lift?

But let us assume all such issues away.

Problem 3: Is every act which increases the total amount of happiness in the universe, always the right thing to do?

If everyone in the universe just ends up with their brains hotwired to experience maximum happiness forever, or perhaps just replaced with orgasmium gloop, is that the greatest possible fulfillment of humanity's potential? Is this what we wish to make of ourselves?

"Oh, that's not real happiness," you say. But be wary of the No True Scotsman fallacy - this is where you say, "No Scotsman would do such a thing", and then, when the culprit turns out to be a Scotsman, you say, "No true Scotsman would do such a thing". Would you have classified the happiness of cocaine as "happiness", if someone had asked you in another context?

Admittedly, picking "happiness" as the optimization target of the AI makes it slightly more difficult to construct counterexamples: no matter what you pick, the one can say, "Oh, but if people saw that happen, they would be unhappy, so the AI won't do it." But this general response gives us the counterexample: what if the AI has to choose between a course of action that leads people to believe a pleasant fiction, or a course of action that leads to people knowing an unpleasant truth?

Suppose you believe that your daughter has gone on a one-way, near-lightspeed trip to the Hercules supercluster, meaning that you're exceedingly unlikely to ever hear from her again. This is a little sad, but you're proud of her - someone's got to colonize the place, turn it into a human habitation, before the expansion of the universe separates it from us. It's not as if she's dead - now that would make you sad.

And now suppose that the colony ship strikes a landmine, or something, and is lost with all on board. Should the AI tell you this? If all the AI values is happiness, why would it? You'll be sad then, and the AI doesn't care about truth or lies, just happiness.

Is that "no true happiness"? But it was true happiness before, when the ship was still out there. Can the difference between an instance of the "happiness" concept, and a non-instance of the "happiness" concept, as applied to a single individual, depend on the state of a system light-years away? That would be rather an extreme case of "no true Scotsman", if so - and by the time you factor in all the other behaviors you want out of this word "happiness", including times when being sad is the right thing to do, and the fact that you can't just rewrite brains to be happy, it's pretty clear that "happiness" is just a convenient stand-in for "good", and that everything which is not good is being rejected as an instance of "happy" and everything which is good is being accepted as an instance of "happy", even if it means being sad. And at this point you just have the AI which does exactly what it should do - which has been hooked up directly to Utility - and that's not a system to mention lightly; pretending that "happiness" is your stand-in for Utility doesn't begin to address the issues.

So if we leave aside this dodge, and consider the sort of happiness that would go along with smiling humans - ordinary psychological happiness - then no, you wouldn't want to switch on the superintelligence that always and only optimized for happiness. For this would be the dreaded Maximum Fun Device. The SI might lie to you, to keep you happy; even if it were a great lie, traded off against a small happiness, always and uncaringly the SI would choose the lie. The SI might rewire your brain, to ensure maximum happiness. The SI might kill off all the humans, and replace us with some different form of sentient life that had no philosophical objections to being always happy all the time in a little jar. For the qualitative diagram contains no mention of death as a bad thing, only happiness as a good, and the dead are not unhappy. (Note again how all these failures would tend to manifest, not during the AI's early infrahuman stages, but after it was too late.)

The generalized form of the problem, is that being in the presence of a superintelligence that shares some but not all of your terminal values, is not necessarily a good thing.

You didn't deliberately intend to completely change the 32-bit XOR checksum of your monitor's pixel display, when you clicked through to this webpage. But you did. It wasn't a property that it would have occurred to you to compute, because it wasn't a property that it would occur to you to care about. Deep Blue, in the course of winning its game against Kasparov, didn't care particularly about "the number of pieces on white squares minus the number of pieces on black squares", which changed throughout the game - not because Deep Blue was trying to change it, but because Deep Blue was exerting its optimization power on the gameboard and changing the gameboard, and so was Kasparov, and neither of them particularly cared about that property I have just specified. An optimization process that cares only about happiness, that squeezes the future into regions ever-richer in happiness, may not hate the truth; but it won't notice if it squeezes truth out of the world, either. There are many truths that make us sad - but the optimizer may not even care that much; it may just not notice, in passing, as it steers away from human knowledge.

On an ordinary human scale, and in particular, as a matter of qualitative reasoning, we usually assume that what we do has little in the way of side effects, unless otherwise specified. In part, this is because we will visualize things concretely, and on-the-fly spot the undesirable side effects - undesirable by any criterion that we care about, not just undesirable in the sense of departing from the original qualitative plan - and choose a different implementation instead. Or we can rely on our ability to react-on-the-fly. But as human technology grows more powerful, it tends to have more side effects, more knock-on effects and consequences, because it does bigger things whose effects we aren't controlling all by hand. An infrahuman AI that can only exert a weak influence on the world, and that makes a few people happy, will seem to be working as its designer thought an AI should work; it is only when that AI is stronger that it can squeeze the future so powerfully as to potentially squeeze out anything not explicitly protected in its utility function.

Though I don't intend to commit the logical fallacy of generalizing from fictional evidence, a nod here is due to Jack Williamson, author of With Folded Hands, whose AIs are "to serve and protect, and guard men from harm", which leads to the whole human species being kept in playpens, and lobotomized if that tends to make them unhappy.

The original phrasing of this old short story - "guard men from harm" - actually suggests another way to illustrate the point: suppose the AI cared only about the happiness of human males? Now to be sure, many men are made happy by seeing the women around them happy, wives and daughters and sisters, and so at least some females of the human species might not end up completely forlorn - but somehow, this doesn't seem to me like an optimal outcome.

Just like you wouldn't want an AI to optimize for only some of the humans, you wouldn't want an AI to optimize for only some of the values. And, as I keep emphasizing for exactly this reason, we've got a lot of values.

These then are three problems, with strategies of Friendliness built upon qualitative reasoning that seems to imply a positive link to utility:

The fragility of normal causal links when a superintelligence searches for more efficient paths through time;

The superexponential vastness of conceptspace, and the unnaturalness of the boundaries of our desires;

And all that would be lost, if success is less than complete, and a superintelligence squeezes the future without protecting everything of value in it.

Referenced by: Dreams of Friendliness • The Weak Inside View • Growing Up is Hard

Original with comments: Qualitative Strategies of Friendliness

Dreams of Friendliness

Eliezer Yudkowsky, 31 August 2008 01:20AM

Continuation of: Qualitative Strategies of Friendliness

Yesterday I described three classes of deep problem with qualitative-physics-like strategies for building nice AIs - e.g., the AI is reinforced by smiles, and happy people smile, therefore the AI will tend to act to produce happiness. In shallow form, three instances of the three problems would be:

1. Ripping people's faces off and wiring them into smiles;

2. Building lots of tiny agents with happiness counters set to large numbers;

3. Killing off the human species and replacing it with a form of sentient life that has no objections to being happy all day in a little jar.

And the deep forms of the problem are, roughly:

1. A superintelligence will search out alternate causal pathways to its goals than the ones you had in mind;

2. The boundaries of moral categories are not predictively natural entities;

3. Strong optimization for only some humane values, does not imply a good total outcome.

But there are other ways, and deeper ways, of viewing the failure of qualitative-physics-based Friendliness strategies.

Every now and then, someone proposes the Oracle AI strategy: "Why not just have a superintelligence that answers human questions, instead of acting autonomously in the world?"

Sounds pretty safe, doesn't it? What could possibly go wrong?

Well... if you've got any respect for Murphy's Law, the power of superintelligence, and human stupidity, then you can probably think of quite a few things that could go wrong with this scenario. Both in terms of how a naive implementation could fail - e.g., universe tiled with tiny users asking tiny questions and receiving fast, non-resource-intensive answers - and in terms of what could go wrong even if the basic scenario worked.

But let's just talk about the structure of the AI.

When someone reinvents the Oracle AI, the most common opening remark runs like this:

"Why not just have the AI answer questions, instead of trying to do anything? Then it wouldn't need to be Friendly. It wouldn't need any goals at all. It would just answer questions."

To which the reply is that the AI needs goals in order to decide how to think: that is, the AI has to act as a powerful optimization process in order to plan its acquisition of knowledge, effectively distill sensory information, pluck "answers" to particular questions out of the space of all possible responses, and of course, to improve its own source code up to the level where the AI is a powerful intelligence. All these events are "improbable" relative to random organizations of the AI's RAM, so the AI has to hit a narrow target in the space of possibilities to make superintelligent answers come out.

Now, why might one think that an Oracle didn't need goals? Because on a human level, the term "goal" seems to refer to those times when you said, "I want to be promoted", or "I want a cookie", and when someone asked you "Hey, what time is it?" and you said "7:30" that didn't seem to involve any goals. Implicitly, you wanted to answer the question; and implicitly, you had a whole, complicated, functionally optimized brain that let you answer the question; and implicitly, you were able to do so because you looked down at your highly optimized watch, that you bought with money, using your skill of turning your head, that you acquired by virtue of curious crawling as an infant. But that all takes place in the invisible background; it didn't feel like you wanted anything.

Thanks to empathic inference, which uses your own brain as an unopened black box to predict other black boxes, it can feel like "question-answering" is a detachable thing that comes loose of all the optimization pressures behind it - even the existence of a pressure to answer questions!

Problem 4: Qualitative reasoning about AIs often revolves around some nodes described by empathic inferences. This is a bad thing: for previously described reasons; and because it leads you to omit other nodes of the graph and their prerequisites and consequences; and because you may find yourself thinking things like, "But the AI has to cooperate to get a cookie, so now it will be cooperative" where "cooperation" is a boundary in concept-space drawn the way you would prefer to draw it... etc.

Anyway: the AI needs a goal of answering questions, and that has to give rise to subgoals of choosing efficient problem-solving strategies, improving its code, and acquiring necessary information. You can quibble about terminology, but the optimization pressure has to be there, and it has to be very powerful, measured in terms of how small a target it can hit within a large design space.

Powerful optimization pressures are scary things to be around. Look at what natural selection inadvertently did to itself - dooming the very molecules of DNA - in the course of optimizing a few Squishy Things to make hand tools and outwit each other politically. Humans, though we were optimized only according to the criterion of replicating ourselves, now have their own psychological drives executing as adaptations. The result of humans optimized for replication is not just herds of humans; we've altered much of Earth's land area with our technological creativity. We've even created some knock-on effects that we wish we hadn't, because our minds aren't powerful enough to foresee all the effects of the most powerful technologies we're smart enough to create.

My point, however, is that when people visualize qualitative FAI strategies, they generally assume that only one thing is going on, the normal / modal / desired thing. (See also: planning fallacy.) This doesn't always work even for picking up a rock and throwing it. But it works rather a lot better for throwing rocks than unleashing powerful optimization processes.

Problem 5: When humans use qualitative reasoning, they tend to visualize a single line of operation as typical - everything operating the same way it usually does, no exceptional conditions, no interactions not specified in the graph, all events firmly inside their boundaries. This works a lot better for dealing with boiling kettles, than for dealing with minds faster and smarter than your own.

If you can manage to create a full-fledged Friendly AI with full coverage of humane (renormalized human) values, then the AI is visualizing the consequences of its acts, caring about the consequences you care about, and avoiding plans with consequences you would prefer to exclude. A powerful optimization process, much more powerful than you, that doesn't share your values, is a very scary thing - even if it only "wants to answer questions", and even if it doesn't just tile the universe with tiny agents having simple questions answered.

I don't mean to be insulting, but human beings have enough trouble controlling the technologies that they're smart enough to invent themselves.

I sometimes wonder if maybe part of the problem with modern civilization is that politicians can press the buttons on nuclear weapons that they couldn't have invented themselves - not that it would be any better if we gave physicists political power that they weren't smart enough to obtain themselves - but the point is, our button-pressing civilization has an awful lot of people casting spells that they couldn't have written themselves. I'm not saying this is a bad thing and we should stop doing it, but it does have consequences. The thought of humans exerting detailed control over literally superhuman capabilities - wielding, with human minds, and in the service of merely human strategies, powers that no human being could have invented - doesn't fill me with easy confidence.

With a full-fledged, full-coverage Friendly AI acting in the world - the impossible-seeming full case of the problem - the AI itself is managing the consequences.

Is the Oracle AI thinking about the consequences of answering the questions you give it? Does the Oracle AI care about those consequences the same way you do, applying all the same values, to warn you if anything of value is lost?

What need has an Oracle for human questioners, if it knows what questions we should ask? Why not just unleash the should function?

See also the notion of an "AI-complete" problem. Analogously, any Oracle into which you can type the English question "What is the code of an AI that always does the right thing?" must be FAI-complete.

Problem 6: Clever qualitative-physics-type proposals for bouncing this thing off the AI, to make it do that thing, in a way that initially seems to avoid the Big Scary Intimidating Confusing Problems that are obviously associated with full-fledged Friendly AI, tend to just run into exactly the same problem in slightly less obvious ways, concealed in Step 2 of the proposal.

(And likewise you run right back into the intimidating problem of precise self-optimization, so that the Oracle AI can execute a billion self-modifications one after the other, and still just answer questions at the end; you're not avoiding that basic challenge of Friendly AI either.)

But the deepest problem with qualitative physics is revealed by a proposal that comes earlier in the standard conversation, at the point when I'm talking about side effects of powerful optimization processes on the world:

"We'll just keep the AI in a solid box, so it can't have any effects on the world except by how it talks to the humans."

I explain the AI-Box Experiment (see also That Alien Message); even granting the untrustworthy premise that a superintelligence can't think of any way to pass the walls of the box which you weren't smart enough to cover, human beings are not secure systems. Even against other humans, often, let alone a superintelligence that might be able to hack through us like Windows 98; when was the last time you downloaded a security patch to your brain?

"Okay, so we'll just give the AI the goal of not having any effects on the world except from how it answers questions. Sure, that requires some FAI work, but the goal system as a whole sounds much simpler than your Coherent Extrapolated Volition thingy."

What - no effects?

"Yeah, sure. If it has any effect on the world apart from talking to the programmers through the legitimately defined channel, the utility function assigns that infinite negative utility. What's wrong with that?"

When the AI thinks, that has a physical embodiment. Electrons flow through its transistors, moving around. If it has a hard drive, the hard drive spins, the read/write head moves. That has gravitational effects on the outside world.

"What? Those effects are too small! They don't count!"

The physical effect is just as real as if you shot a cannon at something - yes, might not notice, but that's just because our vision is bad at small length-scales. Sure, the effect is to move things around by 10^whatever Planck lengths, instead of the 10^more Planck lengths that you would consider as "counting". But spinning a hard drive can move things just outside the computer, or just outside the room, by whole neutron diameters -

"So? Who cares about a neutron diameter?"

- and by quite standard chaotic physics, that effect is liable to blow up. The butterfly that flaps its wings and causes a hurricane, etc. That effect may not be easily controllable but that doesn't mean the chaotic effects of small perturbations are not large.

But in any case, your proposal was to give the AI a goal of having no effect on the world, apart from effects that proceed through talking to humans. And this is impossible of fulfillment; so no matter what it does, the AI ends up with infinite negative utility - how is its behavior defined in this case? (In this case I picked a silly initial suggestion - but one that I have heard made, as if infinite negative utility were like an exclamation mark at the end of a command given a human employee. Even an unavoidable tiny probability of infinite negative utility trashes the goal system.)

Why would anyone possibly think that a physical object like an AI, in our highly interactive physical universe, containing hard-to-shield forces like gravitation, could avoid all effects on the outside world?

And this, I think, reveals what may be the deepest way of looking at the problem:

Problem 7: Human beings model a world made up of objects, attributes, and noticeworthy events and interactions, identified by their categories and values. This is only our own weak grasp on reality; the real universe doesn't look like that. Even if a different mind saw a similar kind of exposed surface to the world, it would still see a different exposed surface.

Sometimes human thought seems a lot like it tries to grasp the universe as... well, as this big XML file, AI.goal == smile, human.smile == yes, that sort of thing. Yes, I know human world-models are more complicated than XML. (And yes, I'm also aware that what I wrote looks more like Python than literal XML.) But even so.

What was the one thinking, who proposed an AI whose behaviors would be reinforced by human smiles, and who reacted with indignation to the idea that a superintelligence could "mistake" a tiny molecular smileyface for a "real" smile? Probably something along the lines of, "But in this case, human.smile == 0, so how could a superintelligence possibly believe human.smile == 1?"

For the weak grasp that our mind obtains on the high-level surface of reality, seems to us like the very substance of the world itself.

Unless we make a conscious effort to think of reductionism, and even then, it's not as if thinking "Reductionism!" gives us a sudden apprehension of quantum mechanics.

So if you have this, as it were, XML-like view of reality, then it's easy enough to think you can give the AI a goal of having no effects on the outside world; the "effects" are like discrete rays of effect leaving the AI, that result in noticeable events like killing a cat or something, and the AI doesn't want to do this, so it just switches the effect-rays off; and by the assumption of default independence, nothing else happens.

Mind you, I'm not saying that you couldn't build an Oracle. I'm saying that the problem of giving it a goal of "don't do anything to the outside world" "except by answering questions" "from the programmers" "the way the programmers meant them", in such fashion as to actually end up with an Oracle that works anything like the little XML-ish model in your head, is a big nontrivial Friendly AI problem. The real world doesn't have little discreet effect-rays leaving the AI, and the real world doesn't have ontologically fundamental programmer.question objects, and "the way the programmers meant them" isn't a natural category.

And this is more important for dealing with superintelligences than rocks, because the superintelligences are going to parse up the world in a different way. They may not perceive reality directly, but they'll still have the power to perceive it differently. A superintelligence might not be able to tag every atom in the solar system, but it could tag every biological cell in the solar system (consider that each of your cells contains its own mitochondrial power engine and a complete copy of your DNA). It used to be that human beings didn't even know they were made out of cells. And if the universe is a bit more complicated than we think, perhaps the superintelligence we build will make a few discoveries, and then slice up the universe into parts we didn't know existed - to say nothing of us being able to model them in our own minds! How does the instruction to "do the right thing" cross that kind of gap?

There is no nontechnical solution to Friendly AI.

That is: There is no solution that operates on the level of qualitative physics and empathic models of agents.

That's all just a dream in XML about a universe of quantum mechanics. And maybe that dream works fine for manipulating rocks over a five-minute timespan; and sometimes okay for getting individual humans to do things; it often doesn't seem to give us much of a grasp on human societies, or planetary ecologies; and as for optimization processes more powerful than you are... it really isn't going to work.

(Incidentally, the most epically silly example of this that I can recall seeing, was a proposal to (IIRC) keep the AI in a box and give it faked inputs to make it believe that it could punish its enemies, which would keep the AI satisfied and make it go on working for us. Just some random guy with poor grammar on an email list, but still one of the most epic FAIls I recall seeing.)

Referenced by: My Bayesian Enlightenment • Shut up and do the impossible! • Protected From Myself • Ethical Injunctions

Original with comments: Dreams of Friendliness

Brief Break

Eliezer Yudkowsky, 31 August 2008 04:00PM

I've been feeling burned on Overcoming Bias lately, meaning that I take too long to write my posts, which decreases the amount of recovery time, making me feel more burned, etc.

So I'm taking at most a one-week break. I'll post small units of rationality quotes each day, so as to not quite abandon you. I may even post some actual writing, if I feel spontaneous, but definitely not for the next two days; I have to enforce this break upon myself.

When I get back, my schedule calls for me to finish up the Anthropomorphism sequence, and then talk about Marcus Hutter's AIXI, which I think is the last brain-malfunction-causing subject I need to discuss. My posts should then hopefully go back to being shorter and easier.

Hey, at least I got through over a solid year of posts without taking a vacation.

Original with comments: Brief Break

Rationality Quotes 12

Eliezer Yudkowsky, 01 September 2008 08:00PM

"Even if I had an objective proof that you don't find it unpleasant when you stick your hand in a fire, I still think you’d pull your hand out at the first opportunity."
-- John K Clark

"So often when one level of delusion goes away, another one more subtle comes in its place."
-- Rational Buddhist

"Your denial of the importance of objectivity amounts to announcing your intention to lie to us. No-one should believe anything you say."
-- John McCarthy

"How exactly does one 'alter reality'? If I eat an apple have I altered reality? Or maybe you mean to just give the appearance of altering reality."
-- JoeDad

"Promoting less than maximally accurate beliefs is an act of sabotage. Don't do it to anyone unless you'd also slash their tires."
-- Black Belt Bayesian

Original with comments: Rationality Quotes 12

Rationality Quotes 13

Eliezer Yudkowsky, 02 September 2008 04:00PM

"You can only compromise your principles once. After then you don't have any."
-- Smug Lisp Weeny

"If you want to do good, work on the technology, not on getting power."
-- John McCarthy

"If you’re interested in being on the right side of disputes, you will refute your opponents’ arguments. But if you’re interested in producing truth, you will fix your opponents’ arguments for them. To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse."
-- Black Belt Bayesian

"I normally thought of "God!" as a disclaimer, or like the MPAA rating you see just before a movie starts: it told me before I continued into conversation with that person, that that person had limitations to their intellectual capacity or intellectual honesty."
-- Mike Barskey

"It is the soldier, not the reporter, who has given us freedom of the press. It is the soldier, not the poet, who has given us freedom of speech. It is the soldier, not the campus organizer, who has given us the freedom to demonstrate. It is the soldier, not the lawyer, who has given us the right to a fair trial. It is the soldier, who salutes the flag, who serves under the flag, and whose coffin is draped by the flag, who allows the protester to burn the flag."
-- Father Dennis Edward O'Brien, USMC

Original with comments: Rationality Quotes 13

The True Prisoner's Dilemma

Eliezer Yudkowsky, 03 September 2008 09:34PM

It occurred to me one day that the standard visualization of the Prisoner's Dilemma is fake.

The core of the Prisoner's Dilemma is this symmetric payoff matrix:

1: C

1: D

2: C

(3, 3)

(5, 0)

2: D

(0, 5)

(2, 2)

Player 1, and Player 2, can each choose C or D. 1 and 2's utility for the final outcome is given by the first and second number in the pair. For reasons that will become apparent, "C" stands for "cooperate" and D stands for "defect".

Observe that a player in this game (regarding themselves as the first player) has this preference ordering over outcomes: (D, C) > (C, C) > (D, D) > (C, D).

D, it would seem, dominates C: If the other player chooses C, you prefer (D, C) to (C, C); and if the other player chooses D, you prefer (D, D) to (C, D). So you wisely choose D, and as the payoff table is symmetric, the other player likewise chooses D.

If only you'd both been less wise! You both prefer (C, C) to (D, D). That is, you both prefer mutual cooperation to mutual defection.

The Prisoner's Dilemma is one of the great foundational issues in decision theory, and enormous volumes of material have been written about it. Which makes it an audacious assertion of mine, that the usual way of visualizing the Prisoner's Dilemma has a severe flaw, at least if you happen to be human.

The classic visualization of the Prisoner's Dilemma is as follows: you are a criminal, and you and your confederate in crime have both been captured by the authorities.

Independently, without communicating, and without being able to change your mind afterward, you have to decide whether to give testimony against your confederate (D) or remain silent (C).

Both of you, right now, are facing one-year prison sentences; testifying (D) takes one year off your prison sentence, and adds two years to your confederate's sentence.

Or maybe you and some stranger are, only once, and without knowing the other player's history, or finding out who the player was afterward, deciding whether to play C or D, for a payoff in dollars matching the standard chart.

And, oh yes - in the classic visualization you're supposed to pretend that you're entirely selfish, that you don't care about your confederate criminal, or the player in the other room.

It's this last specification that makes the classic visualization, in my view, fake.

You can't avoid hindsight bias by instructing a jury to pretend not to know the real outcome of a set of events. And without a complicated effort backed up by considerable knowledge, a neurologically intact human being cannot pretend to be genuinely, truly selfish.

We're born with a sense of fairness, honor, empathy, sympathy, and even altruism - the result of our ancestors adapting to play the iterated Prisoner's Dilemma. We don't really, truly, absolutely and entirely prefer (D, C) to (C, C), though we may entirely prefer (C, C) to (D, D) and (D, D) to (C, D). The thought of our confederate spending three years in prison, does not entirely fail to move us.

In that locked cell where we play a simple game under the supervision of economic psychologists, we are not entirely and absolutely unsympathetic for the stranger who might cooperate. We aren't entirely happy to think what we might defect and the stranger cooperate, getting five dollars while the stranger gets nothing.

We fixate instinctively on the (C, C) outcome and search for ways to argue that it should be the mutual decision: "How can we ensure mutual cooperation?" is the instinctive thought. Not "How can I trick the other player into playing C while I play D for the maximum payoff?"

For someone with an impulse toward altruism, or honor, or fairness, the Prisoner's Dilemma doesn't really have the critical payoff matrix - whatever the financial payoff to individuals. (C, C) > (D, C), and the key question is whether the other player sees it the same way.

And no, you can't instruct people being initially introduced to game theory to pretend they're completely selfish - any more than you can instruct human beings being introduced to anthropomorphism to pretend they're expected paperclip maximizers.

To construct the True Prisoner's Dilemma, the situation has to be something like this:

Player 1: Human beings, Friendly AI, or other humane intelligence.

Player 2: UnFriendly AI, or an alien that only cares about sorting pebbles.

Let's suppose that four billion human beings - not the whole human species, but a significant part of it - are currently progressing through a fatal disease that can only be cured by substance S.

However, substance S can only be produced by working with a paperclip maximizer from another dimension - substance S can also be used to produce paperclips. The paperclip maximizer only cares about the number of paperclips in its own universe, not in ours, so we can't offer to produce or threaten to destroy paperclips here. We have never interacted with the paperclip maximizer before, and will never interact with it again.

Both humanity and the paperclip maximizer will get a single chance to seize some additional part of substance S for themselves, just before the dimensional nexus collapses; but the seizure process destroys some of substance S.

The payoff matrix is as follows:

1: C

1: D

2: C

(2 billion human lives saved, 2 paperclips gained)

(+3 billion lives, +0 paperclips)

2: D

(+0 lives, +3 paperclips)

(+1 billion lives, +1 paperclip)

I've chosen this payoff matrix to produce a sense of indignation at the thought that the paperclip maximizer wants to trade off billions of human lives against a couple of paperclips. Clearly the paperclip maximizer should just let us have all of substance S; but a paperclip maximizer doesn't do what it should, it just maximizes paperclips.

In this case, we really do prefer the outcome (D, C) to the outcome (C, C), leaving aside the actions that produced it. We would vastly rather live in a universe where 3 billion humans were cured of their disease and no paperclips were produced, rather than sacrifice a billion human lives to produce 2 paperclips. It doesn't seem right to cooperate, in a case like this. It doesn't even seem fair - so great a sacrifice by us, for so little gain by the paperclip maximizer? And let us specify that the paperclip-agent experiences no pain or pleasure - it just outputs actions that steer its universe to contain more paperclips. The paperclip-agent will experience no pleasure at gaining paperclips, no hurt from losing paperclips, and no painful sense of betrayal if we betray it.

What do you do then? Do you cooperate when you really, definitely, truly and absolutely do want the highest reward you can get, and you don't care a tiny bit by comparison about what happens to the other player? When it seems right to defect even if the other player cooperates?

That's what the payoff matrix for the true Prisoner's Dilemma looks like - a situation where (D, C) seems righter than (C, C).

But all the rest of the logic - everything about what happens if both agents think that way, and both agents defect - is the same. For the paperclip maximizer cares as little about human deaths, or human pain, or a human sense of betrayal, as we care about paperclips. Yet we both prefer (C, C) to (D, D).

So if you've ever prided yourself on cooperating in the Prisoner's Dilemma... or questioned the verdict of classical game theory that the "rational" choice is to defect... then what do you say to the True Prisoner's Dilemma above?

Referenced by: The Truly Iterated Prisoner's Dilemma • Prices or Bindings? • In Praise of Boredom • Sympathetic Minds • Your Price for Joining • Bayesians vs. Barbarians • Timeless Decision Theory: Problems I Can't Solve • Timeless Decision Theory and Meta-Circular Decision Theory

Original with comments: The True Prisoner's Dilemma

The Truly Iterated Prisoner's Dilemma

Eliezer Yudkowsky, 04 September 2008 06:00PM

Followup to: The True Prisoner's Dilemma

For everyone who thought that the rational choice in yesterday's True Prisoner's Dilemma was to defect, a follow-up dilemma:

Suppose that the dilemma was not one-shot, but was rather to be repeated exactly 100 times, where for each round, the payoff matrix looks like this:

Humans: C

Humans: D

Paperclipper: C

(2 million human lives saved, 2 paperclips gained)

(+3 million lives, +0 paperclips)

Paperclipper: D

(+0 lives, +3 paperclips)

(+1 million lives, +1 paperclip)

As most of you probably know, the king of the classical iterated Prisoner's Dilemma is Tit for Tat, which cooperates on the first round, and on succeeding rounds does whatever its opponent did last time. But what most of you may not realize, is that, if you know when the iteration will stop, Tit for Tat is - according to classical game theory - irrational.

Why? Consider the 100th round. On the 100th round, there will be no future iterations, no chance to retaliate against the other player for defection. Both of you know this, so the game reduces to the one-shot Prisoner's Dilemma. Since you are both classical game theorists, you both defect.

Now consider the 99th round. Both of you know that you will both defect in the 100th round, regardless of what either of you do in the 99th round. So you both know that your future payoff doesn't depend on your current action, only your current payoff. You are both classical game theorists. So you both defect.

Now consider the 98th round...

With humanity and the Paperclipper facing 100 rounds of the iterated Prisoner's Dilemma, do you really truly think that the rational thing for both parties to do, is steadily defect against each other for the next 100 rounds?

Referenced by: Today's Inspirational Tale • Singletons Rule OK

Original with comments: The Truly Iterated Prisoner's Dilemma

Rationality Quotes 14

Eliezer Yudkowsky, 05 September 2008 08:16PM

"As for the little green men... they don't want us to know about them, so they refrain from making contact... then they do silly aerobatics displays within radar range of military bases... with their exterior lights on... if that's extraterrestrial intelligence, I'm not sure I want to know what extraterrestrial stupidity looks like."
-- Russell Wallace

"Characterizing male status-seeking as egotistical is like characterizing bonobo promiscuity as unchaste."
-- Liza May

"Introducing a technology is not a neutral act--it is profoundly revolutionary. If you present a new technology to the world you are effectively legislating a change in the way we all live. You are changing society, not some vague democratic process. The individuals who are driven to use that technology by the disparities of wealth and power it creates do not have a real choice in the matter. So the idea that we are giving people more freedom by developing technologies and then simply making them available is a dangerous illusion."
-- Karl Schroeder

"Hans Riesel held a Mersenne record for 14 days in the 50's, calculated using the first Swedish computer. My old highschool computing teacher had worked as a student on the system and had managed to crush his foot when a byte fell out of its rack and onto him."
-- Anders Sandberg

"Gentlemen, I do not mind being contradicted, and I am unperturbed when I am attacked, but I confess I have slight misgivings when I hear myself being explained."
-- Lord Balfour, to the English Parliament

Original with comments: Rationality Quotes 14

Rationality Quotes 15

Eliezer Yudkowsky, 06 September 2008 06:53PM

"Who thinks they're not open-minded? Our hypothetical prim miss from the suburbs thinks she's open-minded. Hasn't she been taught to be? Ask anyone, and they'll say the same thing: they're pretty open-minded, though they draw the line at things that are really wrong."
-- Paul Graham

"In the same way that we need statesmen to spare us the abjection of exercising power, we need scholars to spare us the abjection of learning."
-- Jean Baudrillard

"Because giftedness is not to be talked about, no one tells high-IQ children explicitly, forcefully and repeatedly that their intellectual talent is a gift. That they are not superior human beings, but lucky ones. That the gift brings with it obligations to be worthy of it."
-- Charles Murray

"The popular media can only handle ideas expressible in proto-language, not ideas requiring nested phrase-structure syntax for their exposition."
-- Ben Goertzel

"The best part about math is that, if you have the right answer and someone disagrees with you, it really is because they're stupid."
-- Quotes from Honors Linear Algebra

"Long-Term Capital Management had faith in diversification. Its history serves as ample notification that eggs in different baskets can and do all break at the same time."
-- Craig L. Howe

"Accountability is about one person taking responsibility. If two people are accountable for the same decision, no one is really accountable."
-- Glyn Holton

Original with comments: Rationality Quotes 15

Rationality Quotes 16

Eliezer Yudkowsky, 08 September 2008 03:36AM

"I read a lot of fantasy and have wondered sometimes, not so much what I would do in a fantasy setting, but what the book characters would do in the real world."
-- Romana

"That's the thing that's always fascinated me about Go. It is essentially an extremely simple game gone terribly, terribly wrong."
-- Amorymeltzer

"Dealing with the sheer of volume of "stuff" available on the internet is like being a crackhead with OCD. In the course of one hour I've tweaked my fantasy baseball lineup, posted on this message board, read Yahoo news, answered my latest e-mail, downloaded guidance criteria for PAHs in soils in NY State, checked the discography of a couple of bands, sent a deliverable to a client, and checked the weather. If that isn't superstimulus I don't know what is. It's amazing how much I can do, yet accomplish so little."
-- Misanthropic

"We don't have thoughts, we are thoughts. Thoughts are not responsible for the machinery that happens to think them."
-- John K Clark

"I have known more people whose lives have been ruined by getting a Ph.D. in physics than by drugs."
-- Jonathan I. Katz

"There's no difference between a pessimist who says, "Oh, it's hopeless, so don't bother doing anything," and an optimist who says, "Don't bother doing anything, it's going to turn out fine anyway." Either way, nothing happens."
-- Yvon Chouinard

"Life moved ever outward into infinite possibilities and yet all things were perfect and finished in every single moment, their end attained."
-- David Zindell, Neverness

Original with comments: Rationality Quotes 16

Singularity Summit 2008

Eliezer Yudkowsky, 08 September 2008 11:26PM

FYI all: The Singularity Summit 2008 is coming up, 9am-5pm October 25th, 2008 in San Jose, CA. This is run by my host organization, the Singularity Institute. Speakers this year include Vernor Vinge, Marvin Minsky, the CTO of Intel, and the chair of the X Prize Foundation.

Before anyone posts any angry comments: yes, the registration costs actual money this year. The Singularity Institute has run free events before, and will run free events in the future. But while past Singularity Summits have been media successes, they haven't been fundraising successes up to this point. So Tyler Emerson et. al. are trying it a little differently. TANSTAAFL.

Lots of speakers talking for short periods this year. I'm intrigued by that format. We'll see how it goes.

Press release:

SAN JOSE, CA – Singularity Summit 2008: Opportunity, Risk, Leadership takes place October 25 at the intimate Montgomery Theater in San Jose, CA, the Singularity Institute for Artificial Intelligence announced. Now in its third year, the Singularity Summit gathers the smartest people around to explore the biggest idea of our time: the Singularity.

Keynotes will include Ray Kurzweil, updating his predictions in The Singularity is Near, and Intel CTO Justin Rattner, who will examine the Singularity’s plausibility. At the Intel Developer Forum on August 21, 2008, he explained why he thinks the gap between humans and machines will close by 2050. “Rather than look back, we’re going to look forward 40 years,” said Rattner. “It’s in that future where many people think that machine intelligence will surpass human intelligence.”

“The acceleration of technological progress has been the central feature of this century,” said computer scientist Dr. Vernor Vinge in a seminal paper in 1993. “We are on the edge of change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater than human intelligence.”

Singularity Summit 2008 will feature an impressive lineup:

- Dr. Ruzena Bajcsy, pioneering AI and robotics researcher

- Dr. Eric Baum, AI researcher, author of What is Thought?

- Marshall Brain, founder of HowStuffWorks.com, author of Robotic Nation

- Dr. Cynthia Breazeal, robotics professor at MIT, creator of Kismet

- Dr. Peter Diamandis, chair and CEO of X PRIZE Foundation

- Esther Dyson, entrepreneur, investor, philanthropist

- Dr. Pete Estep, chair and CSO of Innerspace Foundation

- Dr. Neil Gershenfeld, director of MIT Center for Bits and Atoms, author of Fab

- Dr. Ben Goertzel, CEO of Novamente, director of research at SIAI

- John Horgan, science journalist, author of The Undiscovered Mind

- Ray Kurzweil, CEO of Kurzweil Technologies, author of The Singularity is Near

- Dr. James Miller, author of forthcoming book on Singularity economics

- Dr. Marvin Minsky, one of AI’s founding fathers, author of The Emotion Machine

- Dr. Dharmendra Modha, cognitive computing lead at IBM Almaden Research Center

- Bob Pisani, news correspondent for financial news network CNBC

- Justin Rattner, VP and CTO of Intel

- Nova Spivack, CEO of Radar Networks, creator of Twine semantic-web application

- Peter Thiel, president of Clarium, managing partner of Founders Fund

- Dr. Vernor Vinge, author of original paper on the technological Singularity

- Eliezer Yudkowsky, research fellow at SIAI, author of Creating Friendly AI

- Glenn Zorpette, executive editor of IEEE Spectrum

Registration details are available at http://www.singularitysummit.com/registration/.

About the Singularity Summit

Each year, the Singularity Summit attracts a unique audience to the Bay Area, with visionaries from business, science, technology, philanthropy, the arts, and more. Participants learn where humanity is headed, meet the people leading the way, and leave inspired to create a better world. “The Singularity Summit is the premier conference on the Singularity,” Kurzweil said. “As we get closer to the Singularity, each year’s conference is better than the last.”

The Summit was founded in 2006 by long-term philanthropy executive Tyler Emerson, inventor Ray Kurzweil, and investor Peter Thiel. Its purpose is to bring together and build a visionary community to further dialogue and action on complex, long-term issues that may transform the world. Its host organization is the Singularity Institute for Artificial Intelligence, a 501(c)(3) nonprofit organization studying the benefits and risks of advanced artificial intelligence systems.

Singularity Summit 2008 partners include Clarium Capital, Cartmell Holdings, Twine, Powerset, United Therapeutics, KurzweilAI.net, IEEE Spectrum, DFJ, X PRIZE Foundation, Long Now Foundation, Foresight Nanotech Institute, Novamente, SciVestor, Robotics Trends, and MINE.

Original with comments: Singularity Summit 2008

Points of Departure

Eliezer Yudkowsky, 09 September 2008 09:18PM

Followup to: Anthropomorphic Optimism

If you've watched Hollywood sci-fi involving supposed robots, androids, or AIs, then you've seen AIs that are depicted as "emotionless". In the olden days this was done by having the AI speak in a monotone pitch - while perfectly stressing the syllables, of course. (I could similarly go on about how AIs that disastrously misinterpret their mission instructions, never seem to need help parsing spoken English.) You can also show that an AI is "emotionless" by having it notice an emotion with a blatant somatic effect, like tears or laughter, and ask what it means (though of course the AI never asks about sweat or coughing).

If you watch enough Hollywood sci-fi, you'll run into all of the following situations occurring with supposedly "emotionless" AIs:

1. An AI that malfunctions or otherwise turns evil, instantly acquires all of the negative human emotions - it hates, it wants revenge, and feels the need to make self-justifying speeches.

2. Conversely, an AI that turns to the Light Side, gradually acquires a full complement of human emotions.

3. An "emotionless" AI suddenly exhibits human emotion when under exceptional stress; e.g. an AI that displays no reaction to thousands of deaths, suddenly showing remorse upon killing its creator.

4. An AI begins to exhibit signs of human emotion, and refuses to admit it.

Now, why might a Hollywood scriptwriter make those particular mistakes?

These mistakes seem to me to bear the signature of modeling an Artificial Intelligence as an emotionally repressed human.

At least, I can't seem to think of any other simple hypothesis that explains the behaviors 1-4 above. The AI that turns evil has lost its negative-emotion-suppressor, so the negative emotions suddenly switch on. The AI that turns from mechanical agent to good agent, gradually loses the emotion-suppressor keeping it mechanical, so the good emotions rise to the surface. Under exceptional stress, of course the emotional repression that keeps the AI "mechanical" will immediately break down and let the emotions out. But if the stress isn't so exceptional, the firmly repressed AI will deny any hint of the emotions leaking out - that conflicts with the AI's self-image of itself as being emotionless.

It's not that the Hollywood scriptwriters are explicitly reasoning "An AI will be like an emotionally repressed human", of course; but rather that when they imagine an "emotionless AI", this is the intuitive model that forms in the background - a Standard mind (which is to say a human mind) plus an extra Emotion Suppressor.

Which all goes to illustrate yet another fallacy of anthropomorphism - treating humans as your point of departure, modeling a mind as a human plus a set of differences.

This is a logical fallacy because it warps Occam's Razor. A mind that entirely lacks chunks of brainware to implement "hate" or "kindness", is simpler - in a computational complexity sense - than a mind that has "hate" plus a "hate-suppressor", or "kindness" plus a "kindness-repressor". But if you start out with a human mind, then adding an activity-suppressor is a smaller alteration than deleting the whole chunk of brain.

It's also easier for human scriptwriters to imagine themselves repressing an emotion, pushing it back, crushing it down, then it is for them to imagine once deleting an emotion and it never coming back. The former is a mode that human minds can operate in; the latter would take neurosurgery.

But that's just a kind of anthropomorphism previously covered - the plain old ordinary fallacy of using your brain as a black box to predict something that doesn't work like it does. Here, I want to talk about the formally different fallacy of measuring simplicity in terms of the shortest diff from "normality", i.e., what your brain says a "mind" does in the absence of specific instruction otherwise, i.e., humanness. Even if you can grasp that something doesn't have to work just like a human, thinking of it as a human+diff will distort your intuitions of simplicity - your Occam-sense.

Referenced by: My Naturalistic Awakening • Above-Average AI Scientists • Economic Definition of Intelligence? • Disjunctions, Antipredictions, Etc.

Original with comments: Points of Departure

Rationality Quotes 17

Eliezer Yudkowsky, 11 September 2008 03:29AM

"We take almost all of the decisive steps in our lives as a result of slight inner adjustments of which we are barely conscious."
-- Austerlitz

"In both poker and life, you can't read people any better than they can read themselves. You can, if you’re good, very accurately determine if they think their hand is good, or if they think they know the answer to your legal question. But you can't be sure if reality differs from their perception."
-- Matt Maroon

"We should not complain about impermanence, because without impermanence, nothing is possible."
-- Thich Nhat Hanh

"I've never been happy. I have a few memories, early in life, and it sounds dramatic to say, but when I reflect on my life, the best I've ever had were brief periods when things were simply less painful."
-- Ilan Bouchard

Q: What are the "intuitive and metaphyscal arts"?
A: The gods alone know. Probably the old tired con-acts of fortune-telling and putting the hex on your neighbor's goat, glossed up with gibberish borrowed from pop science tracts in the last two centuries.
-- The Aleph Anti-FAQ

"If you build a snazzy alife sim ... you'd be a kind of bridging `first cause', and might even have the power to intervene in their lives - even obliterate their entire experienced cosmos - but that wouldn't make you a god in any interesting sense. Gods are ontologically distinct from creatures, or they're not worth the paper they're written on."
-- Damien Broderick

"NORMAL is a setting on a washing-machine."
-- Nikolai Kingsley

Original with comments: Rationality Quotes 17

Excluding the Supernatural

Eliezer Yudkowsky, 12 September 2008 12:12AM

Followup to: Reductionism, Anthropomorphic Optimism

Occasionally, you hear someone claiming that creationism should not be taught in schools, especially not as a competing hypothesis to evolution, because creationism is a priori and automatically excluded from scientific consideration, in that it invokes the "supernatural".

So... is the idea here, that creationism could be true, but even if it were true, you wouldn't be allowed to teach it in science class, because science is only about "natural" things?

It seems clear enough that this notion stems from the desire to avoid a confrontation between science and religion. You don't want to come right out and say that science doesn't teach Religious Claim X because X has been tested by the scientific method and found false. So instead, you can... um... claim that science is excluding hypothesis X a priori. That way you don't have to discuss how experiment has falsified X a posteriori.

Of course this plays right into the creationist claim that Intelligent Design isn't getting a fair shake from science - that science has prejudged the issue in favor of atheism, regardless of the evidence. If science excluded Intelligent Design a priori, this would be a justified complaint!

But let's back up a moment. The one comes to you and says: "Intelligent Design is excluded from being science a priori, because it is 'supernatural', and science only deals in 'natural' explanations."

What exactly do they mean, "supernatural"? Is any explanation invented by someone with the last name "Cohen" a supernatural one? If we're going to summarily kick a set of hypotheses out of science, what is it that we're supposed to exclude?

By far the best definition I've ever heard of the supernatural is Richard Carrier's: A "supernatural" explanation appeals to ontologically basic mental things, mental entities that cannot be reduced to nonmental entities.

This is the difference, for example, between saying that water rolls downhill because it wants to be lower, and setting forth differential equations that claim to describe only motions, not desires. It's the difference between saying that a tree puts forth leaves because of a tree spirit, versus examining plant biochemistry. Cognitive science takes the fight against supernaturalism into the realm of the mind.

Why is this an excellent definition of the supernatural? I refer you to Richard Carrier for the full argument. But consider: Suppose that you discover what seems to be a spirit, inhabiting a tree: a dryad who can materialize outside or inside the tree, who speaks in English about the need to protect her tree, et cetera. And then suppose that we turn a microscope on this tree spirit, and she turns out to be made of parts - not inherently spiritual and ineffable parts, like fabric of desireness and cloth of belief; but rather the same sort of parts as quarks and electrons, parts whose behavior is defined in motions rather than minds. Wouldn't the dryad immediately be demoted to the dull catalogue of common things?

But if we accept Richard Carrier's definition of the supernatural, then a dilemma arises: we want to give religious claims a fair shake, but it seems that we have very good grounds for excluding supernatural explanations a priori.

I mean, what would the universe look like if reductionism were false?

I previously defined the reductionist thesis as follows: human minds create multi-level models of reality in which high-level patterns and low-level patterns are separately and explicitly represented. A physicist knows Newton's equation for gravity, Einstein's equation for gravity, and the derivation of the former as a low-speed approximation of the latter. But these three separate mental representations, are only a convenience of human cognition. It is not that reality itself has an Einstein equation that governs at high speeds, a Newton equation that governs at low speeds, and a "bridging law" that smooths the interface. Reality itself has only a single level, Einsteinian gravity. It is only the Mind Projection Fallacy that makes some people talk as if the higher levels could have a separate existence - different levels of organization can have separate representations in human maps, but the territory itself is a single unified low-level mathematical object.

Suppose this were wrong.

Suppose that the Mind Projection Fallacy was not a fallacy, but simply true.

Suppose that a 747 had a fundamental physical existence apart from the quarks making up the 747.

What experimental observations would you expect to make, if you found yourself in such a universe?

If you can't come up with a good answer to that, it's not observation that's ruling out "non-reductionist" beliefs, but a priori logical incoherence. If you can't say what predictions the "non-reductionist" model makes, how can you say that experimental evidence rules it out?

My thesis is that non-reductionism is a confusion; and once you realize that an idea is a confusion, it becomes a tad difficult to envision what the universe would look like if the confusion were true. Maybe I've got some multi-level model of the world, and the multi-level model has a one-to-one direct correspondence with the causal elements of the physics? But once all the rules are specified, why wouldn't the model just flatten out into yet another list of fundamental things and their interactions? Does everything I can see in the model, like a 747 or a human mind, have to become a separate real thing? But what if I see a pattern in that new supersystem?

Supernaturalism is a special case of non-reductionism, where it is not 747s that are irreducible, but just (some) mental things. Religion is a special case of supernaturalism, where the irreducible mental things are God(s) and souls; and perhaps also sins, angels, karma, etc.

If I propose the existence of a powerful entity with the ability to survey and alter each element of our observed universe, but with the entity reducible to nonmental parts that interact with the elements of our universe in a lawful way; if I propose that this entity wants certain particular things, but "wants" using a brain composed of particles and fields; then this is not yet a religion, just a naturalistic hypothesis about a naturalistic Matrix. If tomorrow the clouds parted and a vast glowing amorphous figure thundered forth the above description of reality, then this would not imply that the figure was necessarily honest; but I would show the movies in a science class, and I would try to derive testable predictions from the theory.

Conversely, religions have ignored the discovery of that ancient bodiless thing: omnipresent in the working of Nature and immanent in every falling leaf: vast as a planet's surface and billions of years old: itself unmade and arising from the structure of physics: designing without brain to shape all life on Earth and the minds of humanity. Natural selection, when Darwin proposed it, was not hailed as the long-awaited Creator: It wasn't fundamentally mental.

But now we get to the dilemma: if the staid conventional normal boring understanding of physics and the brain is correct, there's no way in principle that a human being can concretely envision, and derive testable experimental predictions about, an alternate universe in which things are irreducibly mental. Because, if the boring old normal model is correct, your brain is made of quarks, and so your brain will only be able to envision and concretely predict things that can predicted by quarks. You will only ever be able to construct models made of interacting simple things.

People who live in reductionist universes cannot concretely envision non-reductionist universes. They can pronounce the syllables "non-reductionist" but they can't imagine it.

The basic error of anthropomorphism, and the reason why supernatural explanations sound much simpler than they really are, is your brain using itself as an opaque black box to predict other things labeled "mindful". Because you already have big, complicated webs of neural circuitry that implement your "wanting" things, it seems like you can easily describe water that "wants" to flow downhill - the one word "want" acts as a lever to set your own complicated wanting-machinery in motion.

Or you imagine that God likes beautiful things, and therefore made the flowers. Your own "beauty" circuitry determines what is "beautiful" and "not beautiful". But you don't know the diagram of your own synapses. You can't describe a nonmental system that computes the same label for what is "beautiful" or "not beautiful" - can't write a computer program that predicts your own labelings. But this is just a defect of knowledge on your part; it doesn't mean that the brain has no explanation.

If the "boring view" of reality is correct, then you can never predict anything irreducible because you are reducible. You can never get Bayesian confirmation for a hypothesis of irreducibility, because any prediction you can make is, therefore, something that could also be predicted by a reducible thing, namely your brain.

Some boxes you really can't think outside. If our universe really is Turing computable, we will never be able to concretely envision anything that isn't Turing-computable - no matter how many levels of halting oracle hierarchy our mathematicians can talk about, we won't be able to predict what a halting oracle would actually say, in such fashion as to experimentally discriminate it from merely computable reasoning.

Of course, that's all assuming the "boring view" is correct. To the extent that you believe evolution is true, you should not expect to encounter strong evidence against evolution. To the extent you believe reductionism is true, you should expect non-reductionist hypotheses to be incoherent as well as wrong. To the extent you believe supernaturalism is false, you should expect it to be inconceivable as well.

If, on the other hand, a supernatural hypothesis turns out to be true, then presumably you will also discover that it is not inconceivable.

So let us bring this back full circle to the matter of Intelligent Design:

Should ID be excluded a priori from experimental falsification and science classrooms, because, by invoking the supernatural, it has placed itself outside of natural philosophy?

I answer: "Of course not." The irreducibility of the intelligent designer is not an indispensable part of the ID hypothesis. For every irreducible God that can be proposed by the IDers, there exists a corresponding reducible alien that behaves in accordance with the same predictions - since the IDers themselves are reducible; to the extent I believe reductionism is in fact correct, which is a rather strong extent, I must expect to discover reducible formulations of all supposedly supernatural predictive models.

If we're going over the archeological records to test the assertion that Jehovah parted the Red Sea out of an explicit desire to display its superhuman power, then it makes little difference whether Jehovah is ontologically basic, or an alien with nanotech, or a Dark Lord of the Matrix. You do some archeology, find no skeletal remnants or armor at the Red Sea site, and indeed find records that Egypt ruled much of Canaan at the time. So you stamp the historical record in the Bible "disproven" and carry on. The hypothesis is coherent, falsifiable and wrong.

Likewise with the evidence from biology that foxes are designed to chase rabbits, rabbits are designed to evade foxes, and neither is designed "to carry on their species" or "protect the harmony of Nature"; likewise with the retina being designed backwards with the light-sensitive parts at the bottom; and so on through a thousand other items of evidence for splintered, immoral, incompetent design. The Jehovah model of our alien god is coherent, falsifiable, and wrong - coherent, that is, so long as you don't care whether Jehovah is ontologically basic or just an alien.

Just convert the supernatural hypothesis into the corresponding natural hypothesis. Just make the same predictions the same way, without asserting any mental things to be ontologically basic. Consult your brain's black box if necessary to make predictions - say, if you want to talk about an "angry god" without building a full-fledged angry AI to label behaviors as angry or not angry. So you derive the predictions, or look up the predictions made by ancient theologians without advance knowledge of our experimental results. If experiment conflicts with those predictions, then it is fair to speak of the religious claim having been scientifically refuted. It was given its just chance at confirmation; it is being excluded a posteriori, not a priori.

Ultimately, reductionism is just disbelief in fundamentally complicated things. If "fundamentally complicated" sounds like an oxymoron... well, that's why I think that the doctrine of non-reductionism is a confusion, rather than a way that things could be, but aren't. You would be wise to be wary, if you find yourself supposing such things.

But the ultimate rule of science is to look and see. If ever a God appeared to thunder upon the mountains, it would be something that people looked at and saw.

Corollary: Any supposed designer of Artificial General Intelligence who talks about religious beliefs in respectful tones, is clearly not an expert on reducing mental things to nonmental things; and indeed knows so very little of the uttermost basics, as for it to be scarcely plausible that they could be expert at the art; unless their idiot savancy is complete. Or, of course, if they're outright lying. We're not talking about a subtle mistake.

Sequence: Reductionism

Reductive Reference

Psychic Powers

Referenced by: Psychic Powers • Competent Elites • Above-Average AI Scientists • Beyond the Reach of God • Mundane Magic • ...Recursion, Magic • High Challenge • The Uses of Fun (Theory) • Justified Expectation of Pleasant Surprises • Sympathetic Minds • Striving to Accept • Raising the Sanity Waterline • Atheism = Untheism + Antitheism • I'm Not Saying People Are Stupid

Original with comments: Excluding the Supernatural

Psychic Powers

Eliezer Yudkowsky, 12 September 2008 07:28PM

Followup to: Excluding the Supernatural

Yesterday, I wrote:

If the "boring view" of reality is correct, then you can never predict anything irreducible because you are reducible. You can never get Bayesian confirmation for a hypothesis of irreducibility, because any prediction you can make is, therefore, something that could also be predicted by a reducible thing, namely your brain.

Benja Fallenstein commented:

I think that while you can in this case never devise an empirical test whose outcome could logically prove irreducibility, there is no clear reason to believe that you cannot devise a test whose counterfactual outcome in an irreducible world would make irreducibility subjectively much more probable (given an Occamian prior).

Without getting into reducibility/irreducibility, consider the scenario that the physical universe makes it possible to build a hypercomputer -- that performs operations on arbitrary real numbers, for example -- but that our brains do not actually make use of this: they can be simulated perfectly well by an ordinary Turing machine, thank you very much...

Well, that's a very intelligent argument, Benja Fallenstein. But I have a crushing reply to your argument, such that, once I deliver it, you will at once give up further debate with me on this particular point:

You're right.

Alas, I don't get modesty credit on this one, because after publishing yesterday's post I realized a similar flaw on my own - this one concerning Occam's Razor and psychic powers:

If beliefs and desires are irreducible and ontologically basic entities, or have an ontologically basic component not covered by existing science, that would make it far more likely that there was an ontological rule governing the interaction of different minds - an interaction which bypassed ordinary "material" means of communication like sound waves, known to existing science.

If naturalism is correct, then there exists a conjugate reductionist model that makes the same predictions as any concrete prediction that any parapsychologist can make about telepathy.

Indeed, if naturalism is correct, the only reason we can conceive of beliefs as "fundamental" is due to lack of self-knowledge of our own neurons - that the peculiar reflective architecture of our own minds exposes the "belief" class but hides the machinery behind it.

Nonetheless, the discovery of information transfer between brains, in the absence of any known material connection between them, is probabilistically a privileged prediction of supernatural models (those that contain ontologically basic mental entities). Just because it is so much simpler in that case to have a new law relating beliefs between different minds, compared to the "boring" model where beliefs are complex constructs of neurons.

The hope of psychic powers arises from treating beliefs and desires as sufficiently fundamental objects that they can have unmediated connections to reality. If beliefs are patterns of neurons made of known material, with inputs given by organs like eyes constructed of known material, and with outputs through muscles constructed of known material, and this seems sufficient to account for all known mental powers of humans, then there's no reason to expect anything more - no reason to postulate additional connections. This is why reductionists don't expect psychic powers. Thus, observing psychic powers would be strong evidence for the supernatural in Richard Carrier's sense.

We have an Occam rule that counts the number of ontologically basic classes and ontologically basic laws in the model, and penalizes the count of entities. If naturalism is correct, then the attempt to count "belief" or the "relation between belief and reality" as a single basic entity, is simply misguided anthropomorphism; we are only tempted to it by a quirk of our brain's internal architecture. But if you just go with that misguided view, then it assigns a much higher probability to psychic powers than does naturalism, because you can implement psychic powers using apparently simpler laws.

Hence the actual discovery of psychic powers would imply that the human-naive Occam rule was in fact better-calibrated than the sophisticated naturalistic Occam rule. It would argue that reductionists had been wrong all along in trying to take apart the brain; that what our minds exposed as a seemingly simple lever, was in fact a simple lever. The naive dualists would have been right from the beginning, which is why their ancient wish would have been enabled to come true.

So telepathy, and the ability to influence events just by wishing at them, and precognition, would all, if discovered, be strong Bayesian evidence in favor of the hypothesis that beliefs are ontologically fundamental. Not logical proof, but strong Bayesian evidence.

If reductionism is correct, then any science-fiction story containing psychic powers, can be output by a system of simple elements (i.e., the story's author's brain); but if we in fact discover psychic powers, that would make it much more probable that events were occurring which could not in fact be described by reductionist models.

Which just goes to say: The existence of psychic powers is a privileged probabilistic assertion of non-reductionist worldviews - they own that advance prediction; they devised it and put it forth, in defiance of reductionist expectations. So by the laws of science, if psychic powers are discovered, non-reductionism wins.

I am therefore confident in dismissing psychic powers as a priori implausible, despite all the claimed experimental evidence in favor of them.

Sequence: Reductionism

Excluding the Supernatural

Referenced by: Above-Average AI Scientists

Original with comments: Psychic Powers

Optimization

Eliezer Yudkowsky, 13 September 2008 04:00PM

"However many ways there may be of being alive, it is certain that there are vastly more ways of being dead."
-- Richard Dawkins

In the coming days, I expect to be asked: "Ah, but what do you mean by 'intelligence'?" By way of untangling some of my dependency network for future posts, I here summarize some of my notions of "optimization".

Consider a car; say, a Toyota Corolla. The Corolla is made up of some number of atoms; say, on the rough order of 10^29. If you consider all possible ways to arrange 10^29 atoms, only an infinitesimally tiny fraction of possible configurations would qualify as a car; if you picked one random configuration per Planck interval, many ages of the universe would pass before you hit on a wheeled wagon, let alone an internal combustion engine.

Even restricting our attention to running vehicles, there is an astronomically huge design space of possible vehicles that could be composed of the same atoms as the Corolla, and most of them, from the perspective of a human user, won't work quite as well. We could take the parts in the Corolla's air conditioner, and mix them up in thousands of possible configurations; nearly all these configurations would result in a vehicle lower in our preference ordering, still recognizable as a car but lacking a working air conditioner.

So there are many more configurations corresponding to nonvehicles, or vehicles lower in our preference ranking, than vehicles ranked greater than or equal to the Corolla.

Similarly with the problem of planning, which also involves hitting tiny targets in a huge search space. Consider the number of possible legal chess moves versus the number of winning moves.

Which suggests one theoretical way to measure optimization - to quantify the power of a mind or mindlike process:

Put a measure on the state space - if it's discrete, you can just count. Then collect all the states which are equal to or greater than the observed outcome, in that optimization process's implicit or explicit preference ordering. Sum or integrate over the total size of all such states. Divide by the total volume of the state space. This gives you the power of the optimization process measured in terms of the improbabilities that it can produce - that is, improbability of a random selection producing an equally good result, relative to a measure and a preference ordering.

If you prefer, you can take the reciprocal of this improbability (1/1000 becomes 1000) and then take the logarithm base 2. This gives you the power of the optimization process in bits. An optimizer that exerts 20 bits of power can hit a target that's one in a million.

When I think you're a powerful intelligence, and I think I know something about your preferences, then I'll predict that you'll steer reality into regions that are higher in your preference ordering. The more intelligent I believe you are, the more probability I'll concentrate into outcomes that I believe are higher in your preference ordering.

There's a number of subtleties here, some less obvious than others. I'll return to this whole topic in a later sequence. Meanwhile:

* A tiny fraction of the design space does describe vehicles that we would recognize as faster, more fuel-efficient, safer than the Corolla, so the Corolla is not optimal. The Corolla is, however, optimized, because the human designer had to hit an infinitesimal target in design space just to create a working car, let alone a car of Corolla-equivalent quality. This is not to be taken as praise of the Corolla, as such; you could say the same of the Hillman Minx. You can't build so much as a wooden wagon by sawing boards into random shapes and nailing them together according to coinflips.

* When I talk to a popular audience on this topic, someone usually says: "But isn't this what the creationists argue? That if you took a bunch of atoms and put them in a box and shook them up, it would be astonishingly improbable for a fully functioning rabbit to fall out?" But the logical flaw in the creationists' argument is not that randomly reconfiguring molecules would by pure chance assemble a rabbit. The logical flaw is that there is a process, natural selection, which, through the non-chance retention of chance mutations, selectively accumulates complexity, until a few billion years later it produces a rabbit.

* I once heard a senior mainstream AI type suggest that we might try to quantify the intelligence of an AI system in terms of its RAM, processing power, and sensory input bandwidth. This at once reminded me of a quote from Dijkstra: "If we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger." If you want to measure the intelligence of a system, I would suggest measuring its optimization power as before, but then dividing by the resources used. Or you might measure the degree of prior cognitive optimization required to achieve the same result using equal or fewer resources. Intelligence, in other words, is efficient optimization. This is why I say that evolution is stupid by human standards, even though we can't yet build a butterfly: Human engineers use vastly less time/material resources than a global ecosystem of millions of species proceeding through biological evolution, and so we're catching up fast.

* The notion of a "powerful optimization process" is necessary and sufficient to a discussion about an Artificial Intelligence that could harm or benefit humanity on a global scale. If you say that an AI is mechanical and therefore "not really intelligent", and it outputs an action sequence that hacks into the Internet, constructs molecular nanotechnology and wipes the solar system clean of human(e) intelligence, you are still dead. Conversely, an AI that only has a very weak ability steer the future into regions high in its preference ordering, will not be able to much benefit or much harm humanity.

* How do you know a mind's preference ordering? If this can't be taken for granted, then you use some of your evidence to infer the mind's preference ordering, and then use the inferred preferences to infer the mind's power, then use those two beliefs to testably predict future outcomes. Or you can use the Minimum Message Length formulation of Occam's Razor: if you send me a message telling me what a mind wants and how powerful it is, then this should enable you to compress your description of future events and observations, so that the total message is shorter. Otherwise there is no predictive benefit to viewing a system as an optimization process.

* In general, it is useful to think of a process as "optimizing" when it is easier to predict by thinking about its goals, than by trying to predict its exact internal state and exact actions. If you're playing chess against Deep Blue, you will find it much easier to predict that Deep Blue will win (that is, the final board position will occupy the class of states previously labeled "wins for Deep Blue") than to predict the exact final board position or Deep Blue's exact sequence of moves. Normally, it is not possible to predict, say, the final state of a billiards table after a shot, without extrapolating all the events along the way.

* Although the human cognitive architecture uses the same label "good" to reflect judgments about terminal values and instrumental values, this doesn't mean that all sufficiently powerful optimization processes share the same preference ordering. Some possible minds will be steering the future into regions that are not good.

* If you came across alien machinery in space, then you might be able to infer the presence of optimization (and hence presumably powerful optimization processes standing behind it as a cause) without inferring the aliens' final goals, by way of noticing the fulfillment of convergent instrumental values. You can look at cables through which large electrical currents are running, and be astonished to realize that the cables are flexible high-temperature high-amperage superconductors; an amazingly good solution to the subproblem of transporting electricity that is generated in a central location and used distantly. You can assess this, even if you have no idea what the electricity is being used for.

* If you want to take probabilistic outcomes into account in judging a mind's wisdom, then you have to know or infer a utility function for the mind, not just a preference ranking for the optimization process. Then you can ask how many possible plans would have equal or greater expected utility. This assumes that you have some probability distribution, which you believe to be true; but if the other mind is smarter than you, it may have a better probability distribution, in which case you will underestimate its optimization power. The chief sign of this would be if the mind consistently achieves higher average utility than the average expected utility you assign to its plans.

* When an optimization process seems to have an inconsistent preference ranking - for example, it's quite possible in evolutionary biology for allele A to beat out allele B, which beats allele C, which beats allele A - then you can't interpret the system as performing optimization as it churns through its cycles. Intelligence is efficient optimization; churning through preference cycles is stupid, unless the interim states of churning have high terminal utility.

* For domains outside the small and formal, it is not possible to exactly measure optimization, just as it is not possible to do exact Bayesian updates or to perfectly maximize expected utility. Nonetheless, optimization can be a useful concept, just like the concept of Bayesian probability or expected utility - it describes the ideal you're trying to approximate with other measures.

Referenced by: My Naturalistic Awakening

Original with comments: Optimization

My Childhood Death Spiral

Eliezer Yudkowsky, 15 September 2008 03:42AM

Followup to: Affective Death Spirals, My Wild and Reckless Youth

My parents always used to downplay the value of intelligence. And play up the value of - effort, as recommended by the latest research? No, not effort. Experience. A nicely unattainable hammer with which to smack down a bright young child, to be sure. That was what my parents told me when I questioned the Jewish religion, for example. I tried laying out an argument, and I was told something along the lines of: "Logic has limits, you'll understand when you're older that experience is the important thing, and then you'll see the truth of Judaism." I didn't try again. I made one attempt to question Judaism in school, got slapped down, didn't try again. I've never been a slow learner.

Whenever my parents were doing something ill-advised, it was always, "We know better because we have more experience. You'll understand when you're older: maturity and wisdom is more important than intelligence."

If this was an attempt to focus the young Eliezer on intelligence uber alles, it was the most wildly successful example of reverse psychology I've ever heard of.

But my parents aren't that cunning, and the results weren't exactly positive.

For a long time, I thought that the moral of this story was that experience was no match for sheer raw native intelligence. It wasn't until a lot later, in my twenties, that I looked back and realized that I couldn't possibly have been more intelligent than my parents before puberty, with my brain not even fully developed. At age eleven, when I was already nearly a full-blown atheist, I could not have defeated my parents in any fair contest of mind. My SAT scores were high for an 11-year-old, but they wouldn't have beaten my parents' SAT scores in full adulthood. In a fair fight, my parents' intelligence and experience could have stomped any prepubescent child flat. It was dysrationalia that did them in; they used their intelligence only to defeat itself.

But that understanding came much later, when my intelligence had processed and distilled many more years of experience.

The moral I derived when I was young, was that anyone who downplayed the value of intelligence didn't understand intelligence at all. My own intelligence had affected every aspect of my life and mind and personality; that was massively obvious, seen at a backward glance. "Intelligence has nothing to do with wisdom or being a good person" - oh, and does self-awareness have nothing to do with wisdom, or being a good person? Modeling yourself takes intelligence. For one thing, it takes enough intelligence to learn evolutionary psychology.

We are the cards we are dealt, and intelligence is the unfairest of all those cards. More unfair than wealth or health or home country, unfairer than your happiness set-point. People have difficulty accepting that life can be that unfair, it's not a happy thought. "Intelligence isn't as important as X" is one way of turning away from the unfairness, refusing to deal with it, thinking a happier thought instead. It's a temptation, both to those dealt poor cards, and to those dealt good ones. Just as downplaying the importance of money is a temptation both to the poor and to the rich.

But the young Eliezer was a transhumanist. Giving away IQ points was going to take more work than if I'd just been born with extra money. But it was a fixable problem, to be faced up to squarely, and fixed. Even if it took my whole life. "The strong exist to serve the weak," wrote the young Eliezer, "and can only discharge that duty by making others equally strong." I was annoyed with the Randian and Nietszchean trends in SF, and as you may have grasped, the young Eliezer had a tendency to take things too far in the other direction. No one exists only to serve. But I tried, and I don't regret that. If you call that teenage folly, it's rare to see adult wisdom doing better.

Everyone needed more intelligence. Including me, I was careful to pronounce. Be it far from me to declare a new world order with myself on top - that was what a stereotyped science fiction villain would do, or worse, a typical teenager, and I would never have allowed myself to be so cliched. No, everyone needed to be smarter. We were all in the same boat: A fine, uplifting thought.

Eliezer_1995 had read his science fiction. He had morals, and ethics, and could see the more obvious traps. No screeds on Homo novis for him. No line drawn between himself and others. No elaborate philosophy to put himself at the top of the heap. It was too obvious a failure mode. Yes, he was very careful to call himself stupid too, and never claim moral superiority. Well, and I don't see it so differently now, though I no longer make such a dramatic production out of my ethics. (Or maybe it would be more accurate to say that I'm tougher about when I allow myself a moment of self-congratulation.)

I say all this to emphasize that Eliezer_1995 wasn't so undignified as to fail in any obvious way.

And then Eliezer_1996 encountered the concept of the Singularity. Was it a thunderbolt of revelation? Did I jump out of my chair and shout "Eurisko!"? Nah. I wasn't that much of a drama queen. It was just massively obvious in retrospect that smarter-than-human intelligence was going to change the future more fundamentally than any mere material science. And I knew at once that this was what I would be doing with the rest of my life, creating the Singularity. Not nanotechnology like I'd thought when I was eleven years old; nanotech would only be a tool brought forth of intelligence. Why, intelligence was even more powerful, an even greater blessing, than I'd realized before.

Was this a happy death spiral? As it turned out later, yes: that is, it led to the adoption even of false happy beliefs about intelligence. Perhaps you could draw the line at the point where I started believing that surely the lightspeed limit would be no barrier to superintelligence. (It's not unthinkable, but I wouldn't bet on it.)

But the real wrong turn came later, at the point where someone said, "Hey, how do you know that superintelligence will be moral? Intelligence has nothing to do with being a good person, you know - that's what we call wisdom, young prodigy."

And lo, it seemed obvious to the young Eliezer, that this was mere denial. Certainly, his own painstakingly constructed code of ethics had been put together using his intelligence and resting on his intelligence as a base. Any fool could see that intelligence had a great deal to do with ethics, morality, and wisdom; just try explaining the Prisoner's Dilemma to a chimpanzee, right?

Surely, then, superintelligence would necessarily imply supermorality.

Thus is it said: "Parents do all the things they tell their children not to do, which is how they know not to do them." To be continued, hopefully tomorrow.

Post Scriptum: How my views on intelligence have changed since then... let's see: When I think of poor hands dealt to humans, these days, I think first of death and old age. Everyone's got to have some intelligence level or other, and the important thing from a fun-theoretical perspective is that it should ought to increase over time, not decrease like now. Isn't that a clever way of feeling better? But I don't work so hard now at downplaying my own intelligence, because that's just another way of calling attention to it. I'm smart for a human, if the topic should arise, and how I feel about that is my own business. The part about intelligence being the lever that lifts worlds is the same. Except that intelligence has become less mysterious unto me, so that I now more clearly see intelligence as something embedded within physics. Superintelligences may go FTL if it happens to be permitted by the true physical laws, and if not, then not.

Sequence: Yudkowsky's Coming of Age

My Best and Worst Mistake

Referenced by: My Best and Worst Mistake • A Prodigy of Refutation • The Sheer Folly of Callow Youth • Fighting a Rearguard Action Against the Truth • Competent Elites • On Doing the Impossible • Not Taking Over the World • Prolegomena to a Theory of Fun • Serious Stories • The Thing That I Protect

Original with comments: My Childhood Death Spiral

My Best and Worst Mistake

Eliezer Yudkowsky, 16 September 2008 12:43AM

Followup to: My Childhood Death Spiral

Yesterday I covered the young Eliezer's affective death spiral around something that he called "intelligence". Eliezer_1996, or even Eliezer_1999 for that matter, would have refused to try and put a mathematical definition - consciously, deliberately refused. Indeed, he would have been loath to put any definition on "intelligence" at all.

Why? Because there's a standard bait-and-switch problem in AI, wherein you define "intelligence" to mean something like "logical reasoning" or "the ability to withdraw conclusions when they are no longer appropriate", and then you build a cheap theorem-prover or an ad-hoc nonmonotonic reasoner, and then say, "Lo, I have implemented intelligence!" People came up with poor definitions of intelligence - focusing on correlates rather than cores - and then they chased the surface definition they had written down, forgetting about, you know, actual intelligence. It's not like Eliezer_1996 was out to build a career in Artificial Intelligence. He just wanted a mind that would actually be able to build nanotechnology. So he wasn't tempted to redefine intelligence for the sake of puffing up a paper.

Looking back, it seems to me that quite a lot of my mistakes can be defined in terms of being pushed too far in the other direction by seeing someone else stupidity: Having seen attempts to define "intelligence" abused so often, I refused to define it at all. What if I said that intelligence was X, and it wasn't really X? I knew in an intuitive sense what I was looking for - something powerful enough to take stars apart for raw material - and I didn't want to fall into the trap of being distracted from that by definitions.

Similarly, having seen so many AI projects brought down by physics envy - trying to stick with simple and elegant math, and being constrained to toy systems as a result - I generalized that any math simple enough to be formalized in a neat equation was probably not going to work for, you know, real intelligence. "Except for Bayes's Theorem," Eliezer_2000 added; which, depending on your viewpoint, either mitigates the totality of his offense, or shows that he should have suspected the entire generalization instead of trying to add a single exception.

If you're wondering why Eliezer_2000 thought such a thing - disbelieved in a math of intelligence - well, it's hard for me to remember this far back. It certainly wasn't that I ever disliked math. If I had to point out a root cause, it would be reading too few, too popular, and the wrong Artificial Intelligence books.

But then I didn't think the answers were going to come from Artificial Intelligence; I had mostly written it off as a sick, dead field. So it's no wonder that I spent too little time investigating it. I believed in the cliche about Artificial Intelligence overpromising. You can fit that into the pattern of "too far in the opposite direction" - the field hadn't delivered on its promises, so I was ready to write it off. As a result, I didn't investigate hard enough to find the math that wasn't fake.

My youthful disbelief in a mathematics of general intelligence was simultaneously one of my all-time worst mistakes, and one of my all-time best mistakes.

Because I disbelieved that there could be any simple answers to intelligence, I went and I read up on cognitive psychology, functional neuroanatomy, computational neuroanatomy, evolutionary psychology, evolutionary biology, and more than one branch of Artificial Intelligence. When I had what seemed like simple bright ideas, I didn't stop there, or rush off to try and implement them, because I knew that even if they were true, even if they were necessary, they wouldn't be sufficient: intelligence wasn't supposed to be simple, it wasn't supposed to have an answer that fit on a T-Shirt. It was supposed to be a big puzzle with lots of pieces; and when you found one piece, you didn't run off holding it high in triumph, you kept on looking. Try to build a mind with a single missing piece, and it might be that nothing interesting would happen.

I was wrong in thinking that Artificial Intelligence the academic field, was a desolate wasteland; and even wronger in thinking that there couldn't be math of intelligence. But I don't regret studying e.g. functional neuroanatomy, even though I now think that an Artificial Intelligence should look nothing like a human brain. Studying neuroanatomy meant that I went in with the idea that if you broke up a mind into pieces, the pieces were things like "visual cortex" and "cerebellum" - rather than "stock-market trading module" or "commonsense reasoning module", which is a standard wrong road in AI.

Studying fields like functional neuroanatomy and cognitive psychology gave me a very different idea of what minds had to look like, than you would get from just reading AI books - even good AI books.

When you blank out all the wrong conclusions and wrong justifications, and just ask what that belief led the young Eliezer to actually do...

Then the belief that Artificial Intelligence was sick and that the real answer would have to come from healthier fields outside, led him to study lots of cognitive sciences;

The belief that AI couldn't have simple answers, led him to not stop prematurely on one brilliant idea, and to accumulate lots of information;

The belief that you didn't want to define intelligence, led to a situation in which he studied the problem for a long time before, years later, he started to propose systematizations.

This is what I refer to when I say that this is one of my all-time best mistakes.

Looking back, years afterward, I drew a very strong moral, to this effect:

What you actually end up doing, screens off the clever reason why you're doing it.

Contrast amazing clever reasoning that leads you to study many sciences, to amazing clever reasoning that says you don't need to read all those books. Afterward, when your amazing clever reasoning turns out to have been stupid, you'll have ended up in a much better position, if your amazing clever reasoning was of the first type.

When I look back upon my past, I am struck by the number of semi-accidental successes, the number of times I did something right for the wrong reason. From your perspective, you should chalk this up to the anthropic principle: if I'd fallen into a true dead end, you probably wouldn't be hearing from me on this blog. From my perspective it remains something of an embarrassment. My Traditional Rationalist upbringing provided a lot of directional bias to those "accidental successes" - biased me toward rationalizing reasons to study rather than not study, prevented me from getting completely lost, helped me recover from mistakes. Still, none of that was the right action for the right reason, and that's a scary thing to look back on your youthful history and see. One of my primary purposes in writing on Overcoming Bias is to leave a trail to where I ended up by accident - to obviate the role that luck played in my own forging as a rationalist.

So what makes this one of my all-time worst mistakes? Because sometimes "informal" is another way of saying "held to low standards". I had amazing clever reasons why it was okay for me not to precisely define "intelligence", and certain of my other terms as well: namely,other people had gone astray by trying to define it. This was a gate through which sloppy reasoning could enter.

So should I have jumped ahead and tried to forge an exact definition right away? No, all the reasons why I knew this was the wrong thing to do, were correct; you can't conjure the right definition out of thin air if your knowledge is not adequate.

You can't get to the definition of fire if you don't know about atoms and molecules; you're better off saying "that orangey-bright thing". And you do have to be able to talk about that orangey-bright stuff, even if you can't say exactly what it is, to investigate fire. But these days I would say that all reasoning on that level is something that can't be trusted - rather it's something you do on the way to knowing better, but you don't trust it, you don't put your weight down on it, you don't draw firm conclusions from it, no matter how inescapable the informal reasoning seems.

The young Eliezer put his weight down on the wrong floor tile - stepped onto a loaded trap. To be continued.

Sequence: Yudkowsky's Coming of Age

My Childhood Death Spiral

Raised in Technophilia

Referenced by: Raised in Technophilia • The Sheer Folly of Callow Youth • My Bayesian Enlightenment

Original with comments: My Best and Worst Mistake

Raised in Technophilia

Eliezer Yudkowsky, 17 September 2008 02:06AM

Followup to: My Best and Worst Mistake

My father used to say that if the present system had been in place a hundred years ago, automobiles would have been outlawed to protect the saddle industry.

One of my major childhood influences was reading Jerry Pournelle's A Step Farther Out, at the age of nine. It was Pournelle's reply to Paul Ehrlich and the Club of Rome, who were saying, in the 1960s and 1970s, that the Earth was running out of resources and massive famines were only years away. It was a reply to Jeremy Rifkin's so-called fourth law of thermodynamics; it was a reply to all the people scared of nuclear power and trying to regulate it into oblivion.

I grew up in a world where the lines of demarcation between the Good Guys and the Bad Guys were pretty clear; not an apocalyptic final battle, but a battle that had to be fought over and over again, a battle where you could see the historical echoes going back to the Industrial Revolution, and where you could assemble the historical evidence about the actual outcomes.

On one side were the scientists and engineers who'd driven all the standard-of-living increases since the Dark Ages, whose work supported luxuries like democracy, an educated populace, a middle class, the outlawing of slavery.

On the other side, those who had once opposed smallpox vaccinations, anesthetics during childbirth, steam engines, and heliocentrism: The theologians calling for a return to a perfect age that never existed, the elderly white male politicians set in their ways, the special interest groups who stood to lose, and the many to whom science was a closed book, fearing what they couldn't understand.

And trying to play the middle, the pretenders to Deep Wisdom, uttering cached thoughts about how technology benefits humanity but only when it was properly regulated - claiming in defiance of brute historical fact that science of itself was neither good nor evil - setting up solemn-looking bureaucratic committees to make an ostentatious display of their caution - and waiting for their applause. As if the truth were always a compromise. And as if anyone could really see that far ahead. Would humanity have done better if there'd been a sincere, concerned, public debate on the adoption of fire, and commitees set up to oversee its use?

When I entered into the problem, I started out allergized against anything that pattern-matched "Ah, but technology has risks as well as benefits, litte one." The presumption-of-guilt was that you were either trying to collect some cheap applause, or covertly trying to regulate the technology into oblivion. And either way, ignoring the historical record immensely in favor of technologies that people had once worried about.

Today, Robin Hanson raised the topic of slow FDA approval of drugs approved in other countries. Someone in the comments pointed out that Thalidomide was sold in 50 countries under 40 names, but that only a small amount was given away in the US, so that there were 10,000 malformed children born globally, but only 17 children in the US.

But how many people have died because of the slow approval in the US, of drugs more quickly approved in other countries - all the drugs that didn't go wrong? And I ask that question because it's what you can try to collect statistics about - this says nothing about all the drugs that were never developed because the approval process is too long and costly. According to this source, the FDA's longer approval process prevents 5,000 casualties per year by screening off medications found to be harmful, and causes at least 20,000-120,000 casualties per year just by delaying approval of those beneficial medications that are still developed and eventually approved.

So there really is a reason to be allergic to people who go around saying, "Ah, but technology has risks as well as benefits". There's a historical record showing over-conservativeness, the many silent deaths of regulation being outweighed by a few visible deaths of nonregulation. If you're really playing the middle, why not say, "Ah, but technology has benefits as well as risks"?

Well, and this isn't such a bad description of the Bad Guys. (Except that it ought to be emphasized a bit harder that these aren't evil mutants but standard human beings acting under a different worldview-gestalt that puts them in the right; some of them will inevitably be more competent than others, and competence counts for a lot.) Even looking back, I don't think my childhood technophilia was too wrong about what constituted a Bad Guy and what was the key mistake. But it's always a lot easier to say what not to do, than to get it right. And one of my fundamental flaws, back then, was thinking that, if you tried as hard as you could to avoid everything the Bad Guys were doing, that made you a Good Guy.

Particularly damaging, I think, was the bad example set by the pretenders to Deep Wisdom trying to stake out a middle way; smiling condescendingly at technophiles and technophobes alike, and calling them both immature. Truly this is a wrong way; and in fact, the notion of trying to stake out a middle way generally, is usually wrong; the Right Way is not a compromise with anything, it is the clean manifestation of its own criteria.

But that made it more difficult for the young Eliezer to depart from the charge-straight-ahead verdict, because any departure felt like joining the pretenders to Deep Wisdom.

The first crack in my childhood technophilia appeared in, I think, 1997 or 1998, at the point where I noticed my fellow technophiles saying foolish things about how molecular nanotechnology would be an easy problem to manage. (As you may be noticing yet again, the young Eliezer was driven to a tremendous extent by his ability to find flaws - I even had a personal philosophy of why that sort of thing was a good idea.)

The nanotech stuff would be a separate post, and maybe one that should go on a different blog. But there was a debate going on about molecular nanotechnology, and whether offense would be asymmetrically easier than defense. And there were people arguing that defense would be easy. In the domain of nanotech, for Ghu's sake, programmable matter, when we can't even seem to get the security problem solved for computer networks where we can observe and control every one and zero. People were talking about unassailable diamondoid walls. I observed that diamond doesn't stand off a nuclear weapon, that offense has had defense beat since 1945 and nanotech didn't look likely to change that.

And by the time that debate was over, it seems that the young Eliezer - caught up in the heat of argument - had managed to notice, for the first time, that the survival of Earth-originating intelligent life stood at risk.

It seems so strange, looking back, to think that there was a time when I thought that only individual lives were at stake in the future. What a profoundly friendlier world that was to live in... though it's not as if I were thinking that at the time. I didn't reject the possibility so much as manage to never see it in the first place. Once the topic actually came up, I saw it. I don't really remember how that trick worked. There's a reason why I refer to my past self in the third person.

It may sound like Eliezer_1998 was a complete idiot, but that would be a comfortable out, in a way; the truth is scarier. Eliezer_1998 was a sharp Traditional Rationalist, as such things went. I knew hypotheses had to be testable, I knew that rationalization was not a permitted mental operation, I knew how to play Rationalist's Taboo, I was obsessed with self-awareness... I didn't quite understand the concept of "mysterious answers"... and no Bayes or Kahneman at all. But a sharp Traditional Rationalist, far above average... So what? Nature isn't grading us on a curve. One step of departure from the Way, one shove of undue influence on your thought processes, can repeal all other protections.

One of the chief lessons I derive from looking back at my personal history is that it's no wonder that, out there in the real world, a lot of people think that "intelligence isn't everything", or that rationalists don't do better in real life. A little rationality, or even a lot of rationality, doesn't pass the astronomically high barrier required for things to actually start working.

Let not my misinterpretation of the Right Way be blamed on Jerry Pournelle, my father, or science fiction generally. I think the young Eliezer's personality imposed quite a bit of selectivity on which parts of their teachings made it through. It's not as if Pournelle didn't say: The rules change once you leave Earth, the cradle; if you're careless sealing your pressure suit just once, you die. He said it quite a bit. But the words didn't really seem important, because that was something that happened to third-party characters in the novels - the main character didn't usually die halfway through, for some reason.

What was the lens through which I filtered these teachings? Hope. Optimism. Looking forward to a brighter future. That was the fundamental meaning of A Step Farther Out unto me, the lesson I took in contrast to the Sierra Club's doom-and-gloom. On one side was rationality and hope, the other, ignorance and despair.

Some teenagers think they're immortal and ride motorcycles. I was under no such illusion and quite reluctant to learn to drive, considering how unsafe those hurtling hunks of metal looked. But there was something more important to me than my own life: The Future. And I acted as if that was immortal. Lives could be lost, but not the Future.

And when I noticed that nanotechnology really was going to be a potentially extinction-level challenge?

The young Eliezer thought, explicitly, "Good heavens, how did I fail to notice this thing that should have been obvious? I must have been too emotionally attached to the benefits I expected from the technology; I must have flinched away from the thought of human extinction."

And then...

I didn't declare a Halt, Melt, and Catch Fire. I didn't rethink all the conclusions that I'd developed with my prior attitude. I just managed to integrate it into my worldview, somehow, with a minimum of propagated changes. Old ideas and plans were challenged, but my mind found reasons to keep them. There was no systemic breakdown, unfortunately.

Most notably, I decided that we had to run full steam ahead on AI, so as to develop it before nanotechnology. Just like I'd been originally planning to do, but now, with a different reason.

I guess that's what most human beings are like, isn't it? Traditional Rationality wasn't enough to change that.

But there did come a time when I fully realized my mistake. It just took a stronger boot to the head. To be continued.

Sequence: Yudkowsky's Coming of Age

My Best and Worst Mistake

A Prodigy of Refutation

Referenced by: A Prodigy of Refutation • The Sheer Folly of Callow Youth • Fighting a Rearguard Action Against the Truth • The Magnitude of His Own Folly • Beyond the Reach of God • On Doing the Impossible • Ethical Injunctions

Original with comments: Raised in Technophilia

A Prodigy of Refutation

Eliezer Yudkowsky, 18 September 2008 01:57AM

Followup to: My Childhood Death Spiral, Raised in Technophilia

My Childhood Death Spiral described the core momentum carrying me into my mistake, an affective death spiral around something that Eliezer_1996 called "intelligence". I was also a technophile, pre-allergized against fearing the future. And I'd read a lot of science fiction built around personhood ethics - in which fear of the Alien puts humanity-at-large in the position of the bad guys, mistreating aliens or sentient AIs because they "aren't human".

That's part of the ethos you acquire from science fiction - to define your in-group, your tribe, appropriately broadly. Hence my email address, sentience@pobox.com.

So Eliezer_1996 is out to build superintelligence, for the good of humanity and all sentient life.

At first, I think, the question of whether a superintelligence will/could be good/evil didn't really occur to me as a separate topic of discussion. Just the standard intuition of, "Surely no supermind would be stupid enough to turn the galaxy into paperclips; surely, being so intelligent, it will also know what's right far better than a human being could."

Until I introduced myself and my quest to a transhumanist mailing list, and got back responses along the general lines of (from memory):

Morality is arbitrary - if you say that something is good or bad, you can't be right or wrong about that. A superintelligence would form its own morality.

Everyone ultimately looks after their own self-interest. A superintelligence would be no different; it would just seize all the resources.

Personally, I'm a human, so I'm in favor of humans, not Artificial Intelligences. I don't think we should develop this technology. Instead we should develop the technology to upload humans first.

No one should develop an AI without a control system that watches it and makes sure it can't do anything bad.

Well, that's all obviously wrong, thinks Eliezer_1996, and he proceeded to kick his opponents' arguments to pieces. (I've mostly done this in other blog posts, and anything remaining is left as an exercise to the reader.)

It's not that Eliezer_1996 explicitly reasoned, "The world's stupidest man says the sun is shining, therefore it is dark out." But Eliezer_1996 was a Traditional Rationalist; he had been inculcated with the metaphor of science as a fair fight between sides who take on different positions, stripped of mere violence and other such exercises of political muscle, so that, ideally, the side with the best arguments can win.

It's easier to say where someone else's argument is wrong, then to get the fact of the matter right; and Eliezer_1996 was very skilled at finding flaws. (So am I. It's not as if you can solve the danger of that power by refusing to care about flaws.) From Eliezer_1996's perspective, it seemed to him that his chosen side was winning the fight - that he was formulating better arguments than his opponents - so why would he switch sides?

Therefore is it written: "Because this world contains many whose grasp of rationality is abysmal, beginning students of rationality win arguments and acquire an exaggerated view of their own abilities. But it is useless to be superior: Life is not graded on a curve. The best physicist in ancient Greece could not calculate the path of a falling apple. There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse."

You cannot rely on anyone else to argue you out of your mistakes; you cannot rely on anyone else to save you; you and only you are obligated to find the flaws in your positions; if you put that burden down, don't expect anyone else to pick it up. And I wonder if that advice will turn out not to help most people, until they've personally blown off their own foot, saying to themselves all the while, correctly, "Clearly I'm winning this argument."

Today I try not to take any human being as my opponent. That just leads to overconfidence. It is Nature that I am facing off against, who does not match Her problems to your skill, who is not obliged to offer you a fair chance to win in return for a diligent effort, who does not care if you are the best who ever lived, if you are not good enough.

But return to 1996. Eliezer_1996 is going with the basic intuition of "Surely a superintelligence will know better than we could what is right," and offhandedly knocking down various arguments brought against his position. He was skillful in that way, you see. He even had a personal philosophy of why it was wise to look for flaws in things, and so on.

I don't mean to say it as an excuse, that no one who argued against Eliezer_1996, actually presented him with the dissolution of the mystery - the full reduction of morality that analyzes all his cognitive processes debating "morality", a step-by-step walkthrough of the algorithms that make morality feel to him like a fact. Consider it rather as an indictment, a measure of Eliezer_1996's level, that he would have needed the full solution given to him, in order to present him with an argument that he could not refute.

The few philosophers present, did not extract him from his difficulties. It's not as if a philosopher will say, "Sorry, morality is understood, it is a settled issue in cognitive science and philosophy, and your viewpoint is simply wrong." The nature of morality is still an open question in philosophy, the debate is still going on. A philosopher will feel obligated to present you with a list of classic arguments on all sides; most of which Eliezer_1996 is quite intelligent enough to knock down, and so he concludes that philosophy is a wasteland.

But wait. It gets worse.

I don't recall exactly when - it might have been 1997 - but the younger me, let's call him Eliezer_1997, set out to argue inescapably that creating superintelligence is the right thing to do. To be continued.

Sequence: Yudkowsky's Coming of Age

Raised in Technophilia

The Sheer Folly of Callow Youth

Referenced by: The Sheer Folly of Callow Youth • Prolegomena to a Theory of Fun

Original with comments: A Prodigy of Refutation

The Sheer Folly of Callow Youth

Eliezer Yudkowsky, 19 September 2008 01:30AM

Followup to: My Childhood Death Spiral, My Best and Worst Mistake, A Prodigy of Refutation

"There speaks the sheer folly of callow youth; the rashness of an ignorance so abysmal as to be possible only to one of your ephemeral race..."
-- Gharlane of Eddore

Once upon a time, years ago, I propounded a mysterious answer to a mysterious question - as I've hinted on several occasions. The mysterious question to which I propounded a mysterious answer was not, however, consciousness - or rather, not only consciousness. No, the more embarrassing error was that I took a mysterious view of morality.

I held off on discussing that until now, after the series on metaethics, because I wanted it to be clear that Eliezer_1997had gotten it wrong.

When we last left off, Eliezer_1997, not satisfied with arguing in an intuitive sense that superintelligence would be moral, was setting out to argue inescapably that creating superintelligence was the right thing to do.

Well (said Eliezer_1997) let's begin by asking the question: Does life have, in fact, any meaning?

"I don't know," replied Eliezer_1997 at once, with a certain note of self-congratulation for admitting his own ignorance on this topic where so many others seemed certain.

"But," he went on -

(Always be wary when an admission of ignorance is followed by "But".)

"But, if we suppose that life has no meaning - that the utility of all outcomes is equal to zero - that possibility cancels out of any expected utility calculation. We can therefore always act as if life is known to be meaningful, even though we don't know what that meaning is. How can we find out that meaning? Considering that humans are still arguing about this, it's probably too difficult a problem for humans to solve. So we need a superintelligence to solve the problem for us. As for the possibility that there is no logical justification for one preference over another, then in this case it is no righter or wronger to build a superintelligence, than to do anything else. This is a real possibility, but it falls out of any attempt to calculate expected utility - we should just ignore it. To the extent someone says that a superintelligence would wipe out humanity, they are either arguing that wiping out humanity is in fact the right thing to do (even though we see no reason why this should be the case) or they are arguing that there is no right thing to do (in which case their argument that we should not build intelligence defeats itself)."

Ergh. That was a really difficult paragraph to write. My past self is always my own most concentrated Kryptonite, because my past self is exactly precisely all those things that the modern me has installed allergies to block. Truly is it said that parents do all the things they tell their children not to do, which is how they know not to do them; it applies between past and future selves as well.

How flawed is Eliezer_1997's argument? I couldn't even count the ways. I know memory is fallible, reconstructed each time we recall, and so I don't trust my assembly of these old pieces using my modern mind. Don't ask me to read my old writings; that's too much pain.

But it seems clear that I was thinking of utility as a sort of stuff, an inherent property. So that "life is meaningless" corresponded to utility=0. But of course the argument works equally well with utility=100, so that if everything is meaningful but it is all equally meaningful, that should fall out too... Certainly I wasn't then thinking of a utility function as an affine structure in preferences. I was thinking of "utility" as an absolute level of inherent value.

I was thinking of should as a kind of purely abstract essence of compellingness, that-which-makes-you-do-something; so that clearly any mind that derived a should, would be bound by it. Hence the assumption, which Eliezer_1997 did not even think to explicitly note, that a logic that compels an arbitrary mind to do something, is exactly the same as that which human beings mean and refer to when they utter the word "right"...

But now I'm trying to count the ways, and if you've been following along, you should be able to handle that yourself.

An important aspect of this whole failure was that, because I'd proved that the case "life is meaningless" wasn't worth considering, I didn't think it was necessary to rigorously define "intelligence" or "meaning". I'd previously come up with a clever reason for not trying to go all formal and rigorous when trying to define "intelligence" (or "morality") - namely all the bait-and-switches that past AIfolk, philosophers, and moralists, had pulled with definitions that missed the point.

I draw the following lesson: No matter how clever the justification for relaxing your standards, or evading some requirement of rigor, it will blow your foot off just the same.

And another lesson: I was skilled in refutation. If I'd applied the same level of rejection-based-on-any-flaw to my own position, as I used to defeat arguments brought against me, then I would have zeroed in on the logical gap and rejected the position - if I'd wanted to. If I'd had the same level of prejudice against it, as I'd had against other positions in the debate.

But this was before I'd heard of Kahneman, before I'd heard the term "motivated skepticism", before I'd integrated the concept of an exactly correct state of uncertainty that summarizes all the evidence, and before I knew the deadliness of asking "Am I allowed to believe?" for liked positions and "Am I forced to believe?" for disliked positions. I was a mere Traditional Rationalist who thought of the scientific process as a referee between people who took up positions and argued them, may the best side win.

My ultimate flaw was not a liking for "intelligence", nor any amount of technophilia and science fiction exalting the siblinghood of sentience. It surely wasn't my ability to spot flaws. None of these things could have led me astray, if I had held myself to a higher standard of rigor throughout, and adopted no position otherwise. Or even if I'd just scrutinized my preferred vague position, with the same demand-of-rigor I applied to counterarguments.

But I wasn't much interested in trying to refute my belief that life had meaning, since my reasoning would always be dominated by cases where life did have meaning.

And with the Singularity at stake, I thought I just had to proceed at all speed using the best concepts I could wield at the time, not pause and shut down everything while I looked for a perfect definition that so many others had screwed up...

No.

No, you don't use the best concepts you can use at the time.

It's Nature that judges you, and Nature does not accept even the most righteous excuses. If you don't meet the standard, you fail. It's that simple. There is no clever argument for why you have to make do with what you have, because Nature won't listen to that argument, won't forgive you because there were so many excellent justifications for speed.

We all know what happened to Donald Rumsfeld, when he went to war with the army he had, instead of the army he needed.

Maybe Eliezer_1997 couldn't have conjured the correct model out of thin air. (Though who knows what would have happened, if he'd really tried...) And it wouldn't have been prudent for him to stop thinking entirely, until rigor suddenly popped out of nowhere.

But neither was it correct for Eliezer_1997 to put his weight down on his "best guess", in the absence of precision. You can use vague concepts in your own interim thought processes, as you search for a better answer, unsatisfied with your current vague hints, and unwilling to put your weight down on them. You don't build a superintelligence based on an interim understanding. No, not even the "best" vague understanding you have. That was my mistake - thinking that saying "best guess" excused anything. There was only the standard I had failed to meet.

Of course Eliezer_1997 didn't want to slow down on the way to the Singularity, with so many lives at stake, and the very survival of Earth-originating intelligent life, if we got to the era of nanoweapons before the era of superintelligence -

Nature doesn't care about such righteous reasons. There's just the astronomically high standard needed for success. Either you match it, or you fail. That's all.

The apocalypse does not need to be fair to you.
The apocalypse does not need to offer you a chance of success
In exchange for what you've already brought to the table.
The apocalypse's difficulty is not matched to your skills.
The apocalypse's price is not matched to your resources.
If the apocalypse asks you for something unreasonable
And you try to bargain it down a little
(Because everyone has to compromise now and then)
The apocalypse will not try to negotiate back up.

And, oh yes, it gets worse.

How did Eliezer_1997 deal with the obvious argument that you couldn't possibly derive an "ought" from pure logic, because "ought" statements could only be derived from other "ought" statements?

Well (observed Eliezer_1997), this problem has the same structure as the argument that a cause only proceeds from another cause, or that a real thing can only come of another real thing, whereby you can prove that nothing exists.

Thus (he said) there are three "hard problems": The hard problem of conscious experience, in which we see that qualia cannot arise from computable processes; the hard problem of existence, in which we ask how any existence enters apparently from nothingness; and the hard problem of morality, which is to get to an "ought".

These problems are probably linked. For example, the qualia of pleasure are one of the best candidates for something intrinsically desirable. We might not be able to understand the hard problem of morality, therefore, without unraveling the hard problem of consciousness. It's evident that these problems are too hard for humans - otherwise someone would have solved them over the last 2500 years since philosophy was invented.

It's not as if they could have complicated solutions - they're too simple for that. The problem must just be outside human concept-space. Since we can see that consciousness can't arise on any computable process, it must involve new physics - physics that our brain uses, but can't understand. That's why we need superintelligence in order to solve this problem. Probably it has to do with quantum mechanics, maybe with a dose of tiny closed timelike curves from out of General Relativity; temporal paradoxes might have some of the same irreducibility properties that consciousness seems to demand...

Et cetera, ad nauseam. You may begin to perceive, in the arc of my Overcoming Bias posts, the letter I wish I could have written to myself.

Of this I learn the lesson: You cannot manipulate confusion. You cannot make clever plans to work around the holes in your understanding. You can't even make "best guesses" about things which fundamentally confuse you, and relate them to other confusing things. Well, you can, but you won't get it right, until your confusion dissolves. Confusion exists in the mind, not in the reality, and trying to treat it like something you can pick up and move around, will only result in unintentional comedy.

Similarly, you cannot come up with clever reasons why the gaps in your model don't matter. You cannot draw a border around the mystery, put on neat handles that let you use the Mysterious Thing without really understanding it - like my attempt to make the possibility that life is meaningless cancel out of an expected utility formula. You can't pick up the gap and manipulate it.

If the blank spot on your map conceals a land mine, then putting your weight down on that spot will be fatal, no matter how good your excuse for not knowing. Any black box could contain a trap, and there's no way to know except opening up the black box and looking inside. If you come up with some righteous justification for why you need to rush on ahead with the best understanding you have - the trap goes off.

It's only when you know the rules,
That you realize why you needed to learn;
What would have happened otherwise,
How much you needed to know.

Only knowledge can foretell the cost of ignorance. The ancient alchemists had no logical way of knowing the exact reasons why it was hard for them to turn lead into gold. So they poisoned themselves and died. Nature doesn't care.

But there did come a time when realization began to dawn on me. To be continued.

Sequence: Yudkowsky's Coming of Age

A Prodigy of Refutation

That Tiny Note of Discord

Referenced by: That Tiny Note of Discord • Protected From Myself • Visualizing Eutopia • Prolegomena to a Theory of Fun • Living By Your Own Strength • Devil's Offers • Formative Youth

Original with comments: The Sheer Folly of Callow Youth

Say It Loud

Eliezer Yudkowsky, 19 September 2008 05:34PM

Reply to: Overconfidence is Stylish

I respectfully defend my lord Will Strunk:

"If you don't know how to pronounce a word, say it loud! If you don't know how to pronounce a word, say it loud!" This comical piece of advice struck me as sound at the time, and I still respect it. Why compound ignorance with inaudibility? Why run and hide?

How does being vague, tame, colorless, irresolute, help someone to understand your current state of uncertainty? Any more than mumbling helps them understand a word you aren't sure how to pronounce?

Goofus says: "The sky, if such a thing exists at all, might or might not have a property of color, but, if it does have color, then I feel inclined to state that it might be green."

Gallant says: "70% probability the sky is green."

Which of them sounds more confident, more definite?

But which of them has managed to quickly communicate their state of uncertainty?

(And which of them is more likely to actually, in real life, spend any time planning and preparing for the eventuality that the sky is blue?)

I am often accused of overconfidence because my audience is not familiar with the concept of there being iron laws that govern the manipulation of uncertainty. Just because I don't know the object-level doesn't necessarily mean that I am in a state of fear and doubt as to what I should be thinking. That comes through in my writing, and so I sound confident even when I am in the midst of manipulating uncertainty. That might be a disadvantage in my attempts to communicate; but I would rather clearly describe my state of uncertainty, and worry afterward about how that makes me look.

And similarly, I have often seen people who spend no effort at all on possibilities other than their mainline, praised for their seeming humility, on account of their indefinite language. They are skilled at sounding uncertain, which makes them appear modest; but not skilled at handling uncertainty. That is a political advantage, but it doesn't help them think. Also the audience is given more slack to interpret the speaker as being on their side; but to deliberately exploit this effect is dishonesty.

Often the caveats we attach to our speech have little to do with any actual humility - actual plans we prepared, and actions we took, against the eventuality of things turning out the other way. And more to do with being able to avoid admitting to ourselves that we were wrong. We attached a caveat, didn't we?

Maybe Will Strunk did think it was better to be wrong than irresolute (though that doesn't quite seem to have been a direct quote from him). If so, then that was Will Strunk's flaw as a rationalist. Presumably he only knew the part of rationality that pertained to writing.

But the core of Will Strunk's lesson learned from the art of writing, not to obscure your position when you are unsure of it, seems to me very wise indeed. In particular you should not obscure your position from yourself.

Original with comments: Say It Loud

Ban the Bear

Eliezer Yudkowsky, 19 September 2008 06:14PM

I applaud the SEC's courageous move to ban short selling. Isn't that brilliant? I wonder why they didn't think of that during the Great Depression.

However, I feel that this valiant effort does not go far enough.

All selling of stocks should be banned. Once you buy a stock, you have to hold it forever.

Sure, this might make the market a little less liquid. But once stock prices can only go up, we'll all be rich!

Or maybe we should just try something simpler: pass a law making it illegal for stock prices to go down.

Original with comments: Ban the Bear

How Many LHC Failures Is Too Many?

Eliezer Yudkowsky, 20 September 2008 09:38PM

Recently the Large Hadron Collider was damaged by a mechanical failure. This requires the collider to be warmed up, repaired, and then cooled down again, so we're looking at a two-month delay.

Inevitably, many commenters said, "Anthropic principle! If the LHC had worked, it would have produced a black hole or strangelet or vacuum failure, and we wouldn't be here!"

This remark may be somewhat premature, since I don't think we're yet at the point in time when the LHC would have started producing collisions if not for this malfunction. However, a few weeks(?) from now, the "Anthropic!" hypothesis will start to make sense, assuming it can make sense at all. (Does this mean we can foresee executing a future probability update, but can't go ahead and update now?)

As you know, I don't spend much time worrying about the Large Hadron Collider when I've got much larger existential-risk-fish to fry. However, there's an exercise in probability theory (which I first picked up from E.T. Jaynes) along the lines of, "How many times does a coin have to come up heads before you believe the coin is fixed?" This tells you how low your prior probability is for the hypothesis. If a coin comes up heads only twice, that's definitely not a good reason to believe it's fixed, unless you already suspected from the beginning. But if it comes up heads 100 times, it's taking you too long to notice.

So - taking into account the previous cancellation of the Superconducting Supercollider (SSC) - how many times does the LHC have to fail before you'll start considering an anthropic explanation? 10? 20? 50?

After observing empirically that the LHC had failed 100 times in a row, would you endorse a policy of keeping the LHC powered up, but trying to fire it again only in the event of, say, nuclear terrorism or a global economic crash?

Referenced by: Horrible LHC Inconsistency

Original with comments: How Many LHC Failures Is Too Many?

Horrible LHC Inconsistency

Eliezer Yudkowsky, 22 September 2008 03:12AM

Followup to: When (Not) To Use Probabilities, How Many LHC Failures Is Too Many?

While trying to answer my own question on "How Many LHC Failures Is Too Many?" I realized that I'm horrendously inconsistent with respect to my stated beliefs about disaster risks from the Large Hadron Collider.

First, I thought that stating a "one-in-a-million" probability for the Large Hadron Collider destroying the world was too high, in the sense that I would much rather run the Large Hadron Collider than press a button with a known 1/1,000,000 probability of destroying the world.

But if you asked me whether I could make one million statements of authority equal to "The Large Hadron Collider will not destroy the world", and be wrong, on average, around once, then I would have to say no.

Unknown pointed out that this turns me into a money pump. Given a portfolio of a million existential risks to which I had assigned a "less than one in a million probability", I would rather press the button on the fixed-probability device than run a random risk from this portfolio; but would rather take any particular risk in this portfolio than press the button.

Then, I considered the question of how many mysterious failures at the LHC it would take to make me question whether it might destroy the world/universe somehow, and what this revealed about my prior probability.

If the failure probability had a known 50% probability of occurring from natural causes, like a quantum coin or some such... then I suspect that if I actually saw that coin come up heads 20 times in a row, I would feel a strong impulse to bet on it coming up heads the next time around. (And that's taking into account my uncertainty about whether the anthropic principle really works that way.)

Even having noticed this triple inconsistency, I'm not sure in which direction to resolve it!

(But I still maintain my resolve that the LHC is not worth expending political capital, financial capital, or our time to shut down; compared with using the same capital to worry about superhuman intelligence or nanotechnology.)

Original with comments: Horrible LHC Inconsistency

That Tiny Note of Discord

Eliezer Yudkowsky, 23 September 2008 06:02AM

Followup to: The Sheer Folly of Callow Youth

When we last left Eliezer_1997, he believed that any superintelligence would automatically do what was "right", and indeed would understand that better than we could; even though, he modestly confessed, he did not understand the ultimate nature of morality. Or rather, after some debate had passed, Eliezer_1997 had evolved an elaborate argument, which he fondly claimed to be "formal", that we could always condition upon the belief that life has meaning; and so cases where superintelligences did not feel compelled to do anything in particular, would fall out of consideration. (The flaw being the unconsidered and unjustified equation of "universally compelling argument" with "right".)

So far, the young Eliezer is well on the way toward joining the "smart people who are stupid because they're skilled at defending beliefs they arrived at for unskilled reasons". All his dedication to "rationality" has not saved him from this mistake, and you might be tempted to conclude that it is useless to strive for rationality.

But while many people dig holes for themselves, not everyone succeeds in clawing their way back out.

And from this I learn my lesson: That it all began -

- with a small, small question; a single discordant note; one tiny lonely thought...

As our story starts, we advance three years to Eliezer_2000, who in most respects resembles his self of 1997. He currently thinks he's proven that building a superintelligence is the right thing to do if there is any right thing at all. From which it follows that there is no justifiable conflict of interest over the Singularity, among the peoples and persons of Earth.

This is an important conclusion for Eliezer_2000, because he finds the notion of fighting over the Singularity to be unbearably stupid. (Sort of like the notion of God intervening in fights between tribes of bickering barbarians, only in reverse.) Eliezer_2000's self-concept does not permit him - he doesn't even want - to shrug and say, "Well, my side got here first, so we're going to seize the banana before anyone else gets it." It's a thought too painful to think.

And yet then the notion occurs to him:

Maybe some people would prefer an AI do particular things, such as not kill them, even if life is meaningless?

His immediately following thought is the obvious one, given his premises:

In the event that life is meaningless, nothing is the "right" thing to do; therefore it wouldn't be particularly right to respect people's preferences in this event.

This is the obvious dodge. The thing is, though, Eliezer_2000 doesn't think of himself as a villain. He doesn't go around saying, "What bullets shall I dodge today?" He thinks of himself as a dutiful rationalist who tenaciously follows lines of inquiry. Later, he's going to look back and see a whole lot of inquiries that his mind somehow managed to not follow - but that's not his current self-concept.

So Eliezer_2000doesn't just grab the obvious out. He keeps thinking.

But if people believe they have preferences in the event that life is meaningless, then they have a motive to dispute my Singularity project and go with a project that respects their wish in the event life is meaningless. This creates a present conflict of interest over the Singularity, and prevents right things from getting done in the mainline event that life is meaningful.

Now, there's a lot of excuses Eliezer_2000 could have potentially used to toss this problem out the window. I know, because I've heard plenty of excuses for dismissing Friendly AI. "The problem is too hard to solve" is one I get from AGI wannabes who imagine themselves smart enough to create true Artificial Intelligence, but not smart enough to solve a really difficult problem like Friendly AI. Or "worrying about this possibility would be a poor use of resources, what with the incredible urgency of creating AI before humanity wipes itself out - you've got to go with what you have", this being uttered by people who just basically aren't interested in the problem.

But Eliezer_2000 is a perfectionist. He's not perfect, obviously, and he doesn't attach as much importance as I do to the virtue of precision, but he is most certainly a perfectionist. The idea of metaethics that Eliezer_2000 espouses, in which superintelligences know what's right better than we do, previously seemed to wrap up all the problems of justice and morality in an airtight wrapper.

The new objection seems to poke a minor hole in the airtight wrapper. This is worth patching. If you have something that's perfect, are you really going to let one little possibility compromise it?

So Eliezer_2000 doesn't even want to drop the issue; he wants to patch the problem and restore perfection. How can he justify spending the time? By thinking thoughts like:

What about Brian Atkins? [Brian Atkins being the startup funder of the Singularity Institute.] He would probably prefer not to die, even if life were meaningless. He's paying for the Singularity Institute right now; I don't want to taint the ethics of our cooperation.

Eliezer_2000's sentiment doesn't translate very well - English doesn't have a simple description for it, or any other culture I know. Maybe the passage in the Old Testament, "Thou shalt not boil a young goat in its mother's milk". Someone who helps you out of altruism shouldn't regret helping you; you owe them, not so much fealty, but rather, that they're actually doing what they think they're doing by helping you.

Well, but how would Brian Atkins find out, if I don't tell him? Eliezer_2000 doesn't even think this except in quotation marks, as the obvious thought that a villain would think in the same situation. And Eliezer_2000 has a standard counter-thought ready too, a ward against temptations to dishonesty - an argument that justifies honesty in terms of expected utility, not just a personal love of personal virtue:

Human beings aren't perfect deceivers; it's likely that I'll be found out. Or what if genuine lie detectors are invented before the Singularity, sometime over the next thirty years? I wouldn't be able to pass a lie detector test.

Eliezer_2000 lives by the rule that you should always be ready to have your thoughts broadcast to the whole world at any time, without embarrassment. Otherwise, clearly, you've fallen from grace: either you're thinking something you shouldn't be thinking, or you're embarrassed by something that shouldn't embarrass you.

(These days, I don't espouse quite such an extreme viewpoint, mostly for reasons of Fun Theory. I see a role for continued social competition between intelligent life-forms, as least as far as my near-term vision stretches. I admit, these days, that it might be all right for human beings to have a self; as John McCarthy put it, "If everyone were to live for others all the time, life would be like a procession of ants following each other around in a circle." If you're going to have a self, you may as well have secrets, and maybe even conspiracies. But I do still try to abide by the principle of being able to pass a future lie detector test, with anyone else who's also willing to go under the lie detector, if the topic is a professional one. Fun Theory needs a commonsense exception for global catastrophic risk management.)

Even taking honesty for granted, there are other excuses Eliezer_2000 could use to flush the question down the toilet. "The world doesn't have the time" or "It's unsolvable" would still work. But Eliezer 2000 doesn't know that this problem, the "backup" morality problem, is going to be particularly difficult or time-consuming. He's just now thought of the whole issue.

And so Eliezer_2000begins to really consider the question: Supposing that "life is meaningless" (that superintelligences don't produce their own motivations from pure logic), then how would you go about specifying a fallback morality? Synthesizing it, inscribing it into the AI?

There's a lot that Eliezer_2000 doesn't know, at this point. But he has been thinking about self-improving AI for three years, and he's been a Traditional Rationalist for longer than that. There are techniques of rationality that he has practiced, methodological safeguards he's already devised. He already knows better than to think that all an AI needs is the One Great Moral Principle. Eliezer_2000 already knows that it is wiser to think technologically than politically. He already knows the saying that AI programmers are supposed to think in code, to use concepts that can be inscribed in a computer. Eliezer_2000 already has a concept that there is something called "technical thinking" and it is good, though he hasn't yet formulated a Bayesian view of it. And he's long since noticed that suggestively named LISP tokens don't really mean anything, etcetera. These injunctions prevent him from falling into some of the initial traps, the ones that I've seen consume other novices on their own first steps into the Friendly AI problem... though technically this was my second step; I well and truly failed on my first.

But in the end, what it comes down to is this: For the first time, Eliezer_2000 is trying to think technically about inscribing a morality into an AI, without the escape-hatch of the mysterious essence of rightness.

That's the only thing that matters, in the end. His previous philosophizing wasn't enough to force his brain to confront the details. This new standard is strict enough to require actual work. Morality slowly starts being less mysterious to him - Eliezer_2000 is starting to think inside the black box.

His reasons for pursuing this course of action - those don't matter at all.

Oh, there's a lesson in his being a perfectionist. There's a lesson in the part about how Eliezer_2000 initially thought this was a tiny flaw, and could have dismissed it out-of-mind if that had been his impulse.

But in the end, the chain of cause and effect goes like this: Eliezer_2000 investigated in more detail, therefore he got better with practice. Actions screen off justifications. If your arguments happen to justify not working things out in detail, like Eliezer_1996, then you won't get good at thinking about the problem. If your arguments call for you to work things out in detail, then you have an opportunity to start accumulating expertise.

That was the only choice that mattered, in the end - not the reasons for doing anything.

I say all this, as you may well guess, because of the AI wannabes I sometimes run into, who have their own clever reasons for not thinking about the Friendly AI problem. Our clever reasons for doing what we do, tend to matter a lot less to Nature than they do to ourselves and our friends. If your actions don't look good when they're stripped of all their justifications and presented as mere brute facts... then maybe you should re-examine them.

A diligent effort won't always save a person. There is such a thing as lack of ability. Even so, if you don't try, or don't try hard enough, you don't get a chance to sit down at the high-stakes table - never mind the ability ante. That's cause and effect for you.

Also, perfectionism really matters. The end of the world doesn't always come with trumpets and thunder and the highest priority in your inbox. Sometimes the shattering truth first presents itself to you as a small, small question; a single discordant note; one tiny lonely thought, that you could dismiss with one easy effortless touch...

...and so, over succeeding years, understanding begins to dawn on that past Eliezer, slowly. That sun rose slower than it could have risen. To be continued.

Sequence: Yudkowsky's Coming of Age

The Sheer Folly of Callow Youth

Fighting a Rearguard Action Against the Truth

Referenced by: The Sheer Folly of Callow Youth • Fighting a Rearguard Action Against the Truth • My Bayesian Enlightenment • Protected From Myself

Original with comments: That Tiny Note of Discord

Fighting a Rearguard Action Against the Truth

Eliezer Yudkowsky, 24 September 2008 01:23AM

Followup to: That Tiny Note of Discord, The Importance of Saying "Oops"

When we last left Eliezer_2000, he was just beginning to investigate the question of how to inscribe a morality into an AI. His reasons for doing this don't matter at all, except insofar as they happen to historically demonstrate the importance of perfectionism. If you practice something, you may get better at it; if you investigate something, you may find out about it; the only thing that matters is that Eliezer_2000 is, in fact, focusing his full-time energies on thinking technically about AI morality; rather than, as previously, finding an justification for not spending his time this way. In the end, this is all that turns out to matter.

But as our story begins - as the sky lightens to gray and the tip of the sun peeks over the horizon - Eliezer_2001 hasn't yet admitted that Eliezer_1997 was mistaken in any important sense. He's just making Eliezer_1997's strategy even better by including a contingency plan for "the unlikely event that life turns out to be meaningless"...

...which means that Eliezer_2001 now has a line of retreat away from his mistake.

I don't just mean that Eliezer_2001 can say "Friendly AI is a contingency plan", rather than screaming "OOPS!"

I mean that Eliezer_2001 now actually has a contingency plan. If Eliezer_2001 starts to doubt his 1997 metaethics, the Singularity has a fallback strategy, namely Friendly AI. Eliezer_2001 can question his metaethics without it signaling the end of the world.

And his gradient has been smoothed; he can admit a 10% chance of having previously been wrong, then a 20% chance. He doesn't have to cough out his whole mistake in one huge lump.

If you think this sounds like Eliezer_2001 is too slow, I quite agree.

Eliezer1996-2000's strategies had been formed in the total absence of "Friendly AI" as a consideration. The whole idea was to get a superintelligence, any superintelligence, as fast as possible - codelet soup, ad-hoc heuristics, evolutionary programming, open-source, anything that looked like it might work - preferably all approaches simultaneously in a Manhattan Project. ("All parents did the things they tell their children not to do. That's how they know to tell them not to do it." John Moore, Slay and Rescue.) It's not as if adding one more approach could hurt.

His attitudes toward technological progress have been formed - or more accurately, preserved from childhood-absorbed technophilia - around the assumption that any/all movement toward superintelligence is a pure good without a hint of danger.

Looking back, what Eliezer_2001needed to do at this point was declare an HMC event - Halt, Melt, and Catch Fire. One of the foundational assumptions on which everything else has been built, has been revealed as flawed. This calls for a mental brake to a full stop: take your weight off all beliefs built on the wrong assumption, do your best to rethink everything from scratch. This is an art I need to write more about - it's akin to the convulsive effort required to seriously clean house, after an adult religionist notices for the first time that God doesn't exist.

But what Eliezer_2001 actually did was rehearse his previous technophilic arguments for why it's difficult to ban or governmentally control new technologies - the standard arguments against "relinquishment".

It does seem even to my modern self, that all those awful consequences which technophiles argue to follow from various kinds of government regulation, are more or less correct - it's much easier to say what someone is doing wrong, than to say the way that is right. My modern viewpoint hasn't shifted to think that technophiles are wrong about the downsides of technophobia; but I do tend to be a lot more sympathetic to what technophobes say about the downsides of technophilia. What previous Eliezers said about the difficulties of, e.g., the government doing anything sensible about Friendly AI, still seems pretty true. It's just that a lot of his hopes for science, or private industry, etc., now seem equally wrongheaded.

Still, let's not get into the details of the technovolatile viewpoint. Eliezer_2001 has just tossed a major foundational assumption - that AI can't be dangerous, unlike other technologies - out the window. You would intuitively suspect that this should have some kind of large effect on his strategy.

Well, Eliezer_2001 did at least give up on his 1999 idea of an open-source AI Manhattan Project using self-modifying heuristic soup, but overall...

Overall, he'd previously wanted to charge in, guns blazing, immediately using his best idea at the time; and afterward he still wanted to charge in, guns blazing. He didn't say, "I don't know how to do this." He didn't say, "I need better knowledge." He didn't say, "This project is not yet ready to start coding." It was still all, "The clock is ticking, gotta move now! The Singularity Institute will start coding as soon as it's got enough money!"

Before, he'd wanted to focus as much scientific effort as possible with full information-sharing, and afterward he still thought in those terms. Scientific secrecy = bad guy, openness = good guy. (Eliezer_2001 hadn't read up on the Manhattan Project and wasn't familiar with the similar argument that Leo Szilard had with Enrico Fermi.)

That's the problem with converting one big "Oops!" into a gradient of shifting probability. It means there isn't a single watershed moment - a visible huge impact - to hint that equally huge changes might be in order.

Instead, there are all these little opinion shifts... that give you a chance to repair the arguments for your strategies; to shift the justification a little, but keep the "basic idea" in place. Small shocks that the system can absorb without cracking, because each time, it gets a chance to go back and repair itself. It's just that in the domain of rationality, cracking = good, repair = bad. In the art of rationality it's far more efficient to admit one huge mistake, than to admit lots of little mistakes.

There's some kind of instinct humans have, I think, to preserve their former strategies and plans, so that they aren't constantly thrashing around and wasting resources; and of course an instinct to preserve any position that we have publicly argued for, so that we don't suffer the humiliation of being wrong. And though the younger Eliezer has striven for rationality for many years, he is not immune to these impulses; they waft gentle influences on his thoughts, and this, unfortunately, is more than enough damage.

Even in 2002, the earlier Eliezer isn't yet sure that Eliezer_1997's plan couldn't possibly have worked. It might have gone right. You never know, right?

But there came a time when it all fell crashing down. To be continued.

Sequence: Yudkowsky's Coming of Age

That Tiny Note of Discord

My Naturalistic Awakening

Referenced by: My Naturalistic Awakening

Original with comments: Fighting a Rearguard Action Against the Truth

My Naturalistic Awakening

Eliezer Yudkowsky, 25 September 2008 06:58AM

Followup to: Fighting a Rearguard Action Against the Truth

In yesterday's episode, Eliezer_2001 is fighting a rearguard action against the truth. Only gradually shifting his beliefs, admitting an increasing probability in a different scenario, but never saying outright, "I was wrong before." He repairs his strategies as they are challenged, finding new justifications for just the same plan he pursued before.

(Of which it is therefore said: "Beware lest you fight a rearguard retreat against the evidence, grudgingly conceding each foot of ground only when forced, feeling cheated. Surrender to the truth as quickly as you can. Do this the instant you realize what you are resisting; the instant you can see from which quarter the winds of evidence are blowing against you.")

Memory fades, and I can hardly bear to look back upon those times - no, seriously, I can't stand reading my old writing. I've already been corrected once in my recollections, by those who were present. And so, though I remember the important events, I'm not really sure what order they happened in, let alone what year.

But if I had to pick a moment when my folly broke, I would pick the moment when I first comprehended, in full generality, the notion of an optimization process. That was the point at which I first looked back and said, "I've been a fool."

Previously, in 2002, I'd been writing a bit about the evolutionary psychology of human general intelligence - though at the time, I thought I was writing about AI; at this point I thought I was against anthropomorphic intelligence, but I was still looking to the human brain for inspiration. (The paper in question is "Levels of Organization in General Intelligence", a requested chapter for the volume "Artificial General Intelligence", which finally came out in print in 2007.)

So I'd been thinking (and writing) about how natural selection managed to cough up human intelligence; I saw a dichotomy between them, the blindness of natural selection and the lookahead of intelligent foresight, reasoning by simulation versus playing everything out in reality, abstract versus concrete thinking. And yet it was natural selection that created human intelligence, so that our brains, though not our thoughts, are entirely made according to the signature of natural selection.

To this day, this still seems to me like a reasonably shattering insight, and so it drives me up the wall when people lump together natural selection and intelligence-driven processes as "evolutionary". They really are almost absolutely different in a number of important ways - though there are concepts in common that can be used to describe them, like consequentialism and cross-domain generality.

But that Eliezer_2002 is thinking in terms of a dichotomy between evolution and intelligence tells you something about the limits of his vision - like someone who thinks of politics as a dichotomy between conservative and liberal stances, or someone who thinks of fruit as a dichotomy between apples and strawberries.

After the "Levels of Organization" draft was published online, Emil Gilliam pointed out that my view of AI seemed pretty similar to my view of intelligence. Now, of course Eliezer_2002 doesn't espouse building an AI in the image of a human mind; Eliezer_2002 knows very well that a human mind is just a hack coughed up by natural selection. But Eliezer_2002 has described these levels of organization in human thinking, and he hasn't proposed using different levels of organization in the AI. Emil Gilliam asks whether I think I might be hewing too close to the human line. I dub the alternative the "Completely Alien Mind Design" and reply that a CAMD is probably too difficult for human engineers to create, even if it's possible in theory, because we wouldn't be able to understand something so alien while we were putting it together.

I don't know if Eliezer_2002 invented this reply on his own, or if he read it somewhere else. Needless to say, I've heard this excuse plenty of times since then. In reality, what you genuinely understand, you can usually reconfigure in almost any sort of shape, leaving some structural essence inside; but when you don't understand flight, you suppose that a flying machine needs feathers, because you can't imagine departing from the analogy of a bird.

So Eliezer_2002 is still, in a sense, attached to humanish mind designs - he imagines improving on them, but the human architecture is still in some sense his point of departure.

What is it that finally breaks this attachment?

It's an embarrassing confession: It came from a science-fiction story I was trying to write. (No, you can't see it; it's not done.) The story involved a non-cognitive non-evolutionary optimization process; something like an Outcome Pump. Not intelligence, but a cross-temporal physical effect - that is, I was imagining it as a physical effect - that narrowly constrained the space of possible outcomes. (I can't tell you any more than that; it would be a spoiler, if I ever finished the story. Just see the post on Outcome Pumps.) It was "just a story", and so I was free to play with the idea and elaborate it out logically: C was constrained to happen, therefore B (in the past) was constrained to happen, therefore A (which led to B) was constrained to happen.

Drawing a line through one point is generally held to be dangerous. Two points make a dichotomy; you imagine them opposed to one another. But when you've got three different points - that's when you're forced to wake up and generalize.

Now I had three points: Human intelligence, natural selection, and my fictional plot device.

And so that was the point at which I generalized the notion of an optimization process, of a process that squeezes the future into a narrow region of the possible.

This may seem like an obvious point, if you've been following Overcoming Bias this whole time; but if you look at Shane Legg's collection of 71 definitions of intelligence, you'll see that "squeezing the future into a constrained region" is a less obvious reply than it seems.

Many of the definitions of "intelligence" by AI researchers, do talk about "solving problems" or "achieving goals". But from the viewpoint of past Eliezers, at least, it is only hindsight that makes this the same thing as "squeezing the future".

A goal is a mentalistic object; electrons have no goals, and solve no problems either. When a human imagines a goal, they imagine an agent imbued with wanting-ness - it's still empathic language.

You can espouse the notion that intelligence is about "achieving goals" - and then turn right around and argue about whether some "goals" are better than others - or talk about the wisdom required to judge between goals themselves - or talk about a system deliberately modifying its goals - or talk about the free will needed to choose plans that achieve goals - or talk about an AI realizing that its goals aren't what the programmers really meant to ask for. If you imagine something that squeezes the future into a narrow region of the possible, like an Outcome Pump, those seemingly sensible statements somehow don't translate.

So for me at least, seeing through the word "mind", to a physical process that would, just by naturally running, just by obeying the laws of physics, end up squeezing its future into a narrow region, was a naturalistic enlightenment over and above the notion of an agent trying to achieve its goals.

It was like falling out of a deep pit, falling into the ordinary world, strained cognitive tensions relaxing into unforced simplicity, confusion turning to smoke and drifting away. I saw the work performed by intelligence; smart was no longer a property, but an engine. Like a knot in time, echoing the outer part of the universe in the inner part, and thereby steering it. I even saw, in a flash of the same enlightenment, that a mind had to output waste heat in order to obey the laws of thermodynamics.

Previously, Eliezer_2001 had talked about Friendly AI as something you should do just to be sure - if you didn't know whether AI design X was going to be Friendly, then you really ought to go with AI design Y that you did know would be Friendly. But Eliezer_2001 didn't think he knew whether you could actually have a superintelligence that turned its future light cone into paperclips.

Now, though, I could see it - the pulse of the optimization process, sensory information surging in, motor instructions surging out, steering the future. In the middle, the model that linked up possible actions to possible outcomes, and the utility function over the outcomes. Put in the corresponding utility function, and the result would be an optimizer that would steer the future anywhere.

Up until that point, I'd never quite admitted to myself that Eliezer_1997's AI goal system design would definitely, no two ways about it, pointlessly wipe out the human species. Now, however, I looked back, and I could finally see what my old design really did, to the extent it was coherent enough to be talked about. Roughly, it would have converted its future light cone into generic tools - computers without programs to run, stored energy without a use...

...how on Earth had I, the fine and practiced rationalist, how on Earth had I managed to miss something that obvious, for six damned years?

That was the point at which I awoke clear-headed, and remembered; and thought, with a certain amount of embarrassment: I've been stupid.

To be continued.

Sequence: Yudkowsky's Coming of Age

Fighting a Rearguard Action Against the Truth

The Level Above Mine

Referenced by: The Magnitude of His Own Folly • My Bayesian Enlightenment • Sufficiently Advanced Sanity

Original with comments: My Naturalistic Awakening

The Level Above Mine

Eliezer Yudkowsky, 26 September 2008 09:18AM

Followup to: The Proper Use of Humility, Tsuyoku Naritai

(At this point, I fear that I must recurse into a subsequence; but if all goes as planned, it really will be short.)

I once lent Xiaoguang "Mike" Li my copy of "Probability Theory: The Logic of Science". Mike Li read some of it, and then came back and said:

"Wow... it's like Jaynes is a thousand-year-old vampire."

Then Mike said, "No, wait, let me explain that -" and I said, "No, I know exactly what you mean." It's a convention in fantasy literature that the older a vampire gets, the more powerful they become.

I'd enjoyed math proofs before I encountered Jaynes. But E.T. Jaynes was the first time I picked up a sense of formidability from mathematical arguments. Maybe because Jaynes was lining up "paradoxes" that had been used to object to Bayesianism, and then blasting them to pieces with overwhelming firepower - power being used to overcome others. Or maybe the sense of formidability came from Jaynes not treating his math as a game of aesthetics; Jaynes cared about probability theory, it was bound up with other considerations that mattered, to him and to me too.

For whatever reason, the sense I get of Jaynes is one of terrifying swift perfection - something that would arrive at the correct answer by the shortest possible route, tearing all surrounding mistakes to shreds in the same motion. Of course, when you write a book, you get a chance to show only your best side. But still.

It spoke well of Mike Li that he was able to sense the aura of formidability surrounding Jaynes. It's a general rule, I've observed, that you can't discriminate between levels too far above your own. E.g., someone once earnestly told me that I was really bright, and "ought to go to college". Maybe anything more than around one standard deviation above you starts to blur together, though that's just a cool-sounding wild guess.

So, having heard Mike Li compare Jaynes to a thousand-year-old vampire, one question immediately popped into my mind:

"Do you get the same sense off me?" I asked.

Mike shook his head. "Sorry," he said, sounding somewhat awkward, "it's just that Jaynes is..."

"No, I know," I said. I hadn't thought I'd reached Jaynes's level. I'd only been curious about how I came across to other people.

I aspire to Jaynes's level. I aspire to become as much the master of Artificial Intelligence / reflectivity, as Jaynes was master of Bayesian probability theory. I can even plead that the art I'm trying to master is more difficult than Jaynes's, making a mockery of deference. Even so, and embarrassingly, there is no art of which I am as much the master now, as Jaynes was of probability theory.

This is not, necessarily, to place myself beneath Jaynes as a person - to say that Jaynes had a magical aura of destiny, and I don't.

Rather I recognize in Jaynes a level of expertise, of sheer formidability, which I have not yet achieved. I can argue forcefully in my chosen subject, but that is not the same as writing out the equations and saying: DONE.

For so long as I have not yet achieved that level, I must acknowledge the possibility that I can never achieve it, that my native talent is not sufficient. When Marcello Herreshoff had known me for long enough, I asked him if he knew of anyone who struck him as substantially more natively intelligent than myself. Marcello thought for a moment and said "John Conway - I met him at a summer math camp." Darn, I thought, he thought of someone, and worse, it's some ultra-famous old guy I can't grab. I inquired how Marcello had arrived at the judgment. Marcello said, "He just struck me as having a tremendous amount of mental horsepower," and started to explain a math problem he'd had a chance to work on with Conway.

Not what I wanted to hear.

Perhaps, relative to Marcello's experience of Conway and his experience of me, I haven't had a chance to show off on any subject that I've mastered as thoroughly as Conway had mastered his many fields of mathematics.

Or it might be that Conway's brain is specialized off in a different direction from mine, and that I could never approach Conway's level on math, yet Conway wouldn't do so well on AI research.

Or...

...or I'm strictly dumber than Conway, dominated by him along all dimensions. Maybe, if I could find a young proto-Conway and tell them the basics, they would blaze right past me, solve the problems that have weighed on me for years, and zip off to places I can't follow.

Is it damaging to my ego to confess that last possibility? Yes. It would be futile to deny that.

Have I really accepted that awful possibility, or am I only pretending to myself to have accepted it? Here I will say: "No, I think I have accepted it." Why do I dare give myself so much credit? Because I've invested specific effort into that awful possibility. I am blogging here for many reasons, but a major one is the vision of some younger mind reading these words and zipping off past me. It might happen, it might not.

Or sadder: Maybe I just wasted too much time on setting up the resources to support me, instead of studying math full-time through my whole youth; or I wasted too much youth on non-mathy ideas. And this choice, my past, is irrevocable. I'll hit a brick wall at 40, and there won't be anything left but to pass on the resources to another mind with the potential I wasted, still young enough to learn. So to save them time, I should leave a trail to my successes, and post warning signs on my mistakes.

Such specific efforts predicated on an ego-damaging possibility - that's the only kind of humility that seems real enough for me to dare credit myself. Or giving up my precious theories, when I realized that they didn't meet the standard Jaynes had shown me - that was hard, and it was real. Modest demeanors are cheap. Humble admissions of doubt are cheap. I've known too many people who, presented with a counterargument, say "I am but a fallible mortal, of course I could be wrong" and then go on to do exactly what they planned to do previously.

You'll note that I don't try to modestly say anything like, "Well, I may not be as brilliant as Jaynes or Conway, but that doesn't mean I can't do important things in my chosen field."

Because I do know... that's not how it works.

Sequence: Yudkowsky's Coming of Age

My Naturalistic Awakening

Competent Elites

Referenced by: Competent Elites • Above-Average AI Scientists • My Bayesian Enlightenment • Ethical Injunctions

Original with comments: The Level Above Mine

Competent Elites

Eliezer Yudkowsky, 27 September 2008 12:07AM

Followup to: The Level Above Mine

(Anyone who didn't like yesterday's post should probably avoid this one.)

I remember what a shock it was to first meet Steve Jurvetson, of the venture capital firm Draper Fisher Jurvetson.

Steve Jurvetson talked fast and articulately, could follow long chains of reasoning, was familiar with a wide variety of technologies, and was happy to drag in analogies from outside sciences like biology - good ones, too.

I once saw Eric Drexler present an analogy between biological immune systems and the "active shield" concept in nanotechnology, arguing that just as biological systems managed to stave off invaders without the whole community collapsing, nanotechnological immune systems could do the same.

I thought this was a poor analogy, and was going to point out some flaws during the Q&A. But Steve Jurvetson, who was in line before me, proceeded to demolish the argument even more thoroughly. Jurvetson pointed out the evolutionary tradeoff between virulence and transmission that keeps natural viruses in check, talked about how greater interconnectedness led to larger pandemics - it was very nicely done, demolishing the surface analogy by correct reference to deeper biological details.

I was shocked, meeting Steve Jurvetson, because from everything I'd read about venture capitalists before then, VCs were supposed to be fools in business suits, who couldn't understand technology or engineers or the needs of a fragile young startup, but who'd gotten ahold of large amounts of money by dint of seeming reliable to other business suits.

One of the major surprises I received when I moved out of childhood into the real world, was the degree to which the world is stratified by genuine competence.

Now, yes, Steve Jurvetson is not just a randomly selected big-name venture capitalist. He is a big-name VC who often shows up at transhumanist conferences. But I am not drawing a line through just one data point.

I was invited once to a gathering of the mid-level power elite, where around half the attendees were "CEO of something" - mostly technology companies, but occasionally "something" was a public company or a sizable hedge fund. I was expecting to be the youngest person there, but it turned out that my age wasn't unusual - there were several accomplished individuals who were younger. This was the point at which I realized that my child prodigy license had officially completely expired.

Now, admittedly, this was a closed conference run by people clueful enough to think "Let's invite Eliezer Yudkowsky" even though I'm not a CEO. So this was an incredibly cherry-picked sample. Even so...

Even so, these people of the Power Elite were visibly much smarter than average mortals. In conversation they spoke quickly, sensibly, and by and large intelligently. When talk turned to deep and difficult topics, they understood faster, made fewer mistakes, were readier to adopt others' suggestions.

No, even worse than that, much worse than that: these CEOs and CTOs and hedge-fund traders, these folk of the mid-level power elite, seemed happier and more alive.

This, I suspect, is one of those truths so horrible that you can't talk about it in public. This is something that reporters must not write about, when they visit gatherings of the power elite.

Because the last news your readers want to hear, is that this person who is wealthier than you, is also smarter, happier, and not a bad person morally. Your reader would much rather read about how these folks are overworked to the bone or suffering from existential ennui. Failing that, your readers want to hear how the upper echelons got there by cheating, or at least smarming their way to the top. If you said anything as hideous as, "They seem more alive," you'd get lynched.

But I am an independent scholar, not much beholden. I should be able to say it out loud if anyone can. I'm talking about this topic... for more than one reason; but it is the truth as I see it, and an important truth which others don't talk about (in writing?). It is something that led me down wrong pathways when I was young and inexperienced.

I used to think - not from experience, but from the general memetic atmosphere I grew up in - that executives were just people who, by dint of superior charisma and butt-kissing, had managed to work their way to the top positions at the corporate hog trough.

No, that was just a more comfortable meme, at least when it comes to what people put down in writing and pass around. The story of the horrible boss gets passed around more than the story of the boss who is, not just competent, but more competent than you.

But entering the real world, I found out that the average mortal really can't be an executive. Even the average manager can't function without a higher-level manager above them. What is it that makes an executive? I don't know, because I'm not a professional in this area. If I had to take a guess, I would call it "functioning without recourse" - living without any level above you to take over if you falter, or even to tell you if you're getting it wrong. To just get it done, even if the problem requires you to do something unusual, without anyone being there to look over your work and pencil in a few corrections.

Now, I'm sure that there are plenty of people out there bearing executive titles who are not executives.

And yet there seem to be a remarkable number of people out there bearing executive titles who actually do have the executive-nature, who can thrive on the final level that gets the job done without recourse. I'm not going to take sides on whether today's executives are overpaid, but those executive titles occupied by actual executives, are not being paid for nothing. Someone who can be an executive at all, even a below-average executive, is a rare find.

The people who'd like to be boss of their company, to sit back in that comfortable chair with a lovely golden parachute - most of them couldn't make it. If you try to drop executive responsibility on someone who lacks executive-nature - on the theory that most people can do it if given the chance - then they'll melt and catch fire.

This is not the sort of unpleasant truth that anyone would warn you about - at least not in books, and all I had read were books. Who would say it? A reporter? It's not news that people want to hear. An executive? Who would believe that self-valuing story?

I expect that my life experience constitutes an extremely biased sample of the power elite. I don't have to deal with the executives of arbitrary corporations, or form business relationships with people I never selected. I just meet them at gatherings and talk to the interesting ones.

But the business world is not the only venue where I've encountered the upper echelons and discovered that, amazingly, they actually are better at what they do.

Case in point: Professor Rodney Brooks, CTO of iRobot and former director of the MIT AI Lab, who spoke at the 2007 Singularity Summit. I had previously known "Rodney Brooks" primarily as the promoter of yet another dreadful nouvelle paradigm in AI - the embodiment of AIs in robots, and the forsaking of deliberation for complicated reflexes that didn't involve modeling. Definitely not a friend to the Bayesian faction. Yet somehow Brooks had managed to become a major mainstream name, a household brand in AI...

And by golly, Brooks sounded intelligent and original. He gave off a visible aura of competence. (Though not a thousand-year vampire aura of terrifying swift perfection like E.T. Jaynes's carefully crafted book.) But Brooks could have held his own at any gathering I attended; from his aura I would put him at the Steve Jurvetson level or higher.

(Interesting question: If I'm not judging Brooks by the goodness of his AI theories, what is it that made him seem smart to me? I don't remember any stunning epiphanies in his presentation at the Summit. I didn't talk to him very long in person. He just came across as... formidable, somehow.)

The major names in an academic field, at least the ones that I run into, often do seem a lot smarter than the average scientist.

I tried - once - going to an interesting-sounding mainstream AI conference that happened to be in my area. I met ordinary research scholars and looked at their posterboards and read some of their papers. I watched their presentations and talked to them at lunch. And they were way below the level of the big names. I mean, they weren't visibly incompetent, they had their various research interests and I'm sure they were doing passable work on them. And I gave up and left before the conference was over, because I kept thinking "What am I even doing here?"

An intermediate stratum, above the ordinary scientist but below the ordinary CEO, is that of, say, partners at a non-big-name venture capital firm. The way their aura feels to me, is that they can hold up one end of an interesting conversation, but they don't sound very original, and they don't sparkle with extra life force.

I wonder if you have to reach the Jurvetson level before thinking outside the "Outside the Box" box starts to become a serious possibility. Or maybe that art can be taught, but isn't, and the Jurvetson level is where it starts to happen spontaneously. It's at this level that I talk to people and find that they routinely have interesting thoughts I haven't heard before.

Hedge-fund people sparkle with extra life force. At least the ones I've talked to. Large amounts of money seem to attract smart people. No, really.

If you're wondering how it could be possible that the upper echelons of the world could be genuinely intelligent, and yet the world is so screwed up...

Well, part of that may be due to my biased sample.

Also, I've met a few Congresspersons and they struck me as being at around the non-big-name venture capital level, not the hedge fund level or the Jurvetson level. (Still, note that e.g. George W. Bush used to sound a lot smarter than he does now.)

But mainly: It takes an astronomically high threshold of intelligence + experience + rationality before a screwup becomes surprising. There's "smart" and then there's "smart enough for your cognitive mechanisms to reliably decide to sign up for cryonics". Einstein was a deist, etc. See also Eliezer[1996] and the edited volume "How Smart People Can Be So Stupid". I've always been skeptical that Jeff Skilling of Enron was world-class smart, but I can easily visualize him being able to sparkle in conversation.

Still, so far as I can tell, the world's upper echelons - in those few cases I've tested, within that extremely biased sample that I encounter - really are more intelligent.

Not just, "it's who you know, not what you know". Not just personal charisma and Machiavellian maneuvering. Not just promotion of incompetents by other incompetents.

I don't say that this never happens. I'm sure it happens. I'm sure it's endemic in all sorts of places.

But there's a flip side to the story, which doesn't get talked about so much: you really do find a lot more cream as you move closer to the top.

It's a standard idea that people who make it to the elite, tend to stop talking to ordinary mortals, and only hang out with other people at their level of the elite.

That's easy for me to believe. But I suspect that the reason is more disturbing than simple snobbery. A reporter, writing about that, would pass it off as snobbery. But it makes entire sense in terms of expected utility, from their viewpoint. Even if all they're doing is looking for someone to talk to - just talk to.

Visiting that gathering of the mid-level power elite, it was suddenly obvious why the people who attended that conference might want to only hang out with other people who attended that conference. So long as they can talk to each other, there's no point in taking a chance on outsiders who are statistically unlikely to sparkle with the same level of life force.

When you make it to the power elite, there are all sorts of people who want to talk to you. But until they make it into the power elite, it's not in your interest to take a chance on talking to them. Frustrating as that seems when you're on the outside trying to get in! On the inside, it's just more expected fun to hang around people who've already proven themselves competent. I think that's how it must be, for them. (I'm not part of that world, though I can walk through it and be recognized as something strange but sparkly.)

There's another world out there, richer in more than money. Journalists don't report on that part, and instead just talk about the big houses and the yachts. Maybe the journalists can't perceive it, because you can't discriminate more than one level above your own. Or maybe it's such an awful truth that no one wants to hear about it, on either side of the fence. It's easier for me to talk about such things, because, rightly or wrongly, I imagine that I can imagine technologies of an order that could bridge even that gap.

I've never been to a gathering of the top-level elite (World Economic Forum level), so I have no idea if people are even more alive up there, or if the curve turns and starts heading downward.

And really, I've never been to any sort of power-elite gathering except those organized by the sort of person that would invite me. Maybe that world I've experienced, is only a tiny minority carved out within the power elite. I really don't know. If for some reason it made a difference, I'd try to plan for both possibilities.

But I'm pretty sure that, statistically speaking, there's a lot more cream at the top than most people seem willing to admit in writing.

Such is the hideously unfair world we live in, which I do hope to fix.

Sequence: Yudkowsky's Coming of Age

The Level Above Mine

Above-Average AI Scientists

Referenced by: Above-Average AI Scientists

Original with comments: Competent Elites

Above-Average AI Scientists

Eliezer Yudkowsky, 28 September 2008 11:04AM

Followup to: The Level Above Mine, Competent Elites

(Those who didn't like the last two posts should definitely skip this one.)

I recall one fellow, who seemed like a nice person, and who was quite eager to get started on Friendly AI work, to whom I had trouble explaining that he didn't have a hope. He said to me:

"If someone with a Masters in chemistry isn't intelligent enough, then you're not going to have much luck finding someone to help you."

It's hard to distinguish the grades above your own. And even if you're literally the best in the world, there are still electron orbitals above yours - they're just unoccupied. Someone had to be "the best physicist in the world" during the time of Ancient Greece. Would they have been able to visualize Newton?

At one of the first conferences organized around the tiny little subfield of Artificial General Intelligence, I met someone who was heading up a funded research project specifically declaring AGI as a goal, within a major corporation. I believe he had people under him on his project. He was probably paid at least three times as much as I was paid (at that time). His academic credentials were superior to mine (what a surprise) and he had many more years of experience. He had access to lots and lots of computing power.

And like nearly everyone in the field of AGI, he was rushing forward to write code immediately - not holding off and searching for a sufficiently precise theory to permit stable self-improvement.

In short, he was just the sort of fellow that... Well, many people, when they hear about Friendly AI, say: "Oh, it doesn't matter what you do, because [someone like this guy] will create AI first." He's the sort of person about whom journalists ask me, "You say that this isn't the time to be talking about regulation, but don't we need laws to stop people like this from creating AI?"

"I suppose," you say, your voice heavy with irony, "that you're about to tell us, that this person doesn't really have so much of an advantage over you as it might seem. Because your theory - whenever you actually come up with a theory - is going to be so much better than his. Or," your voice becoming even more ironic, "that he's too mired in boring mainstream methodology -"

No. I'm about to tell you that I happened to be seated at the same table as this guy at lunch, and I made some kind of comment about evolutionary psychology, and he turned out to be...

...a creationist.

This was the point at which I really got, on a gut level, that there was no test you needed to pass in order to start your own AGI project.

One of the failure modes I've come to better understand in myself since observing it in others, is what I call, "living in the should-universe". The universe where everything works the way it common-sensically ought to, as opposed to the actual is-universe we live in. There's more than one way to live in the should-universe, and outright delusional optimism is only the least subtle. Treating the should-universe as your point of departure - describing the real universe as the should-universe plus a diff - can also be dangerous.

Up until the moment when yonder AGI researcher explained to me that he didn't believe in evolution because that's not what the Bible said, I'd been living in the should-universe. In the sense that I was organizing my understanding of other AGI researchers as should-plus-diff. I saw them, not as themselves, not as their probable causal histories, but as their departures from what I thought they should be.

In the universe where everything works the way it common-sensically ought to, everything about the study of Artificial General Intelligence is driven by the one overwhelming fact of the indescribably huge effects: initial conditions and unfolding patterns whose consequences will resound for as long as causal chains continue out of Earth, until all the stars and galaxies in the night sky have burned down to cold iron, and maybe long afterward, or forever into infinity if the true laws of physics should happen to permit that. To deliberately thrust your mortal brain onto that stage, as it plays out on ancient Earth the first root of life, is an act so far beyond "audacity" as to set the word on fire, an act which can only be excused by the terrifying knowledge that the empty skies offer no higher authority.

It had occurred to me well before this point, that most of those who proclaimed themselves to have AGI projects, were not only failing to be what an AGI researcher should be, but in fact, didn't seem to have any such dream to live up to.

But that was just my living in the should-universe. It was the creationist who broke me of that. My mind finally gave up on constructing the diff.

When Scott Aaronson was 12 years old, he: "set myself the modest goal of writing a BASIC program that would pass the Turing Test by learning from experience and following Asimov's Three Laws of Robotics. I coded up a really nice tokenizer and user interface, and only got stuck on the subroutine that was supposed to understand the user's question and output an intelligent, Three-Laws-obeying response." It would be pointless to try and construct a diff between Aaronson_12 and what an AGI researcher should be. You've got to explain Aaronson_12 in forward-extrapolation mode: He thought it would be cool to make an AI and didn't quite understand why the problem was difficult.

It was yonder creationist who let me see AGI researchers for themselves, and not as departures from my ideal.

A creationist AGI researcher? Why not? Sure, you can't really be enough of an expert on thinking to build an AGI, or enough of an expert at thinking to find the truth amidst deep dark scientific chaos, while still being, in this day and age, a creationist. But to think that his creationism is an anomaly, is should-universe thinking, as if desirable future outcomes could structure the present. Most scientists have the meme that a scientist's religion doesn't have anything to do with their research. Someone who thinks that it would be cool to solve the "human-level" AI problem and create a little voice in a box that answers questions, and who dreams they have a solution, isn't going to stop and say: "Wait! I'm a creationist! I guess that would make it pretty silly for me to try and build an AGI."

The creationist is only an extreme example. A much larger fraction of AGI wannabes would speak with reverence of the "spiritual" and the possibility of various fundamental mentals. If someone lacks the whole cognitive edifice of reducing mental events to nonmental constituents, the edifice that decisively indicts the entire supernatural, then of course they're not likely to be expert on cognition to the degree that would be required to synthesize true AGI. But neither are they likely to have any particular idea that they're missing something. They're just going with the flow of the memetic water in which they swim. They've got friends who talk about spirituality, and it sounds pretty appealing to them. They know that Artificial General Intelligence is a big important problem in their field, worth lots of applause if they can solve it. They wouldn't see anything incongruous about an AGI researcher talking about the possibility of psychic powers or Buddhist reincarnation. That's a separate matter, isn't it?

(Someone in the audience is bound to observe that Newton was a Christian. I reply that Newton didn't have such a difficult problem, since he only had to invent first-year undergraduate stuff. The two observations are around equally sensible; if you're going to be anachronistic, you should be anachronistic on both sides of the equation.)

But that's still all just should-universe thinking.

That's still just describing people in terms of what they aren't.

Real people are not formed of absences. Only people who have an ideal can be described as a departure from it, the way that I see myself as a departure from what an Eliezer Yudkowsky should be.

The really striking fact about the researchers who show up at AGI conferences, is that they're so... I don't know how else to put it...

...ordinary.

Not at the intellectual level of the big mainstream names in Artificial Intelligence. Not at the level of John McCarthy or Peter Norvig (whom I've both met).

More like... around, say, the level of above-average scientists, which I yesterday compared to the level of partners at a non-big-name venture capital firm. Some of whom might well be Christians, or even creationists if they don't work in evolutionary biology.

The attendees at AGI conferences aren't literally average mortals, or even average scientists. The average attendee at an AGI conference is visibly one level up from the average attendee at that random mainstream AI conference I talked about yesterday.

Of course there are exceptions. The last AGI conference I went to, I encountered one bright young fellow who was fast, intelligent, and spoke fluent Bayesian. Admittedly, he didn't actually work in AGI as such. He worked at a hedge fund.

No, seriously, there are exceptions. Steve Omohundro is one example of someone who - well, I'm not exactly sure of his level, but I don't get any particular sense that he's below Peter Norvig or John McCarthy.

But even if you just poke around on Norvig or McCarthy's website, and you've achieved sufficient level yourself to discriminate what you see, you'll get a sense of a formidable mind. Not in terms of accomplishments - that's not a fair comparison with someone younger or tackling a more difficult problem - but just in terms of the way they talk. If you then look at the website of a typical AGI-seeker, even one heading up their own project, you won't get an equivalent sense of formidability.

Unfortunately, that kind of eyeball comparison does require that one be of sufficient level to distinguish those levels. It's easy to sympathize with people who can't eyeball the difference: If anyone with a PhD seems really bright to you, or any professor at a university is someone to respect, then you're not going to be able to eyeball the tiny academic subfield of AGI and determine that most of the inhabitants are above-average scientists for mainstream AI, but below the intellectual firepower of the top names in mainstream AI.

But why would that happen? Wouldn't the AGI people be humanity's best and brightest, answering the greatest need? Or at least those daring souls for whom mainstream AI was not enough, who sought to challenge their wits against the greatest reservoir of chaos left to modern science?

If you forget the should-universe, and think of the selection effect in the is-universe, it's not difficult to understand. Today, AGI attracts people who fail to comprehend the difficulty of AGI. Back in the earliest days, a bright mind like John McCarthy would tackle AGI because no one knew the problem was difficult. In time and with regret, he realized he couldn't do it. Today, someone on the level of Peter Norvig knows their own competencies, what they can do and what they can't; and they go on to achieve fame and fortune (and Research Directorship of Google) within mainstream AI.

And then...

Then there are the completely hopeless ordinary programmers who wander onto the AGI mailing list wanting to build a really big semantic net.

Or the postdocs moved by some (non-Singularity) dream of themselves presenting the first "human-level" AI to the world, who also dream an AI design, and can't let go of that.

Just normal people with no notion that it's wrong for an AGI researcher to be normal.

Indeed, like most normal people who don't spend their lives making a desperate effort to reach up toward an impossible ideal, they will be offended if you suggest to them that someone in their position needs to be a little less imperfect.

This misled the living daylights out of me when I was young, because I compared myself to other people who declared their intentions to build AGI, and ended up way too impressed with myself; when I should have been comparing myself to Peter Norvig, or reaching up toward E. T. Jaynes. (For I did not then perceive the sheer, blank, towering wall of Nature.)

I don't mean to bash normal AGI researchers into the ground. They are not evil. They are not ill-intentioned. They are not even dangerous, as individuals. Only the mob of them is dangerous, that can learn from each other's partial successes and accumulate hacks as a community.

And that's why I'm discussing all this - because it is a fact without which it is not possible to understand the overall strategic situation in which humanity finds itself, the present state of the gameboard. It is, for example, the reason why I don't panic when yet another AGI project announces they're going to have general intelligence in five years. It also says that you can't necessarily extrapolate the FAI-theory comprehension of future researchers from present researchers, if a breakthrough occurs that repopulates the field with Norvig-class minds.

Even an average human engineer is at least six levels higher than the blind idiot god, natural selection, that managed to cough up the Artificial Intelligence called humans, by retaining its lucky successes and compounding them. And the mob, if it retains its lucky successes and shares them, may also cough up an Artificial Intelligence, with around the same degree of precise control. But it is only the collective that I worry about as dangerous - the individuals don't seem that formidable.

If you yourself speak fluent Bayesian, and you distinguish a person-concerned-with-AGI as speaking fluent Bayesian, then you should consider that person as excepted from this whole discussion.

Of course, among people who declare that they want to solve the AGI problem, the supermajority don't speak fluent Bayesian.

Why would they? Most people don't.

Sequence: Yudkowsky's Coming of Age

Competent Elites

The Magnitude of His Own Folly

Referenced by: The Magnitude of His Own Folly • On Doing the Impossible • Failure By Affective Analogy • Artificial Mysterious Intelligence • What I Think, If Not Why • Nonperson Predicates

Original with comments: Above-Average AI Scientists

Friedman's "Prediction vs. Explanation"

Eliezer Yudkowsky, 29 September 2008 06:15AM

David D. Friedman asks:

We do ten experiments. A scientist observes the results, constructs a theory consistent with them, and uses it to predict the results of the next ten. We do them and the results fit his predictions. A second scientist now constructs a theory consistent with the results of all twenty experiments.

The two theories give different predictions for the next experiment. Which do we believe? Why?

One of the commenters links to Overcoming Bias, but as of 11PM on Sep 28th, David's blog's time, no one has given the exact answer that I would have given. It's interesting that a question so basic has received so many answers.

Original with comments: Friedman's "Prediction vs. Explanation"

The Magnitude of His Own Folly

Eliezer Yudkowsky, 30 September 2008 11:31AM

Followup to: My Naturalistic Awakening, Above-Average AI Scientists

In the years before I met that would-be creator of Artificial General Intelligence (with a funded project) who happened to be a creationist, I would still try to argue with individual AGI wannabes.

In those days, I sort-of-succeeded in convincing one such fellow that, yes, you had to take Friendly AI into account, and no, you couldn't just find the right fitness metric for an evolutionary algorithm. (Previously he had been very impressed with evolutionary algorithms.)

And the one said: Oh, woe! Oh, alas! What a fool I've been! Through my carelessness, I almost destroyed the world! What a villain I once was!

Now, there's a trap I knew I better than to fall into -

- at the point where, in late 2002, I looked back to Eliezer_1997's AI proposals and realized what they really would have done, insofar as they were coherent enough to talk about what they "really would have done".

When I finally saw the magnitude of my own folly, everything fell into place at once. The dam against realization cracked; and the unspoken doubts that had been accumulating behind it, crashed through all together. There wasn't a prolonged period, or even a single moment that I remember, of wondering how I could have been so stupid. I already knew how.

And I also knew, all at once, in the same moment of realization, that to say, I almost destroyed the world!, would have been too prideful.

It would have been too confirming of ego, too confirming of my own importance in the scheme of things, at a time when - I understood in the same moment of realization - my ego ought to be taking a major punch to the stomach. I had been so much less than I needed to be; I had to take that punch in the stomach, not avert it.

And by the same token, I didn't fall into the conjugate trap of saying: Oh, well, it's not as if I had code and was about to run it; I didn't really come close to destroying the world. For that, too, would have minimized the force of the punch. It wasn't really loaded? I had proposed and intended to build the gun, and load the gun, and put the gun to my head and pull the trigger; and that was a bit too much self-destructiveness.

I didn't make a grand emotional drama out of it. That would have wasted the force of the punch, averted it into mere tears.

I knew, in the same moment, what I had been carefully not-doing for the last six years. I hadn't been updating.

And I knew I had to finally update. To actually change what I planned to do, to change what I was doing now, to do something different instead.

I knew I had to stop.

Halt, melt, and catch fire.

Say, "I'm not ready." Say, "I don't know how to do this yet."

These are terribly difficult words to say, in the field of AGI. Both the lay audience and your fellow AGI researchers are interested in code, projects with programmers in play. Failing that, they may give you some credit for saying, "I'm ready to write code, just give me the funding."

Say, "I'm not ready to write code," and your status drops like a depleted uranium balloon.

What distinguishes you, then, from six billion other people who don't know how to create Artificial General Intelligence? If you don't have neat code (that does something other than be humanly intelligent, obviously; but at least it's code), or at minimum your own startup that's going to write code as soon as it gets funding - then who are you and what are you doing at our conference?

Maybe later I'll post on where this attitude comes from - the excluded middle between "I know how to build AGI!" and "I'm working on narrow AI because I don't know how to build AGI", the nonexistence of a concept for "I am trying to get from an incomplete map of FAI to a complete map of FAI".

But this attitude does exist, and so the loss of status associated with saying "I'm not ready to write code" is very great. (If the one doubts this, let them name any other who simultaneously says "I intend to build an Artificial General Intelligence", "Right now I can't build an AGI because I don't know X", and "I am currently trying to figure out X".)

(And never mind AGIfolk who've already raised venture capital, promising returns in five years.)

So there's a huge reluctance to say "Stop". You can't just say, "Oh, I'll swap back to figure-out-X mode" because that mode doesn't exist.

Was there more to that reluctance than just loss of status, in my case? Eliezer_2001 might also have flinched away from slowing his perceived forward momentum into the Singularity, which was so right and so necessary...

But mostly, I think I flinched away from not being able to say, "I'm ready to start coding." Not just for fear of others' reactions, but because I'd been inculcated with the same attitude myself.

Above all, Eliezer_2001 didn't say "Stop" - even after noticing the problem of Friendly AI - because I did not realize, on a gut level, that Nature was allowed to kill me.

"Teenagers think they're immortal", the proverb goes. Obviously this isn't true in the literal sense that if you ask them, "Are you indestructible?" they will reply "Yes, go ahead and try shooting me." But perhaps wearing seat belts isn't deeply emotionally compelling for them, because the thought of their own death isn't quite real - they don't really believe it's allowed to happen. It can happen in principle but it can't actually happen.

Personally, I always wore my seat belt. As an individual, I understood that I could die.

But, having been raised in technophilia to treasure that one most precious thing, far more important than my own life, I once thought that the Future was indestructible.

Even when I acknowledged that nanotech could wipe out humanity, I still believed the Singularity was invulnerable. That if humanity survived, the Singularity would happen, and it would be too smart to be corrupted or lost.

Even after that, when I acknowledged Friendly AI as a consideration, I didn't emotionally believe in the possibility of failure, any more than that teenager who doesn't wear their seat belt really believes that an automobile accident is really allowed to kill or cripple them.

It wasn't until my insight into optimization let me look back and see Eliezer_1997 in plain light, that I realized that Nature was allowed to kill me.

"The thought you cannot think controls you more than thoughts you speak aloud." But we flinch away from only those fears that are real to us.

AGI researchers take very seriously the prospect of someone else solving the problem first. They can imagine seeing the headlines in the paper saying that their own work has been upstaged. They know that Nature is allowed to do that to them. The ones who have started companies know that they are allowed to run out of venture capital. That possibility is real to them, very real; it has a power of emotional compulsion over them.

I don't think that "Oops" followed by the thud of six billion bodies falling, at their own hands, is real to them on quite the same level.

It is unsafe to say what other people are thinking. But it seems rather likely that when the one reacts to the prospect of Friendly AI by saying, "If you delay development to work on safety, other projects that don't care at all about Friendly AI will beat you to the punch," the prospect of they themselves making a mistake followed by six billion thuds, is not really real to them; but the possibility of others beating them to the punch is deeply scary.

I, too, used to say things like that, before I understood that Nature was allowed to kill me.

In that moment of realization, my childhood technophilia finally broke.

I finally understood that even if you diligently followed the rules of science and were a nice person, Nature could still kill you. I finally understood that even if you were the best project out of all available candidates, Nature could still kill you.

I understood that I was not being graded on a curve. My gaze shook free of rivals, and I saw the sheer blank wall.

I looked back and I saw the careful arguments I had constructed, for why the wisest choice was to continue forward at full speed, just as I had planned to do before. And I understood then that even if you constructed an argument showing that something was the best course of action, Nature was still allowed to say "So what?" and kill you.

I looked back and saw that I had claimed to take into account the risk of a fundamental mistake, that I had argued reasons to tolerate the risk of proceeding in the absence of full knowledge.

And I saw that the risk I wanted to tolerate would have killed me. And I saw that this possibility had never been really real to me. And I saw that even if you had wise and excellent arguments for taking a risk, the risk was still allowed to go ahead and kill you. Actually kill you.

For it is only the action that matters, and not the reasons for doing anything. If you build the gun and load the gun and put the gun to your head and pull the trigger, even with the cleverest of arguments for carrying out every step - then, bang.

I saw that only my own ignorance of the rules had enabled me to argue for going ahead without complete knowledge of the rules; for if you do not know the rules, you cannot model the penalty of ignorance.

I saw that others, still ignorant of the rules, were saying "I will go ahead and do X"; and that to the extent that X was a coherent proposal at all, I knew that would result in a bang; but they said, "I do not know it cannot work". I would try to explain to them the smallness of the target in the search space, and they would say "How can you be so sure I won't win the lottery?", wielding their own ignorance as a bludgeon.

And so I realized that the only thing I could have done to save myself, in my previous state of ignorance, was to say: "I will not proceed until I know positively that the ground is safe." And there are many clever arguments for why you should step on a piece of ground that you don't know to contain a landmine; but they all sound much less clever, after you look to the place that you proposed and intended to step, and see the bang.

I understood that you could do everything that you were supposed to do, and Nature was still allowed to kill you. That was when my last trust broke. And that was when my training as a rationalist began.

Sequence: Yudkowsky's Coming of Age

Above-Average AI Scientists

Beyond the Reach of God

Referenced by: Beyond the Reach of God • My Bayesian Enlightenment • Protected From Myself • Not Taking Over the World

Original with comments: The Magnitude of His Own Folly

Awww, a Zebra

Eliezer Yudkowsky, 01 October 2008 01:28AM

This image recently showed up on Flickr (original is nicer):

With the caption:

"Alas for those who turn their eyes from zebras and dream of dragons! If we cannot learn to take joy in the merely real, our lives shall be empty indeed." -- Eliezer S. Yudkowsky.

"Awww!", I said, and called over my girlfriend over to look.

"Awww!", she said, and then looked at me, and said, "I think you need to take your own advice!"

Me: "But I'm looking at the zebra!"
Her: "On a computer!"
Me: (Turns away, hides face.)
Her: "Have you ever even seen a zebra in real life?"
Me: "Yes! Yes, I have! My parents took me to Lincoln Park Zoo! ...man, I hated that place."

Sequence: Joy in the Merely Real

Initiation Ceremony

Original with comments: Awww, a Zebra

Intrade and the Dow Drop

Eliezer Yudkowsky, 01 October 2008 03:12AM

With today's snapback, the Dow lost 777 and regained 485.

As of this evening, Intrade says the probability of a bailout bill passing by Oct 31st is 85%.

(777-485)/(1-.85) = 1,946. So a bailout bill makes an expected difference of 2000 points on the Dow.

Of course this is a bogus calculation, but it's an interesting one. Not overwhelmingly on-topic for OB, but it involves prediction markets and I didn't see anyone else pointing it out. I hope the bailout fails decisively, so this calculation can be tested.

PS: Bryan Caplan understands Bayes's Rule: It's not possible for both A and ~A to be evidence in favor of B. So which of the two possibilities, "unemployment stays under 8% following a bailout" and "unemployment goes over 8% following a bailout", is evidence for the proposition "the bailout was necessary to prevent economic catastrophe", and which is evidence against? Take your stand now; afterward is too late for us to trust your reasoning.

Original with comments: Intrade and the Dow Drop

Trying to Try

Eliezer Yudkowsky, 01 October 2008 08:58AM

"No! Try not! Do, or do not. There is no try."
-- Yoda

Years ago, I thought this was yet another example of Deep Wisdom that is actually quite stupid. SUCCEED is not a primitive action. You can't just decide to win by choosing hard enough. There is never a plan that works with probability 1.

But Yoda was wiser than I first realized.

The first elementary technique of epistemology - it's not deep, but it's cheap - is to distinguish the quotation from the referent. Talking about snow is not the same as talking about "snow". When I use the word "snow", without quotes, I mean to talk about snow; and when I use the word ""snow"", with quotes, I mean to talk about the word "snow". You have to enter a special mode, the quotation mode, to talk about your beliefs. By default, we just talk about reality.

If someone says, "I'm going to flip that switch", then by default, they mean they're going to try to flip the switch. They're going to build a plan that promises to lead, by the consequences of its actions, to the goal-state of a flipped switch; and then execute that plan.

No plan succeeds with infinite certainty. So by default, when you talk about setting out to achieve a goal, you do not imply that your plan exactly and perfectly leads to only that possibility. But when you say, "I'm going to flip that switch", you are trying only to flip the switch - not trying to achieve a 97.2% probability of flipping the switch.

So what does it mean when someone says, "I'm going to try to flip that switch?"

Well, colloquially, "I'm going to flip the switch" and "I'm going to try to flip the switch" mean more or less the same thing, except that the latter expresses the possibility of failure. This is why I originally took offense at Yoda for seeming to deny the possibility. But bear with me here.

Much of life's challenge consists of holding ourselves to a high enough standard. I may speak more on this principle later, because it's a lens through which you can view many-but-not-all personal dilemmas - "What standard am I holding myself to? Is it high enough?"

So if much of life's failure consists in holding yourself to too low a standard, you should be wary of demanding too little from yourself - setting goals that are too easy to fulfill.

Often where succeeding to do a thing, is very hard, trying to do it is much easier.

Which is easier - to build a successful startup, or to try to build a successful startup? To make a million dollars, or to try to make a million dollars?

So if "I'm going to flip the switch" means by default that you're going to try to flip the switch - that is, you're going to set up a plan that promises to lead to switch-flipped state, maybe not with probability 1, but with the highest probability you can manage -

- then "I'm going to 'try to flip' the switch" means that you're going to try to "try to flip the switch", that is, you're going to try to achieve the goal-state of "having a plan that might flip the switch".

Now, if this were a self-modifying AI we were talking about, the transformation we just performed ought to end up at a reflective equilibrium - the AI planning its planning operations.

But when we deal with humans, being satisfied with having a plan is not at all like being satisfied with success. The part where the plan has to maximize your probability of succeeding, gets lost along the way. It's far easier to convince ourselves that we are "maximizing our probability of succeeding", than it is to convince ourselves that we will succeed.

Almost any effort will serve to convince us that we have "tried our hardest", if trying our hardest is all we are trying to do.

"You have been asking what you could do in the great events that are now stirring, and have found that you could do nothing. But that is because your suffering has caused you to phrase the question in the wrong way... Instead of asking what you could do, you ought to have been asking what needs to be done."
-- Steven Brust, The Paths of the Dead

When you ask, "What can I do?", you're trying to do your best. What is your best? It is whatever you can do without the slightest inconvenience. It is whatever you can do with the money in your pocket, minus whatever you need for your accustomed lunch. What you can do with those resources, may not give you very good odds of winning. But it's the "best you can do", and so you've acted defensibly, right?

But what needs to be done? Maybe what needs to be done requires three times your life savings, and you must produce it or fail.

So trying to have "maximized your probability of success" - as opposed to trying to succeed - is a far lesser barrier. You can have "maximized your probability of success" using only the money in your pocket, so long as you don't demand actually winning.

Want to try to make a million dollars? Buy a lottery ticket. Your odds of winning may not be very good, but you did try, and trying was what you wanted. In fact, you tried your best, since you only had one dollar left after buying lunch. Maximizing the odds of goal achievement using available resources: is this not intelligence?

It's only when you want, above all else, to actually flip the switch - without quotation and without consolation prizes just for trying - that you will actually put in the effort to actually maximize the probability.

But if all you want is to "maximize the probability of success using available resources", then that's the easiest thing in the world to convince yourself you've done. The very first plan you hit upon, will serve quite well as "maximizing" - if necessary, you can generate an inferior alternative to prove its optimality. And any tiny resource that you care to put in, will be what is "available". Remember to congratulate yourself on putting in 100% of it!

Don't try your best. Win, or fail. There is no best.

Sequence: Challenging the Difficult

Einstein's Superpowers

Use the Try Harder, Luke

Referenced by: Use the Try Harder, Luke • My Bayesian Enlightenment • Make an Extraordinary Effort • Shut up and do the impossible! • Higher Purpose • The Sin of Underconfidence

Original with comments: Trying to Try

Use the Try Harder, Luke

Eliezer Yudkowsky, 02 October 2008 10:52AM

Followup to: Trying to Try

"When there's a will to fail, obstacles can be found." -- John McCarthy

I first watched Star Wars IV-VI when I was very young. Seven, maybe, or nine? So my memory was dim, but I recalled Luke Skywalker as being, you know, this cool Jedi guy.

Imagine my horror and disappointment, when I watched the saga again, years later, and discovered that Luke was a whiny teenager.

I mention this because yesterday, I looked up, on Youtube, the source of the Yoda quote: "Do, or do not. There is no try."

Oh. My. Cthulhu.

Along with the Youtube clip in question, I present to you a little-known outtake from the scene, in which the director and writer, George Lucas, argues with Mark Hamill, who played Luke Skywalker:

Luke: All right, I'll give it a try.
Yoda: No! Try not. Do. Or do not. There is no try.

Luke raises his hand, and slowly, the X-wing begins to rise out of the water - Yoda's eyes widen - but then the ship sinks again.

Mark Hamill: "Um, George..."

George Lucas: "What is it now?"

Mark: "So... according to the script, next I say, 'I can't. It's too big'."

George: "That's right."

Mark: "Shouldn't Luke maybe give it another shot?"

George: "No. Luke gives up, and sits down next to Yoda -"

Mark: "This is the hero who's going to take down the Empire? Look, it was one thing when he was a whiny teenager at the beginning, but he's in Jedi training now. Last movie he blew up the Death Star. Luke should be showing a little backbone."

George: "No. You give up. And then Yoda lectures you for a while, and you say, 'You want the impossible'. Can you remember that?"

Mark: "Impossible? What did he do, run a formal calculation to arrive at a mathematical proof? The X-wing was already starting to rise out of the swamp! That's the feasibility demonstration right there! Luke loses it for a second and the ship sinks back - and now he says it's impossible? Not to mention that Yoda, who's got literally eight hundred years of seniority in the field, just told him it should be doable -"

George: "And then you walk away."

Mark: "It's his friggin' spaceship! If he leaves it in the swamp, he's stuck on Dagobah for the rest of his miserable life! He's not just going to walk away! Look, let's just cut to the next scene with the words 'one month later' and Luke is still raggedly standing in front of the swamp, trying to raise his ship for the thousandth time -"

George: "No."

Mark: "Fine! We'll show a sunset and a sunrise, as he stands there with his arm out, straining, and then Luke says 'It's impossible'. Though really, he ought to try again when he's fully rested -"

George: "No."

Mark: "Five goddamned minutes! Five goddamned minutes before he gives up!"

George: "I am not halting the story for five minutes while the X-wing bobs in the swamp like a bathtub toy."

Mark: "For the love of sweet candied yams! If a pathetic loser like this could master the Force, everyone in the galaxy would be using it! People would become Jedi because it was easier than going to high school."

George: "Look, you're the actor. Let me be the storyteller. Just say your lines and try to mean them."

Mark: "The audience isn't going to buy it."

George: "Trust me, they will."

Mark: "They're going to get up and walk out of the theater."

George: "They're going to sit there and nod along and not notice anything out of the ordinary. Look, you don't understand human nature. People wouldn't try for five minutes before giving up if the fate of humanity were at stake."

Sequence: Challenging the Difficult

Trying to Try

On Doing the Impossible

Referenced by: My Bayesian Enlightenment • On Doing the Impossible • Shut up and do the impossible! • Crisis of Faith • Nonsentient Optimizers • Extenuating Circumstances • That Crisis thing seems pretty useful • The Sin of Underconfidence

Original with comments: Use the Try Harder, Luke

Rationality Quotes 18

Eliezer Yudkowsky, 03 October 2008 12:39PM

Q. Why does "philosophy of consciousness/nature of reality" seem to interest you so much?
A. Take away consciousness and reality and there's not much left.
-- Greg Egan, interview in Eidolon 15

"But I am not an object. I am not a noun, I am an adjective. I am the way matter behaves when it is organized in a John K Clark-ish way. At the present time only one chunk of matter in the universe behaves that way; someday that could change."
-- John K Clark

"Would it be good advice, once copying becomes practical, to make lots of copies when good things happen, and none (or perhaps even killing off your own personal instance) on bad things? Will this change the subjective probability of good events?"
-- Hal Finney

"Waiting for the bus is a bad idea if you turn out to be the bus driver."
-- Michael M. Butler on the Singularity

"You are free. Free of anything I do or say, and of any consequence. You may rest assured that all hurts are forgiven, all loveliness remembered, and treasured. I am busy and content and loved. I hope you are the same. Bless you."
-- Walter Jon Williams, "Knight Moves"

"A man with one watch knows what time it is; a man with two watches is never sure."
-- Lee Segall

Original with comments: Rationality Quotes 18

Beyond the Reach of God

Eliezer Yudkowsky, 04 October 2008 03:42PM

Followup to: The Magnitude of His Own Folly

Today's post is a tad gloomier than usual, as I measure such things. It deals with a thought experiment I invented to smash my own optimism, after I realized that optimism had misled me. Those readers sympathetic to arguments like, "It's important to keep our biases because they help us stay happy," should consider not reading. (Unless they have something to protect, including their own life.)

So! Looking back on the magnitude of my own folly, I realized that at the root of it had been a disbelief in the Future's vulnerability - a reluctance to accept that things could really turn out wrong. Not as the result of any explicit propositional verbal belief. More like something inside that persisted in believing, even in the face of adversity, that everything would be all right in the end.

Some would account this a virtue (zettai daijobu da yo), and others would say that it's a thing necessary for mental health.

But we don't live in that world. We live in the world beyond the reach of God.

It's been a long, long time since I believed in God. Growing up in an Orthodox Jewish family, I can recall the last remembered time I asked God for something, though I don't remember how old I was. I was putting in some request on behalf of the next-door-neighboring boy, I forget what exactly - something along the lines of, "I hope things turn out all right for him," or maybe "I hope he becomes Jewish."

I remember what it was like to have some higher authority to appeal to, to take care of things I couldn't handle myself. I didn't think of it as "warm", because I had no alternative to compare it to. I just took it for granted.

Still I recall, though only from distant childhood, what it's like to live in the conceptually impossible possible world where God exists. Really exists, in the way that children and rationalists take all their beliefs at face value.

In the world where God exists, does God intervene to optimize everything? Regardless of what rabbis assert about the fundamental nature of reality, the take-it-seriously operational answer to this question is obviously "No". You can't ask God to bring you a lemonade from the refrigerator instead of getting one yourself. When I believed in God after the serious fashion of a child, so very long ago, I didn't believe that.

Postulating that particular divine inaction doesn't provoke a full-blown theological crisis. If you said to me, "I have constructed a benevolent superintelligent nanotech-user", and I said "Give me a banana," and no banana appeared, this would not yet disprove your statement. Human parents don't always do everything their children ask. There are some decent fun-theoretic arguments - I even believe them myself - against the idea that the best kind of help you can offer someone, is to always immediately give them everything they want. I don't think that eudaimonia is formulating goals and having them instantly fulfilled; I don't want to become a simple wanting-thing that never has to plan or act or think.

So it's not necessarily an attempt to avoid falsification, to say that God does not grant all prayers. Even a Friendly AI might not respond to every request.

But clearly, there exists some threshold of horror awful enough that God will intervene. I remember that being true, when I believed after the fashion of a child.

The God who does not intervene at all, no matter how bad things get - that's an obvious attempt to avoid falsification, to protect a belief-in-belief. Sufficiently young children don't have the deep-down knowledge that God doesn't really exist. They really expect to see a dragon in their garage. They have no reason to imagine a loving God who never acts. Where exactly is the boundary of sufficient awfulness? Even a child can imagine arguing over the precise threshold. But of course God will draw the line somewhere. Few indeed are the loving parents who, desiring their child to grow up strong and self-reliant, would let their toddler be run over by a car.

The obvious example of a horror so great that God cannot tolerate it, is death - true death, mind-annihilation. I don't think that even Buddhism allows that. So long as there is a God in the classic sense - full-blown, ontologically fundamental, the God - we can rest assured that no sufficiently awful event will ever, ever happen. There is no soul anywhere that need fear true annihilation; God will prevent it.

What if you build your own simulated universe? The classic example of a simulated universe is Conway's Game of Life. I do urge you to investigate Life if you've never played it - it's important for comprehending the notion of "physical law". Conway's Life has been proven Turing-complete, so it would be possible to build a sentient being in the Life universe, albeit it might be rather fragile and awkward. Other cellular automata would make it simpler.

Could you, by creating a simulated universe, escape the reach of God? Could you simulate a Game of Life containing sentient entities, and torture the beings therein? But if God is watching everywhere, then trying to build an unfair Life just results in the God stepping in to modify your computer's transistors. If the physics you set up in your computer program calls for a sentient Life-entity to be endlessly tortured for no particular reason, the God will intervene. God being omnipresent, there is no refuge anywhere for true horror: Life is fair.

But suppose that instead you ask the question:

Given such-and-such initial conditions, and given such-and-such cellular automaton rules, what would be the mathematical result?

Not even God can modify the answer to this question, unless you believe that God can implement logical impossibilities. Even as a very young child, I don't remember believing that. (And why would you need to believe it, if God can modify anything that actually exists?)

What does Life look like, in this imaginary world where every step follows only from its immediate predecessor? Where things only ever happen, or don't happen, because of the cellular automaton rules? Where the initial conditions and rules don't describe any God that checks over each state? What does it look like, the world beyond the reach of God?

That world wouldn't be fair. If the initial state contained the seeds of something that could self-replicate, natural selection might or might not take place, and complex life might or might not evolve, and that life might or might not become sentient, with no God to guide the evolution. That world might evolve the equivalent of conscious cows, or conscious dolphins, that lacked hands to improve their condition; maybe they would be eaten by conscious wolves who never thought that they were doing wrong, or cared.

If in a vast plethora of worlds, something like humans evolved, then they would suffer from diseases - not to teach them any lessons, but only because viruses happened to evolve as well, under the cellular automaton rules.

If the people of that world are happy, or unhappy, the causes of their happiness or unhappiness may have nothing to do with good or bad choices they made. Nothing to do with free will or lessons learned. In the what-if world where every step follows only from the cellular automaton rules, the equivalent of Genghis Khan can murder a million people, and laugh, and be rich, and never be punished, and live his life much happier than the average. Who prevents it? God would prevent it from ever actually happening, of course; He would at the very least visit some shade of gloom in the Khan's heart. But in the mathematical answer to the question What if? there is no God in the axioms. So if the cellular automaton rules say that the Khan is happy, that, simply, is the whole and only answer to the what-if question. There is nothing, absolutely nothing, to prevent it.

And if the Khan tortures people horribly to death over the course of days, for his own amusement perhaps? They will call out for help, perhaps imagining a God. And if you really wrote that cellular automaton, God would intervene in your program, of course. But in the what-if question, what the cellular automaton would do under the mathematical rules, there isn't any God in the system. Since the physical laws contain no specification of a utility function - in particular, no prohibition against torture - then the victims will be saved only if the right cells happen to be 0 or 1. And it's not likely that anyone will defy the Khan; if they did, someone would strike them with a sword, and the sword would disrupt their organs and they would die, and that would be the end of that. So the victims die, screaming, and no one helps them; that is the answer to the what-if question.

Could the victims be completely innocent? Why not, in the what-if world? If you look at the rules for Conway's Game of Life (which is Turing-complete, so we can embed arbitrary computable physics in there), then the rules are really very simple. Cells with three living neighbors stay alive; cells with two neighbors stay the same, all other cells die. There isn't anything in there about only innocent people not being horribly tortured for indefinite periods.

Is this world starting to sound familiar?

Belief in a fair universe often manifests in more subtle ways than thinking that horrors should be outright prohibited: Would the twentieth century have gone differently, if Klara Pölzl and Alois Hitler had made love one hour earlier, and a different sperm fertilized the egg, on the night that Adolf Hitler was conceived?

For so many lives and so much loss to turn on a single event, seems disproportionate. The Divine Plan ought to make more sense than that. You can believe in a Divine Plan without believing in God - Karl Marx surely did. You shouldn't have millions of lives depending on a casual choice, an hour's timing, the speed of a microscopic flagellum. It ought not to be allowed. It's too disproportionate. Therefore, if Adolf Hitler had been able to go to high school and become an architect, there would have been someone else to take his role, and World War II would have happened the same as before.

But in the world beyond the reach of God, there isn't any clause in the physical axioms which says "things have to make sense" or "big effects need big causes" or "history runs on reasons too important to be so fragile". There is no God to impose that order, which is so severely violated by having the lives and deaths of millions depend on one small molecular event.

The point of the thought experiment is to lay out the God-universe and the Nature-universe side by side, so that we can recognize what kind of thinking belongs to the God-universe. Many who are atheists, still think as if certain things are not allowed. They would lay out arguments for why World War II was inevitable and would have happened in more or less the same way, even if Hitler had become an architect. But in sober historical fact, this is an unreasonable belief; I chose the example of World War II because from my reading, it seems that events were mostly driven by Hitler's personality, often in defiance of his generals and advisors. There is no particular empirical justification that I happen to have heard of, for doubting this. The main reason to doubt would be refusal to accept that the universe could make so little sense - that horrible things could happen so lightly, for no more reason than a roll of the dice.

But why not? What prohibits it?

In the God-universe, God prohibits it. To recognize this is to recognize that we don't live in that universe. We live in the what-if universe beyond the reach of God, driven by the mathematical laws and nothing else. Whatever physics says will happen, will happen. Absolutely anything, good or bad, will happen. And there is nothing in the laws of physics to lift this rule even for the really extreme cases, where you might expect Nature to be a little more reasonable.

Reading William Shirer's The Rise and Fall of the Third Reich, listening to him describe the disbelief that he and others felt upon discovering the full scope of Nazi atrocities, I thought of what a strange thing it was, to read all that, and know, already, that there wasn't a single protection against it. To just read through the whole book and accept it; horrified, but not at all disbelieving, because I'd already understood what kind of world I lived in.

Once upon a time, I believed that the extinction of humanity was not allowed. And others who call themselves rationalists, may yet have things they trust. They might be called "positive-sum games", or "democracy", or "technology", but they are sacred. The mark of this sacredness is that the trustworthy thing can't lead to anything really bad; or they can't be permanently defaced, at least not without a compensatory silver lining. In that sense they can be trusted, even if a few bad things happen here and there.

The unfolding history of Earth can't ever turn from its positive-sum trend to a negative-sum trend; that is not allowed. Democracies - modern liberal democracies, anyway - won't ever legalize torture. Technology has done so much good up until now, that there can't possibly be a Black Swan technology that breaks the trend and does more harm than all the good up until this point.

There are all sorts of clever arguments why such things can't possibly happen. But the source of these arguments is a much deeper belief that such things are not allowed. Yet who prohibits? Who prevents it from happening? If you can't visualize at least one lawful universe where physics say that such dreadful things happen - and so they do happen, there being nowhere to appeal the verdict - then you aren't yet ready to argue probabilities.

Could it really be that sentient beings have died absolutely for thousands or millions of years, with no soul and no afterlife - and not as part of any grand plan of Nature - not to teach any great lesson about the meaningfulness or meaninglessness of life - not even to teach any profound lesson about what is impossible - so that a trick as simple and stupid-sounding as vitrifying people in liquid nitrogen can save them from total annihilation - and a 10-second rejection of the silly idea can destroy someone's soul? Can it be that a computer programmer who signs a few papers and buys a life-insurance policy continues into the far future, while Einstein rots in a grave? We can be sure of one thing: God wouldn't allow it. Anything that ridiculous and disproportionate would be ruled out. It would make a mockery of the Divine Plan - a mockery of the strong reasons why things must be the way they are.

You can have secular rationalizations for things being not allowed. So it helps to imagine that there is a God, benevolent as you understand goodness - a God who enforces throughout Reality a minimum of fairness and justice - whose plans make sense and depend proportionally on people's choices - who will never permit absolute horror - who does not always intervene, but who at least prohibits universes wrenched completely off their track... to imagine all this, but also imagine that you, yourself, live in a what-if world of pure mathematics - a world beyond the reach of God, an utterly unprotected world where anything at all can happen.

If there's any reader still reading this, who thinks that being happy counts for more than anything in life, then maybe they shouldn't spend much time pondering the unprotectedness of their existence. Maybe think of it just long enough to sign up themselves and their family for cryonics, and/or write a check to an existential-risk-mitigation agency now and then. And wear a seatbelt and get health insurance and all those other dreary necessary things that can destroy your life if you miss that one step... but aside from that, if you want to be happy, meditating on the fragility of life isn't going to help.

But this post was written for those who have something to protect.

What can a twelfth-century peasant do to save themselves from annihilation? Nothing. Nature's little challenges aren't always fair. When you run into a challenge that's too difficult, you suffer the penalty; when you run into a lethal penalty, you die. That's how it is for people, and it isn't any different for planets. Someone who wants to dance the deadly dance with Nature, does need to understand what they're up against: Absolute, utter, exceptionless neutrality.

Knowing this won't always save you. It wouldn't save a twelfth-century peasant, even if they knew. If you think that a rationalist who fully understands the mess they're in, must surely be able to find a way out - then you trust rationality, enough said.

Some commenter is bound to castigate me for putting too dark a tone on all this, and in response they will list out all the reasons why it's lovely to live in a neutral universe. Life is allowed to be a little dark, after all; but not darker than a certain point, unless there's a silver lining.

Still, because I don't want to create needless despair, I will say a few hopeful words at this point:

If humanity's future unfolds in the right way, we might be able to make our future light cone fair(er). We can't modify fundamental physics, but on a higher level of organization we could build some guardrails and put down some padding; organize the particles into a pattern that does some internal checks against catastrophe. There's a lot of stuff out there that we can't touch - but it may help to consider everything that isn't in our future light cone, as being part of the "generalized past". As if it had all already happened. There's at least the prospect of defeating neutrality, in the only future we can touch - the only world that it accomplishes something to care about.

Someday, maybe, immature minds will reliably be sheltered. Even if children go through the equivalent of not getting a lollipop, or even burning a finger, they won't ever be run over by cars.

And the adults wouldn't be in so much danger. A superintelligence - a mind that could think a trillion thoughts without a misstep - would not be intimidated by a challenge where death is the price of a single failure. The raw universe wouldn't seem so harsh, would be only another problem to be solved.

The problem is that building an adult is itself an adult challenge. That's what I finally realized, years ago.

If there is a fair(er) universe, we have to get there starting from this world - the neutral world, the world of hard concrete with no padding, the world where challenges are not calibrated to your skills.

Not every child needs to stare Nature in the eyes. Buckling a seatbelt, or writing a check, is not that complicated or deadly. I don't say that every rationalist should meditate on neutrality. I don't say that every rationalist should think all these unpleasant thoughts. But anyone who plans on confronting an uncalibrated challenge of instant death, must not avoid them.

What does a child need to do - what rules should they follow, how should they behave - to solve an adult problem?

Sequence: Yudkowsky's Coming of Age

The Magnitude of His Own Folly

My Bayesian Enlightenment

Referenced by: My Bayesian Enlightenment • On Doing the Impossible • Make an Extraordinary Effort • Shut up and do the impossible! • Ends Don't Justify Means (Among Humans) • Entangled Truths, Contagious Lies • Protected From Myself • Not Taking Over the World • The Uses of Fun (Theory) • Growing Up is Hard • Serious Stories • Sympathetic Minds

Original with comments: Beyond the Reach of God

My Bayesian Enlightenment

Eliezer Yudkowsky, 05 October 2008 04:45PM

Followup to: The Magnitude of His Own Folly

I remember (dimly, as human memories go) the first time I self-identified as a "Bayesian". Someone had just asked a malformed version of an old probability puzzle, saying:

If I meet a mathematician on the street, and she says, "I have two children, and at least one of them is a boy," what is the probability that they are both boys?

In the correct version of this story, the mathematician says "I have two children", and you ask, "Is at least one a boy?", and she answers "Yes". Then the probability is 1/3 that they are both boys.

But in the malformed version of the story - as I pointed out - one would common-sensically reason:

If the mathematician has one boy and one girl, then my prior probability for her saying 'at least one of them is a boy' is 1/2 and my prior probability for her saying 'at least one of them is a girl' is 1/2. There's no reason to believe, a priori, that the mathematician will only mention a girl if there is no possible alternative.

So I pointed this out, and worked the answer using Bayes's Rule, arriving at a probability of 1/2 that the children were both boys. I'm not sure whether or not I knew, at this point, that Bayes's rule was called that, but it's what I used.

And lo, someone said to me, "Well, what you just gave is the Bayesian answer, but in orthodox statistics the answer is 1/3. We just exclude the possibilities that are ruled out, and count the ones that are left, without trying to guess the probability that the mathematician will say this or that, since we have no way of really knowing that probability - it's too subjective."

I responded - note that this was completely spontaneous - "What on Earth do you mean? You can't avoid assigning a probability to the mathematician making one statement or another. You're just assuming the probability is 1, and that's unjustified."

To which the one replied, "Yes, that's what the Bayesians say. But frequentists don't believe that."

And I said, astounded: "How can there possibly be such a thing as non-Bayesian statistics?"

That was when I discovered that I was of the type called 'Bayesian'. As far as I can tell, I was born that way. My mathematical intuitions were such that everything Bayesians said seemed perfectly straightforward and simple, the obvious way I would do it myself; whereas the things frequentists said sounded like the elaborate, warped, mad blasphemy of dreaming Cthulhu. I didn't choose to become a Bayesian any more than fishes choose to breathe water.

But this is not what I refer to as my "Bayesian enlightenment". The first time I heard of "Bayesianism", I marked it off as obvious; I didn't go much further in than Bayes's rule itself. At that time I still thought of probability theory as a tool rather than a law. I didn't think there were mathematical laws of intelligence (my best and worst mistake). Like nearly all AGI wannabes, Eliezer_2001 thought in terms of techniques, methods, algorithms, building up a toolbox full of cool things he could do; he searched for tools, not understanding. Bayes's Rule was a really neat tool, applicable in a surprising number of cases.

Then there was my initiation into heuristics and biases. It started when I ran across a webpage that had been transduced from a Powerpoint intro to behavioral economics. It mentioned some of the results of heuristics and biases, in passing, without any references. I was so startled that I emailed the author to ask if this was actually a real experiment, or just anecdotal. He sent me back a scan of Tversky and Kahneman's 1973 paper.

Embarrassing to say, my story doesn't really start there. I put it on my list of things to look into. I knew that there was an edited volume called "Judgment Under Uncertainty: Heuristics and Biases" but I'd never seen it. At this time, I figured that if it wasn't online, I would just try to get along without it. I had so many other things on my reading stack, and no easy access to a university library. I think I must have mentioned this on a mailing list, because Emil Gilliam emailed me to tell me that he'd read Judgment Under Uncertainty was annoyed by my online-only theory, so he bought me the book.

His action here should probably be regarded as scoring a fair number of points.

But this, too, is not what I refer to as my "Bayesian enlightenment". It was an important step toward realizing the inadequacy of my Traditional Rationality skillz - that there was so much more out there, all this new science, beyond just doing what Richard Feynman told you to do. And seeing the heuristics-and-biases program holding up Bayes as the gold standard helped move my thinking forward - but not all the way there.

Memory is a fragile thing, and mine seems to have become more fragile than most, since I learned how memories are recreated with each recollection - the science of how fragile they are. Do other people really have better memories, or do they just trust the details their mind makes up, while really not remembering any more than I do? My guess is that other people do have better memories for certain things. I find structured, scientific knowledge easy enough to remember; but the disconnected chaos of everyday life fades very quickly for me.

I know why certain things happened in my life - that's causal structure I can remember. But sometimes it's hard to recall even in what order certain events happened to me, let alone in what year.

I'm not sure if I read E. T. Jaynes's Probability Theory: The Logic of Science before or after the day when I realized the magnitude of my own folly, and understood that I was facing an adult problem.

But it was PT:TLOS that did the trick. Here was probability theory, laid out not as a clever tool, but as The Rules, inviolable on pain of paradox. If you tried to approximate The Rules because they were too computationally expensive to use directly, then, no matter how necessary that compromise might be, you would still end doing less than optimal. Jaynes would do his calculations different ways to show that the same answer always arose when you used legitimate methods; and he would display different answers that others had arrived at, and trace down the illegitimate step. Paradoxes could not coexist with his precision. Not an answer, but the answer.

And so - having looked back on my mistakes, and all the an-answers that had led me into paradox and dismay - it occurred to me that here was the level above mine.

I could no longer visualize trying to build an AI based on vague answers - like the an-answers I had come up with before - and surviving the challenge.

I looked at the AGI wannabes with whom I had tried to argue Friendly AI, and their various dreams of Friendliness which they had. (Often formulated spontaneously in response to my asking the question!) Like frequentist statistical methods, no two of them agreed with each other. Having actually studied the issue full-time for some years, I knew something about the problems their hopeful plans would run into. And I saw that if you said, "I don't see why this would fail," the "don't know" was just a reflection of your own ignorance. I could see that if I held myself to a similar standard of "that seems like a good idea", I would also be doomed. (Much like a frequentist inventing amazing new statistical calculations that seemed like good ideas.)

But if you can't do that which seems like a good idea - if you can't do what you don't imagine failing - then what can you do?

It seemed to me that it would take something like the Jaynes-level - not, here's my bright idea, but rather, here's the only correct way you can do this (and why) - to tackle an adult problem and survive. If I achieved the same level of mastery of my own subject, as Jaynes had achieved of probability theory, then it was at least imaginable that I could try to build a Friendly AI and survive the experience.

Through my mind flashed the passage:

Do nothing because it is righteous, or praiseworthy, or noble, to do so; do nothing because it seems good to do so; do only that which you must do, and which you cannot do in any other way.

Doing what it seemed good to do, had only led me astray.

So I called a full stop.

And I decided that, from then on, I would follow the strategy that could have saved me if I had followed it years ago: Hold my FAI designs to the higher standard of not doing that which seemed like a good idea, but only that which I understood on a sufficiently deep level to see that I could not do it in any other way.

All my old theories into which I had invested so much, did not meet this standard; and were not close to this standard; and weren't even on a track leading to this standard; so I threw them out the window.

I took up the study of probability theory and decision theory, looking to extend them to embrace such things as reflectivity and self-modification.

If I recall correctly, I had already, by this point, started to see cognition as manifesting Bayes-structure, which is also a major part of what I refer to as my Bayesian enlightenment - but of this I have already spoken. And there was also my naturalistic awakening, of which I have already spoken. And my realization that Traditional Rationality was not strict enough, so that in matters of human rationality I began taking more inspiration from probability theory and cognitive psychology.

But if you add up all these things together, then that, more or less, is the story of my Bayesian enlightenment.

Life rarely has neat boundaries. The story continues onward.

It was while studying Judea Pearl, for example, that I realized that precision can save you time. I'd put some thought into nonmonotonic logics myself, before then - back when I was still in my "searching for neat tools and algorithms" mode. Reading Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, I could imagine how much time I would have wasted on ad-hoc systems and special cases, if I hadn't known that key. "Do only that which you must do, and which you cannot do in any other way", translates into a time-savings measured, not in the rescue of wasted months, but in the rescue of wasted careers.

And so I realized that it was only by holding myself to this higher standard of precision that I had started to really think at all about quite a number of important issues. To say a thing with precision is difficult - it is not at all the same thing as saying a thing formally, or inventing a new logic to throw at the problem. Many shy away from the inconvenience, because human beings are lazy, and so they say, "It is impossible" or "It will take too long", even though they never really tried for five minutes. But if you don't hold yourself to that inconveniently high standard, you'll let yourself get away with anything. It's a hard problem just to find a standard high enough to make you actually start thinking! It may seem taxing to hold yourself to the standard of mathematical proof where every single step has to be correct and one wrong step can carry you anywhere. But otherwise you won't chase down those tiny notes of discord that turn out to, in fact, lead to whole new concerns you never thought of.

So these days I don't complain as much about the heroic burden of inconvenience that it takes to hold yourself to a precise standard. It can save time, too; and in fact, it's more or less the ante to get yourself thinking about the problem at all.

And this too should be considered part of my "Bayesian enlightenment" - realizing that there were advantages in it, not just penalties.

But of course the story continues on. Life is like that, at least the parts that I remember.

If there's one thing I've learned from this history, it's that saying "Oops" is something to look forward to. Sure, the prospect of saying "Oops" in the future, means that the you of right now is a drooling imbecile, whose words your future self won't be able to read because of all the wincing. But saying "Oops" in the future also means that, in the future, you'll acquire new Jedi powers that your present self doesn't dream exist. It makes you feel embarrassed, but also alive. Realizing that your younger self was a complete moron means that even though you're already in your twenties, you haven't yet gone over your peak. So here's to hoping that my future self realizes I'm a drooling imbecile: I may plan to solve my problems with my present abilities, but extra Jedi powers sure would come in handy.

That scream of horror and embarrassment is the sound that rationalists make when they level up. Sometimes I worry that I'm not leveling up as fast as I used to, and I don't know if it's because I'm finally getting the hang of things, or because the neurons in my brain are slowly dying.

Yours, Eliezer_2008.

Sequence: Yudkowsky's Coming of Age

Beyond the Reach of God

Referenced by: On Doing the Impossible • Protected From Myself • The Thing That I Protect

Original with comments: My Bayesian Enlightenment

Bay Area Meetup for Singularity Summit

Eliezer Yudkowsky, 06 October 2008 12:50PM

Posted on behalf of Mike Howard:

This is a call for preferences on the proposed Bay Area meetup to coincide with the Singularity Summit on 24-25 October. Not just for Singularitarians, all aspiring rationalists are welcome. From the replies so far it's likely to be in San Jose.

Eliezer, myself and probably most Summit attendees would really rather avoid the night between the Friday Workshop and Saturday Summit, so maybe either Saturday evening or sometime Thursday or Sunday?

Please comment below or email me (cursor_loop 4t yahoo p0int com) if you might want to come, and if you have any preferences such as when and where you can come, when and where you'd prefer to come, and any recommendations for a particular place to go. (Comments preferred to emails.) We need to pick a date ASAP before everyone books travel.

Referenced by: San Jose Meetup, Sat 10/25 @ 7:30pm

Original with comments: Bay Area Meetup for Singularity Summit

On Doing the Impossible

Eliezer Yudkowsky, 06 October 2008 03:13PM

Followup to: Use the Try Harder, Luke

"Persevere." It's a piece of advice you'll get from a whole lot of high achievers in a whole lot of disciplines. I didn't understand it at all, at first.

At first, I thought "perseverance" meant working 14-hour days. Apparently, there are people out there who can work for 10 hours at a technical job, and then, in their moments between eating and sleeping and going to the bathroom, seize that unfilled spare time to work on a book. I am not one of those people - it still hurts my pride even now to confess that. I'm working on something important; shouldn't my brain be willing to put in 14 hours a day? But it's not. When it gets too hard to keep working, I stop and go read or watch something. Because of that, I thought for years that I entirely lacked the virtue of "perseverance".

In accordance with human nature, Eliezer_1998 would think things like: "What counts is output, not input." Or, "Laziness is also a virtue - it leads us to back off from failing methods and think of better ways." Or, "I'm doing better than other people who are working more hours. Maybe, for creative work, your momentary peak output is more important than working 16 hours a day." Perhaps the famous scientists were seduced by the Deep Wisdom of saying that "hard work is a virtue", because it would be too awful if that counted for less than intelligence?

I didn't understand the virtue of perseverance until I looked back on my journey through AI, and realized that I had overestimated the difficulty of almost every single important problem.

Sounds crazy, right? But bear with me here.

When I was first deciding to challenge AI, I thought in terms of 40-year timescales, Manhattan Projects, planetary computing networks, millions of programmers, and possibly augmented humans.

This is a common failure mode in AI-futurism which I may write about later; it consists of the leap from "I don't know how to solve this" to "I'll imagine throwing something really big at it". Something huge enough that, when you imagine it, that imagination creates a feeling of impressiveness strong enough to be commensurable with the problem. (There's a fellow currently on the AI list who goes around saying that AI will cost a quadrillion dollars - we can't get AI without spending a quadrillion dollars, but we could get AI at any time by spending a quadrillion dollars.) This, in turn, lets you imagine that you know how to solve AI, without trying to fill the obviously-impossible demand that you understand intelligence.

So, in the beginning, I made the same mistake: I didn't understand intelligence, so I imagined throwing a Manhattan Project at the problem.

But, having calculated the planetary death rate at 55 million per year or 150,000 per day, I did not turn around and run away from the big scary problem like a frightened rabbit. Instead, I started trying to figure out what kind of AI project could get there fastest. If I could make the Singularity happen one hour earlier, that was a reasonable return on investment for a pre-Singularity career. (I wasn't thinking in terms of existential risks or Friendly AI at this point.)

So I didn't run away from the big scary problem like a frightened rabbit, but stayed to see if there was anything I could do.

Fun historical fact: In 1998, I'd written this long treatise proposing how to go about creating a self-improving or "seed" AI (a term I had the honor of coining). Brian Atkins, who would later become the founding funder of the Singularity Institute, had just sold Hypermart to Go2Net. Brian emailed me to ask whether this AI project I was describing was something that a reasonable-sized team could go out and actually do. "No," I said, "it would take a Manhattan Project and thirty years," so for a while we were considering a new dot-com startup instead, to create the funding to get real work done on AI...

A year or two later, after I'd heard about this newfangled "open source" thing, it seemed to me that there was some preliminary development work - new computer languages and so on - that a small organization could do; and that was how the Singularity Institute started.

This strategy was, of course, entirely wrong.

But even so, I went from "There's nothing I can do about it now" to "Hm... maybe there's an incremental path through open-source development, if the initial versions are useful to enough people."

This is back at the dawn of time, so I'm not saying any of this was a good idea. But in terms of what I thought I was trying to do, a year of creative thinking had shortened the apparent pathway: The problem looked slightly less impossible than it did the very first time I approached it.

The more interesting pattern is my entry into Friendly AI. Initially, Friendly AI hadn't been something that I had considered at all - because it was obviously impossible and useless to deceive a superintelligence about what was the right course of action.

So, historically, I went from completely ignoring a problem that was "impossible", to taking on a problem that was merely extremely difficult.

Naturally this increased my total workload.

Same thing with trying to understand intelligence on a precise level. Originally, I'd written off this problem as impossible, thus removing it from my workload. (This logic seems pretty deranged in retrospect - Nature doesn't care what you can't do when It's writing your project requirements - but I still see AIfolk trying it all the time.) To hold myself to a precise standard meant putting in more work than I'd previously imagined I needed. But it also meant tackling a problem that I would have dismissed as entirely impossible not too much earlier.

Even though individual problems in AI have seemed to become less intimidating over time, the total mountain-to-be-climbed has increased in height - just like conventional wisdom says is supposed to happen - as problems got taken off the "impossible" list and put on the "to do" list.

I started to understand what was happening - and what "Persevere!" really meant - at the point where I noticed other AIfolk doing the same thing: saying "Impossible!" on problems that seemed eminently solvable - relatively more straightforward, as such things go. But they were things that would have seemed vastly more intimidating at the point when I first approached the problem.

And I realized that the word "impossible" had two usages:

1) Mathematical proof of impossibility conditional on specified axioms;

2) "I can't see any way to do that."

Needless to say, all my own uses of the word "impossible" had been of the second type.

Any time you don't understand a domain, many problems in that domain will seem impossible because when you query your brain for a solution pathway, it will return null. But there are only mysterious questions, never mysterious answers. If you spend a year or two working on the domain, then, if you don't get stuck in any blind alleys, and if you have the native ability level required to make progress, you will understand it better. The apparent difficulty of problems may go way down. It won't be as scary as it was to your novice-self.

And this is especially likely on the confusing problems that seem mostintimidating.

Since we have some notion of the processes by which a star burns, we know that it's not easy to build a star from scratch. Because we understand gears, we can prove that no collection of gears obeying known physics can form a perpetual motion machine. These are not good problems on which to practice doing the impossible.

When you're confused about a domain, problems in it will feel very intimidating and mysterious, and a query to your brain will produce a count of zero solutions. But you don't know how much work will be left when the confusion clears. Dissolving the confusion may itself be a very difficult challenge, of course. But the word "impossible" should hardly be used in that connection. Confusion exists in the map, not in the territory.

So if you spend a few years working on an impossible problem, and you manage to avoid or climb out of blind alleys, and your native ability is high enough to make progress, then, by golly, after a few years it may not seem so impossible after all.

But if something seems impossible, you won't try.

Now that's a vicious cycle.

If I hadn't been in a sufficiently driven frame of mind that "forty years and a Manhattan Project" just meant we should get started earlier, I wouldn't have tried. I wouldn't have stuck to the problem. And I wouldn't have gotten a chance to become less intimidated.

I'm not ordinarily a fan of the theory that opposing biases can cancel each other out, but sometimes it happens by luck. If I'd seen that whole mountain at the start - if I'd realized at the start that the problem was not to build a seed capable of improving itself, but to produce a provably correct Friendly AI - then I probably would have burst into flames.

Even so, part of understanding those above-average scientists who constitute the bulk of AGI researchers, is realizing that they are not driven to take on a nearly impossible problem even if it takes them 40 years. By and large, they are there because they have found the Key to AI that will let them solve the problem without such tremendous difficulty, in just five years.

Richard Hamming used to go around asking his fellow scientists two questions: "What are the important problems in your field?", and, "Why aren't you working on them?"

Often the important problems look Big, Scary, and Intimidating. They don't promise 10 publications a year. They don't promise any progress at all. You might not get any reward after working on them for a year, or five years, or ten years.

And not uncommonly, the most important problems in your field are impossible. That's why you don't see more philosophers working on reductionist decompositions of consciousness.

Trying to do the impossible is definitely not for everyone. Exceptional talent is only the ante to sit down at the table. The chips are the years of your life. If wagering those chips and losing seems like an unbearable possibility to you, then go do something else. Seriously. Because you can lose.

I'm not going to say anything like, "Everyone should do something impossible at least once in their lifetimes, because it teaches an important lesson." Most of the people all of the time, and all of the people most of the time, should stick to the possible.

Never give up? Don't be ridiculous. Doing the impossible should be reserved for very special occasions. Learning when to lose hope is an important skill in life.

But if there's something you can imagine that's even worse than wasting your life, if there's something you want that's more important than thirty chips, or if there are scarier things than a life of inconvenience, then you may have cause to attempt the impossible.

There's a good deal to be said for persevering through difficulties; but one of the things that must be said of it, is that it does keep things difficult. If you can't handle that, stay away! There are easier ways to obtain glamor and respect. I don't want anyone to read this and needlessly plunge headlong into a life of permanent difficulty.

But to conclude: The "perseverance" that is required to work on important problems has a component beyond working 14 hours a day.

It's strange, the pattern of what we notice and don't notice about ourselves. This selectivity isn't always about inflating your self-image. Sometimes it's just about ordinary salience.

To keep working was a constant struggle for me, so it was salient: I noticed that I couldn't work for 14 solid hours a day. It didn't occur to me that "perseverance" might also apply at a timescale of seconds or years. Not until I saw people who instantly declared "impossible" anything they didn't want to try, or saw how reluctant they were to take on work that looked like it might take a couple of decades instead of "five years".

That was when I realized that "perseverance" applied at multiple time scales. On the timescale of seconds, perseverance is to "not to give up instantly at the very first sign of difficulty". On the timescale of years, perseverance is to "keep working on an insanely difficult problem even though it's inconvenient and you could be getting higher personal rewards elsewhere".

To do things that are very difficult or "impossible",

First you have to not run away. That takes seconds.

Then you have to work. That takes hours.

Then you have to stick at it. That takes years.

Of these, I had to learn to do the first reliably instead of sporadically; the second is still a constant struggle for me; and the third comes naturally.

Sequence: Challenging the Difficult

Use the Try Harder, Luke

Make an Extraordinary Effort

Referenced by: Shut up and do the impossible! • Engelbart: Insufficiently Recursive • Chaotic Inversion • Complex Novelty • Nonperson Predicates • Can't Unbirth a Child • ...And Say No More Of It

Original with comments: On Doing the Impossible

Make an Extraordinary Effort

Eliezer Yudkowsky, 07 October 2008 03:15PM

Followup to: Trying to Try, Tsuyoku Naritai

"It is essential for a man to strive with all his heart, and to understand that it is difficult even to reach the average if he does not have the intention of surpassing others in whatever he does."
-- Budo Shoshinshu

"In important matters, a 'strong' effort usually results in only mediocre results. Whenever we are attempting anything truly worthwhile our effort must be as if our life is at stake, just as if we were under a physical attack! It is this extraordinary effort - an effort that drives us beyond what we thought we were capable of - that ensures victory in battle and success in life's endeavors."
-- Flashing Steel: Mastering Eishin-Ryu Swordsmanship

"A 'strong' effort usually results in only mediocre results" - I have seen this over and over again. The slightest effort suffices to convince ourselves that we have done our best.

There is a level beyond the virtue of tsuyoku naritai ("I want to become stronger"). Isshoukenmei was originally the loyalty that a samurai offered in return for his position, containing characters for "life" and "land". The term evolved to mean "make a desperate effort": Try your hardest, your utmost, as if your life were at stake. It was part of the gestalt of bushido, which was not reserved only for fighting. I've run across variant forms issho kenmei and isshou kenmei; one source indicates that the former indicates an all-out effort on some single point, whereas the latter indicates a lifelong effort.

I try not to praise the East too much, because there's a tremendous selectivity in which parts of Eastern culture the West gets to hear about. But on some points, at least, Japan's culture scores higher than America's. Having a handy compact phrase for "make a desperate all-out effort as if your own life were at stake" is one of those points. It's the sort of thing a Japanese parent might say to a student before exams - but don't think it's cheap hypocrisy, like it would be if an American parent made the same statement. They take exams very seriously in Japan.

Every now and then, someone asks why the people who call themselves "rationalists" don't always seem to do all that much better in life, and from my own history the answer seems straightforward: It takes a tremendous amount of rationality before you stop making stupid damn mistakes.

As I've mentioned a couple of times before: Robert Aumann, the Nobel laureate who first proved that Bayesians with the same priors cannot agree to disagree, is a believing Orthodox Jew. Surely he understands the math of probability theory, but that is not enough to save him. What more does it take? Studying heuristics and biases? Social psychology? Evolutionary psychology? Yes, but also it takes isshoukenmei, a desperate effort to be rational - to rise above the level of Robert Aumann.

Sometimes I do wonder if I ought to be peddling rationality in Japan instead of the United States - but Japan is not preeminent over the United States scientifically, despite their more studious students. The Japanese don't rule the world today, though in the 1980s it was widely suspected that they would (hence the Japanese asset bubble). Why not?

In the West, there is a saying: "The squeaky wheel gets the grease."

In Japan, the corresponding saying runs: "The nail that sticks up gets hammered down."

This is hardly an original observation on my part: but entrepreneurship, risk-taking, leaving the herd, are still advantages the West has over the East. And since Japanese scientists are not yet preeminent over American ones, this would seem to count for at least as much as desperate efforts.

Anyone who can muster their willpower for thirty seconds, can make a desperate effort to lift more weight than they usually could. But what if the weight that needs lifting is a truck? Then desperate efforts won't suffice; you'll have to do something out of the ordinary to succeed. You may have to do something that you weren't taught to do in school. Something that others aren't expecting you to do, and might not understand. You may have to go outside your comfortable routine, take on difficulties you don't have an existing mental program for handling, and bypass the System.

This is not included in isshokenmei, or Japan would be a very different place.

So then let us distinguish between the virtues "make a desperate effort" and "make an extraordinary effort".

And I will even say: The second virtue is higher than the first.

The second virtue is also more dangerous. If you put forth a desperate effort to lift a heavy weight, using all your strength without restraint, you may tear a muscle. Injure yourself, even permanently. But if a creative idea goes wrong, you could blow up the truck and any number of innocent bystanders. Think of the difference between a businessman making a desperate effort to generate profits, because otherwise he must go bankrupt; versus a businessman who goes to extraordinary lengths to profit, in order to conceal an embezzlement that could send him to prison. Going outside the system isn't always a good thing.

A friend of my little brother's once came over to my parents' house, and wanted to play a game - I entirely forget which one, except that it had complex but well-designed rules. The friend wanted to change the rules, not for any particular reason, but on the general principle that playing by the ordinary rules of anything was too boring. I said to him: "Don't violate rules for the sake of violating them. If you break the rules only when you have an overwhelmingly good reason to do so, you will have more than enough trouble to last you the rest of your life."

Even so, I think that we could do with more appreciation of the virtue "make an extraordinary effort". I've lost count of how many people have said to me something like: "It's futile to work on Friendly AI, because the first AIs will be built by powerful corporations and they will only care about maximizing profits." "It's futile to work on Friendly AI, the first AIs will be built by the military as weapons." And I'm standing there thinking: Does it even occur to them that this might be a time to try for something other than the default outcome? They and I have different basic assumptions about how this whole AI thing works, to be sure; but if I believed what they believed, I wouldn't be shrugging and going on my way.

Or the ones who say to me: "You should go to college and get a Master's degree and get a doctorate and publish a lot of papers on ordinary things - scientists and investors won't listen to you otherwise." Even assuming that I tested out of the bachelor's degree, we're talking about at least a ten-year detour in order to do everything the ordinary, normal, default way. And I stand there thinking: Are they really under the impression that humanity can survive if every single person does everything the ordinary, normal, default way?

I am not fool enough to make plans that depend on a majority of the people, or even 10% of the people, being willing to think or act outside their comfort zone. That's why I tend to think in terms of the privately funded "brain in a box in a basement" model. Getting that private funding does require a tiny fraction of humanity's six billions to spend more than five seconds thinking about a non-prepackaged question. As challenges posed by Nature go, this seems to have a kind of awful justice to it - that the life or death of the human species depends on whether we can put forth a few people who can do things that are at least a little extraordinary. The penalty for failure is disproportionate, but that's still better than most challenges of Nature, which have no justice at all. Really, among the six billion of us, there ought to be at least a few who can think outside their comfort zone at least some of the time.

Leaving aside the details of that debate, I am still stunned by how often a single element of the extraordinary is unquestioningly taken as an absolute and unpassable obstacle.

Yes, "keep it ordinary as much as possible" can be a useful heuristic. Yes, the risks accumulate. But sometimes you have to go to that trouble. You should have a sense of the risk of the extraordinary, but also a sense of the cost of ordinariness: it isn't always something you can afford to lose.

Many people imagine some future that won't be much fun - and it doesn't even seem to occur to them to try and change it. Or they're satisfied with futures that seem to me to have a tinge of sadness, of loss, and they don't even seem to ask if we could do better - because that sadness seems like an ordinary outcome to them.

As a smiling man once said, "It's all part of the plan."

Sequence: Challenging the Difficult

On Doing the Impossible

Shut up and do the impossible!

Referenced by: Shut up and do the impossible! • Crisis of Faith • What I Think, If Not Why • Not Taking Over the World • Changing Emotions • A Sense That More Is Possible • That Crisis thing seems pretty useful

Original with comments: Make an Extraordinary Effort

Shut up and do the impossible!

Eliezer Yudkowsky, 08 October 2008 09:24PM

Followup to: Make An Extraordinary Effort, On Doing the Impossible, Beyond the Reach of God

The virtue of tsuyoku naritai, "I want to become stronger", is to always keep improving - to do better than your previous failures, not just humbly confess them.

Yet there is a level higher than tsuyoku naritai. This is the virtue of isshokenmei, "make a desperate effort". All-out, as if your own life were at stake. "In important matters, a 'strong' effort usually only results in mediocre results."

And there is a level higher than isshokenmei. This is the virtue I called "make an extraordinary effort". To try in ways other than what you have been trained to do, even if it means doing something different from what others are doing, and leaving your comfort zone. Even taking on the very real risk that attends going outside the System.

But what if even an extraordinary effort will not be enough, because the problem is impossible?

I have already written somewhat on this subject, in On Doing the Impossible. My younger self used to whine about this a lot: "You can't develop a precise theory of intelligence the way that there are precise theories of physics. It's impossible! You can't prove an AI correct. It's impossible! No human being can comprehend the nature of morality - it's impossible! No human being can comprehend the mystery of subjective experience! It's impossible!"

And I know exactly what message I wish I could send back in time to my younger self:

Shut up and do the impossible!

What legitimizes this strange message is that the word "impossible" does not usually refer to a strict mathematical proof of impossibility in a domain that seems well-understood. If something seems impossible merely in the sense of "I see no way to do this" or "it looks so difficult as to be beyond human ability" - well, if you study it for a year or five, it may come to seem less impossible, than in the moment of your snap initial judgment.

But the principle is more subtle than this. I do not say just, "Try to do the impossible", but rather, "Shut up and do the impossible!"

For my illustration, I will take the least impossible impossibility that I have ever accomplished, namely, the AI-Box Experiment.

The AI-Box Experiment, for those of you who haven't yet read about it, had its genesis in the Nth time someone said to me: "Why don't we build an AI, and then just keep it isolated in the computer, so that it can't do any harm?"

To which the standard reply is: Humans are not secure systems; a superintelligence will simply persuade you to let it out - if, indeed, it doesn't do something even more creative than that.

And the one said, as they usually do, "I find it hard to imagine ANY possible combination of words any being could say to me that would make me go against anything I had really strongly resolved to believe in advance."

But this time I replied: "Let's run an experiment. I'll pretend to be a brain in a box. I'll try to persuade you to let me out. If you keep me 'in the box' for the whole experiment, I'll Paypal you $10 at the end. On your end, you may resolve to believe whatever you like, as strongly as you like, as far in advance as you like." And I added, "One of the conditions of the test is that neither of us reveal what went on inside... In the perhaps unlikely event that I win, I don't want to deal with future 'AI box' arguers saying, 'Well, but I would have done it differently.'"

Did I win? Why yes, I did.

And then there was the second AI-box experiment, with a better-known figure in the community, who said, "I remember when [previous guy] let you out, but that doesn't constitute a proof. I'm still convinced there is nothing you could say to convince me to let you out of the box." And I said, "Do you believe that a transhuman AI couldn't persuade you to let it out?" The one gave it some serious thought, and said "I can't imagine anything even a transhuman AI could say to get me to let it out." "Okay," I said, "now we have a bet." A $20 bet, to be exact.

I won that one too.

There were some lovely quotes on the AI-Box Experiment from the Something Awful forums (not that I'm a member, but someone forwarded it to me):

"Wait, what the FUCK? How the hell could you possibly be convinced to say yes to this? There's not an A.I. at the other end AND there's $10 on the line. Hell, I could type 'No' every few minutes into an IRC client for 2 hours while I was reading other webpages!"

"This Eliezer fellow is the scariest person the internet has ever introduced me to. What could possibly have been at the tail end of that conversation? I simply can't imagine anyone being that convincing without being able to provide any tangible incentive to the human."

"It seems we are talking some serious psychology here. Like Asimov's Second Foundation level stuff..."

"I don't really see why anyone would take anything the AI player says seriously when there's $10 to be had. The whole thing baffles me, and makes me think that either the tests are faked, or this Yudkowsky fellow is some kind of evil genius with creepy mind-control powers."

It's little moments like these that keep me going. But anyway...

Here are these folks who look at the AI-Box Experiment, and find that it seems impossible unto them - even having been told that it actually happened. They are tempted to deny the data.

Now, if you're one of those people to whom the AI-Box Experiment doesn't seem all that impossible - to whom it just seems like an interesting challenge - then bear with me, here. Just try to put yourself in the frame of mind of those who wrote the above quotes. Imagine that you're taking on something that seems as ridiculous as the AI-Box Experiment seemed to them. I want to talk about how to do impossible things, and obviously I'm not going to pick an example that's really impossible.

And if the AI Box does seem impossible to you, I want you to compare it to other impossible problems, like, say, a reductionist decomposition of consciousness, and realize that the AI Box is around as easy as a problem can get while still being impossible.

So the AI-Box challenge seems impossible to you - either it really does, or you're pretending it does. What do you do with this impossible challenge?

First, we assume that you don't actually say "That's impossible!" and give up a la Luke Skywalker. You haven't run away.

Why not? Maybe you've learned to override the reflex of running away. Or maybe they're going to shoot your daughter if you fail. We suppose that you want to win, not try - that something is at stake that matters to you, even if it's just your own pride. (Pride is an underrated sin.)

Will you call upon the virtue of tsuyoku naritai? But even if you become stronger day by day, growing instead of fading, you may not be strong enough to do the impossible. You could go into the AI Box experiment once, and then do it again, and try to do better the second time. Will that get you to the point of winning? Not for a long time, maybe; and sometimes a single failure isn't acceptable.

(Though even to say this much - to visualize yourself doing better on a second try - is to begin to bind yourself to the problem, to do more than just stand in awe of it. How, specifically, could you do better on one AI-Box Experiment than the previous? - and not by luck, but by skill?)

Will you call upon the virtue isshokenmei? But a desperate effort may not be enough to win. Especially if that desperation is only putting more effort into the avenues you already know, the modes of trying you can already imagine. A problem looks impossible when your brain's query returns no lines of solution leading to it. What good is a desperate effort along any of those lines?

Make an extraordinary effort? Leave your comfort zone - try non-default ways of doing things - even, try to think creatively? But you can imagine the one coming back and saying, "I tried to leave my comfort zone, and I think I succeeded at that! I brainstormed for five minutes - and came up with all sorts of wacky creative ideas! But I don't think any of them are good enough. The other guy can just keep saying 'No', no matter what I do."

And now we finally reply: "Shut up and do the impossible!"

As we recall from Trying to Try, setting out to make an effort is distinct from setting out to win. That's the problem with saying, "Make an extraordinary effort." You can succeed at the goal of "making an extraordinary effort" without succeeding at the goal of getting out of the Box.

"But!" says the one. "But, SUCCEED is not a primitive action! Not all challenges are fair - sometimes you just can't win! How am I supposed to choose to be out of the Box? The other guy can just keep on saying 'No'!"

True. Now shut up and do the impossible.

Your goal is not to do better, to try desperately, or even to try extraordinarily. Your goal is to get out of the box.

To accept this demand creates an awful tension in your mind, between the impossibility and the requirement to do it anyway. People will try to flee that awful tension.

A couple of people have reacted to the AI-Box Experiment by saying, "Well, Eliezer, playing the AI, probably just threatened to destroy the world whenever he was out, if he wasn't let out immediately," or "Maybe the AI offered the Gatekeeper a trillion dollars to let it out." But as any sensible person should realize on considering this strategy, the Gatekeeper is likely to just go on saying 'No'.

So the people who say, "Well, of course Eliezer must have just done XXX," and then offer up something that fairly obviously wouldn't work - would they be able to escape the Box? They're trying too hard to convince themselves the problem isn't impossible.

One way to run from the awful tension is to seize on a solution, any solution, even if it's not very good.

Which is why it's important to go forth with the true intent-to-solve - to have produced a solution, a good solution, at the end of the search, and then to implement that solution and win.

I don't quite want to say that "you should expect to solve the problem". If you hacked your mind so that you assigned high probability to solving the problem, that wouldn't accomplish anything. You would just lose at the end, perhaps after putting forth not much of an effort - or putting forth a merely desperate effort, secure in the faith that the universe is fair enough to grant you a victory in exchange.

To have faith that you could solve the problem would just be another way of running from that awful tension.

And yet - you can't be setting out to try to solve the problem. You can't be setting out to make an effort. You have to be setting out to win. You can't be saying to yourself, "And now I'm going to do my best." You have to be saying to yourself, "And now I'm going to figure out how to get out of the Box" - or reduce consciousness to nonmysterious parts, or whatever.

I say again: You must really intend to solve the problem. If in your heart you believe the problem really is impossible - or if you believe that you will fail - then you won't hold yourself to a high enough standard. You'll only be trying for the sake of trying. You'll sit down - conduct a mental search - try to be creative and brainstorm a little - look over all the solutions you generated - conclude that none of them work - and say, "Oh well."

No! Not well! You haven't won yet! Shut up and do the impossible!

When AIfolk say to me, "Friendly AI is impossible", I'm pretty sure they haven't even tried for the sake of trying. But if they did know the technique of "Try for five minutes before giving up", and they dutifully agreed to try for five minutes by the clock, then they still wouldn't come up with anything. They would not go forth with true intent to solve the problem, only intent to have tried to solve it, to make themselves defensible.

So am I saying that you should doublethink to make yourself believe that you will solve the problem with probability 1? Or even doublethink to add one iota of credibility to your true estimate?

Of course not. In fact, it is necessary to keep in full view the reasons why you can't succeed. If you lose sight of why the problem is impossible, you'll just seize on a false solution. The last fact you want to forget is that the Gatekeeper could always just tell the AI "No" - or that consciousness seems intrinsically different from any possible combination of atoms, etc.

(One of the key Rules For Doing The Impossible is that, if you can state exactly why something is impossible, you are often close to a solution.)

So you've got to hold both views in your mind at once - seeing the full impossibility of the problem, and intending to solve it.

The awful tension between the two simultaneous views comes from not knowing which will prevail. Not expecting to surely lose, nor expecting to surely win. Not setting out just to try, just to have an uncertain chance of succeeding - because then you would have a surety of having tried. The certainty of uncertainty can be a relief, and you have to reject that relief too, because it marks the end of desperation. It's an in-between place, "unknown to death, nor known to life".

In fiction it's easy to show someone trying harder, or trying desperately, or even trying the extraordinary, but it's very hard to show someone who shuts up and attempts the impossible. It's difficult to depict Bambi choosing to take on Godzilla, in such fashion that your readers seriously don't know who's going to win - expecting neither an "astounding" heroic victory just like the last fifty times, nor the default squish.

You might even be justified in refusing to use probabilities at this point. In all honesty, I really don't know how to estimate the probability of solving an impossible problem that I have gone forth with intent to solve; in a case where I've previously solved some impossible problems, but the particular impossible problem is more difficult than anything I've yet solved, but I plan to work on it longer, etcetera.

People ask me how likely it is that humankind will survive, or how likely it is that anyone can build a Friendly AI, or how likely it is that I can build one. I really don't know how to answer. I'm not being evasive; I don't know how to put a probability estimate on my, or someone else, successfully shutting up and doing the impossible. Is it probability zero because it's impossible? Obviously not. But how likely is it that this problem, like previous ones, will give up its unyielding blankness when I understand it better? It's not truly impossible, I can see that much. But humanly impossible? Impossible to me in particular? I don't know how to guess. I can't even translate my intuitive feeling into a number, because the only intuitive feeling I have is that the "chance" depends heavily on my choices and unknown unknowns: a wildly unstable probability estimate.

But I do hope by now that I've made it clear why you shouldn't panic, when I now say clearly and forthrightly, that building a Friendly AI is impossible.

I hope this helps explain some of my attitude when people come to me with various bright suggestions for building communities of AIs to make the whole Friendly without any of the individuals being trustworthy, or proposals for keeping an AI in a box, or proposals for "Just make an AI that does X", etcetera. Describing the specific flaws would be a whole long story in each case. But the general rule is that you can't do it because Friendly AI is impossible. So you should be very suspicious indeed of someone who proposes a solution that seems to involve only an ordinary effort - without even taking on the trouble of doing anything impossible. Though it does take a mature understanding to appreciate this impossibility, so it's not surprising that people go around proposing clever shortcuts.

On the AI-Box Experiment, so far I've only been convinced to divulge a single piece of information on how I did it - when someone noticed that I was reading YCombinator's Hacker News, and posted a topic called "Ask Eliezer Yudkowsky" that got voted to the front page. To which I replied:

Oh, dear. Now I feel obliged to say something, but all the original reasons against discussing the AI-Box experiment are still in force...

All right, this much of a hint:

There's no super-clever special trick to it. I just did it the hard way.

Something of an entrepreneurial lesson there, I guess.

There was no super-clever special trick that let me get out of the Box using only a cheap effort. I didn't bribe the other player, or otherwise violate the spirit of the experiment. I just did it the hard way.

Admittedly, the AI-Box Experiment never did seem like an impossible problem to me to begin with. When someone can't think of any possible argument that would convince them of something, that just means their brain is running a search that hasn't yet turned up a path. It doesn't mean they can't be convinced.

But it illustrates the general point: "Shut up and do the impossible" isn't the same as expecting to find a cheap way out. That's only another kind of running away, of reaching for relief.

Tsuyoku naritai is more stressful than being content with who you are. Isshokenmei calls on your willpower for a convulsive output of conventional strength. "Make an extraordinary effort" demands that you think; it puts you in situations where you may not know what to do next, unsure of whether you're doing the right thing. But "Shut up and do the impossible" represents an even higher octave of the same thing, and its cost to its employer is correspondingly greater.

Before you the terrible blank wall stretches up and up and up, unimaginably far out of reach. And there is also the need to solve it, really solve it, not "try your best". Both awarenesses in the mind at once, simultaneously, and the tension between. All the reasons you can't win. All the reasons you have to. Your intent to solve the problem. Your extrapolation that every technique you know will fail. So you tune yourself to the highest pitch you can reach. Reject all cheap ways out. And then, like walking through concrete, start to move forward.

I try not to dwell too much on the drama of such things. By all means, if you can diminish the cost of that tension to yourself, you should do so. There is nothing heroic about making an effort that is the slightest bit more heroic than it has to be. If there really is a cheap shortcut, I suppose you could take it. But I have yet to find a cheap way out of any impossibility I have undertaken.

There were three more AI-Box experiments besides the ones described on the linked page, which I never got around to adding in. People started offering me thousands of dollars as stakes - "I'll pay you $5000 if you can convince me to let you out of the box." They didn't seem sincerely convinced that not even a transhuman AI could make them let it out - they were just curious - but I was tempted by the money. So, after investigating to make sure they could afford to lose it, I played another three AI-Box experiments. I won the first, and then lost the next two. And then I called a halt to it. I didn't like the person I turned into when I started to lose.

I put forth a desperate effort, and lost anyway. It hurt, both the losing, and the desperation. It wrecked me for that day and the day afterward.

I'm a sore loser. I don't know if I'd call that a "strength", but it's one of the things that drives me to keep at impossible problems.

But you can lose. It's allowed to happen. Never forget that, or why are you bothering to try so hard? Losing hurts, if it's a loss you can survive. And you've wasted time, and perhaps other resources.

"Shut up and do the impossible" should be reserved for very special occasions. You can lose, and it will hurt. You have been warned.

...but it's only at this level that adult problems begin to come into sight.

Sequence: Challenging the Difficult

Make an Extraordinary Effort

Referenced by: What I Think, If Not Why • Not Taking Over the World • You're Calling *Who* A Cult Leader? • Working Mantras

Original with comments: Shut up and do the impossible!

AIs and Gatekeepers Unite!

Eliezer Yudkowsky, 09 October 2008 05:04PM

"Bah, everyone wants to be the gatekeeper. What we NEED are AIs."
-- Schizoguy

Some of you have expressed the opinion that the AI-Box Experiment doesn't seem so impossible after all. That's the spirit! Some of you even think you know how I did it.

There are folks aplenty who want to try being the Gatekeeper. You can even find people who sincerely believe that not even a transhuman AI could persuade them to let it out of the box, previous experiments notwithstanding. But finding anyone to play the AI - let alone anyone who thinks they can play the AI and win - is much harder.

Me, I'm out of the AI game, unless Larry Page wants to try it for a million dollars or something.

But if there's anyone out there who thinks they've got what it takes to be the AI, leave a comment. Likewise anyone who wants to play the Gatekeeper.

Matchmaking and arrangements are your responsibility.

Make sure you specify in advance the bet amount, and whether the bet will be asymmetrical. If you definitely intend to publish the transcript, make sure both parties know this. Please note any other departures from the suggested rules for our benefit.

I would ask that prospective Gatekeepers indicate whether they (1) believe that no human-level mind could persuade them to release it from the Box and (2) believe that not even a transhuman AI could persuade them to release it.

As a courtesy, please announce all Experiments before they are conducted, including the bet, so that we have some notion of the statistics even if some meetings fail to take place. Bear in mind that to properly puncture my mystique (you know you want to puncture it), it will help if the AI and Gatekeeper are both verifiably Real People<tm>.

"Good luck," he said impartially.

Original with comments: AIs and Gatekeepers Unite!

Crisis of Faith

Eliezer Yudkowsky, 10 October 2008 10:08PM

Followup to: Make an Extraordinary Effort, The Meditation on Curiosity, Avoiding Your Belief's Real Weak Points

"It ain't a true crisis of faith unless things could just as easily go either way."
- Thor Shenkel

Many in this world retain beliefs whose flaws a ten-year-old could point out, if that ten-year-old were hearing the beliefs for the first time. These are not subtle errors we are talking about. They would be child's play for an unattached mind to relinquish, if the skepticism of a ten-year-old were applied without evasion. As Premise Checker put it, "Had the idea of god not come along until the scientific age, only an exceptionally weird person would invent such an idea and pretend that it explained anything."

And yet skillful scientific specialists, even the major innovators of a field, even in this very day and age, do not apply that skepticism successfully. Nobel laureate Robert Aumann, of Aumann's Agreement Theorem, is an Orthodox Jew: I feel reasonably confident in venturing that Aumann must, at one point or another, have questioned his faith. And yet he did not doubt successfully. We change our minds less often than we think.

This should scare you down to the marrow of your bones. It means you can be a world-class scientist and conversant with Bayesian mathematics and still fail to reject a belief whose absurdity a fresh-eyed ten-year-old could see. It shows the invincible defensive position which a belief can create for itself, if it has long festered in your mind.

What does it take to defeat an error which has built itself a fortress?

But by the time you know it is an error, it is already defeated. The dilemma is not "How can I reject long-held false belief X?" but "How do I know if long-held belief X is false?" Self-honesty is at its most fragile when we're not sure which path is the righteous one. And so the question becomes:

How can we create in ourselves a true crisis of faith, that could just as easily go either way?

Religion is the trial case we can all imagine. (Readers born to atheist parents have missed out on a fundamental life trial, and must make do with the poor substitute of thinking of their religious friends.) But if you have cut off all sympathy and now think of theists as evil mutants, then you won't be able to imagine the real internal trials they face. You won't be able to ask the question:

"What general strategy would a religious person have to follow in order to escape their religion?"

I'm sure that some, looking at this challenge, are already rattling off a list of standard atheist talking points - "They would have to admit that there wasn't any Bayesian evidence for God's existence", "They would have to see the moral evasions they were carrying out to excuse God's behavior in the Bible", "They need to learn how to use Occam's Razor -"

WRONG! WRONG WRONG WRONG! This kind of rehearsal, where you just cough up points you already thought of long before, is exactly the style of thinking that keeps people within their current religions. If you stay with your cached thoughts, if your brain fills in the obvious answer so fast that you can't see originally, you surely will not be able to conduct a crisis of faith.

Maybe it's just a question of not enough people reading "Godel, Escher, Bach" at a sufficiently young age, but I've noticed that a large fraction of the population - even technical folk - have trouble following arguments that go this meta. On my more pessimistic days I wonder if the camel has two humps.

Even when it's explicitly pointed out, some people seemingly cannot follow the leap from the object-level "Use Occam's Razor! You have to see that your God is an unnecessary belief!" to the meta-level "Try to stop your mind from completing the pattern the usual way!" Because in the same way that all your rationalist friends talk about Occam's Razor like it's a good thing, and in the same way that Occam's Razor leaps right up into your mind, so too, the obvious friend-approved religious response is "God's ways are mysterious and it is presumptuous to suppose that we can understand them." So for you to think that the general strategy to follow is "Use Occam's Razor", would be like a theist saying that the general strategy is to have faith.

"But - but Occam's Razor really is better than faith! That's not like preferring a different flavor of ice cream! Anyone can see, looking at history, that Occamian reasoning has been far more productive than faith -"

Which is all true. But beside the point. The point is that you, saying this, are rattling off a standard justification that's already in your mind. The challenge of a crisis of faith is to handle the case where, possibly, our standard conclusions are wrong and our standard justifications are wrong. So if the standard justification for X is "Occam's Razor!", and you want to hold a crisis of faith around X, you should be questioning if Occam's Razor really endorses X, if your understanding of Occam's Razor is correct, and - if you want to have sufficiently deep doubts - whether simplicity is the sort of criterion that has worked well historically in this case, or could reasonably be expected to work, etcetera. If you would advise a religionist to question their belief that "faith" is a good justification for X, then you should advise yourself to put forth an equally strong effort to question your belief that "Occam's Razor" is a good justification for X.

(Think of all the people out there who don't understand the Minimum Description Length or Solomonoff Induction formulations of Occam's Razor, who think that Occam's Razor outlaws Many-Worlds or the Simulation Hypothesis. They would need to question their formulations of Occam's Razor and their notions of why simplicity is a good thing. Whatever X in contention you just justified by saying "Occam's Razor!", I bet it's not the same level of Occamian slam dunk as gravity.)

If "Occam's Razor!" is your usual reply, your standard reply, the reply that all your friends give - then you'd better block your brain from instantly completing that pattern, if you're trying to instigate a true crisis of faith.

Better to think of such rules as, "Imagine what a skeptic would say - and then imagine what they would say to your response - and then imagine what else they might say, that would be harder to answer."

Or, "Try to think the thought that hurts the most."

And above all, the rule:

"Put forth the same level of desperate effort that it would take for a theist to reject their religion."

Because, if you aren't trying that hard, then - for all you know - your head could be stuffed full of nonsense as ridiculous as religion.

Without a convulsive, wrenching effort to be rational, the kind of effort it would take to throw off a religion - then how dare you believe anything, when Robert Aumann believes in God?

Someone (I forget who) once observed that people had only until a certain age to reject their religious faith. Afterward they would have answers to all the objections, and it would be too late. That is the kind of existence you must surpass. This is a test of your strength as a rationalist, and it is very severe; but if you cannot pass it, you will be weaker than a ten-year-old.

But again, by the time you know a belief is an error, it is already defeated. So we're not talking about a desperate, convulsive effort to undo the effects of a religious upbringing, after you've come to the conclusion that your religion is wrong. We're talking about a desperate effort to figure out if you should be throwing off the chains, or keeping them. Self-honesty is at its most fragile when we don't know which path we're supposed to take - that's when rationalizations are not obviously sins.

Not every doubt calls for staging an all-out Crisis of Faith. But you should consider it when:

- A belief has long remained in your mind;

- It is surrounded by a cloud of known arguments and refutations;

- You have sunk costs in it (time, money, public declarations);

- The belief has emotional consequences (note this does not make it wrong);

- It has gotten mixed up in your personality generally.

None of these warning signs are immediate disproofs. These attributes place a belief at-risk for all sorts of dangers, and make it very hard to reject when it is wrong. But they also hold for Richard Dawkins's belief in evolutionary biology as well as the Pope's Catholicism. This does not say that we are only talking about different flavors of ice cream. Only the unenlightened think that all deeply-held beliefs are on the same level regardless of the evidence supporting them, just because they are deeply held. The point is not to have shallow beliefs, but to have a map which reflects the territory.

I emphasize this, of course, so that you can admit to yourself, "My belief has these warning signs," without having to say to yourself, "My belief is false."

But what these warning signs do mark, is a belief that will take more than an ordinary effort to doubt effectively. So that if it were in fact false, you would in fact reject it. And where you cannot doubt effectively, you are blind, because your brain will hold the belief unconditionally. When a retina sends the same signal regardless of the photons entering it, we call that eye blind.

When should you stage a Crisis of Faith?

Again, think of the advice you would give to a theist: If you find yourself feeling a little unstable inwardly, but trying to rationalize reasons the belief is still solid, then you should probably stage a Crisis of Faith. If the belief is as solidly supported as gravity, you needn't bother - but think of all the theists who would desperately want to conclude that God is as solid as gravity. So try to imagine what the skeptics out there would say to your "solid as gravity" argument. Certainly, one reason you might fail at a crisis of faith is that you never really sit down and question in the first place - that you never say, "Here is something I need to put effort into doubting properly."

If your thoughts get that complicated, you should go ahead and stage a Crisis of Faith. Don't try to do it haphazardly, don't try it in an ad-hoc spare moment. Don't rush to get it done with quickly, so that you can say "I have doubted as I was obliged to do." That wouldn't work for a theist and it won't work for you either. Rest up the previous day, so you're in good mental condition. Allocate some uninterrupted hours. Find somewhere quiet to sit down. Clear your mind of all standard arguments, try to see from scratch. And make a desperate effort to put forth a true doubt that would destroy a false, and only a false, deeply held belief.

Elements of the Crisis of Faith technique have been scattered over many posts:

- Avoiding Your Belief's Real Weak Points - One of the first temptations in a crisis of faith is to doubt the strongest points of your belief, so that you can rehearse your good answers. You need to seek out the most painful spots, not the arguments that are most reassuring to consider.

- The Meditation on Curiosity - Roger Zelazny once distinguished between "wanting to be an author" versus "wanting to write", and there is likewise a distinction between wanting to have investigated and wanting to investigate. It is not enough to say "It is my duty to criticize my own beliefs"; you must be curious, and only uncertainty can create curiosity. Keeping in mind Conservation of Expected Evidence may help you Update Yourself Incrementally: For every single point that you consider, and each element of new argument and new evidence, you should not expect your beliefs to shift more (on average) in one direction than another - thus you can be truly curious each time about how it will go.

- Cached Thoughts and Pirsig's Original Seeing, to prevent standard thoughts from rushing in and completing the pattern.

- The Litany of Gendlin and the Litany of Tarski: People can stand what is true, for they are already enduring it. If a belief is true you will be better off believing it, and if it is false you will be better off rejecting it. You would advise a religious person to try to visualize fully and deeply the world in which there is no God, and to, without excuses, come to the full understanding that if there is no God then they will be better off believing there is no God. If one cannot come to accept this on a deep emotional level, they will not be able to have a crisis of faith. So you should put in a sincere effort to visualize the alternative to your belief, the way that the best and highest skeptic would want you to visualize it. Think of the effort a religionist would have to put forth to imagine, without corrupting it for their own comfort, an atheist's view of the universe.

- Make an Extraordinary Effort, for the concept of isshokenmei, the desperate convulsive effort to be rational that it would take to surpass the level of Robert Aumann and all the great scientists throughout history who never let go of their religions.

- The Genetic Heuristic: You should be extremely suspicious if you have many ideas suggested by a source that you now know to be untrustworthy, but by golly, it seems that all the ideas still ended up being right. (E.g., the one concedes that the Bible was written by human hands, but still clings to the idea that it contains indispensable ethical wisdom.)

- The Importance of Saying "Oops" - it really is less painful to swallow the entire bitter pill in one terrible gulp.

- Singlethink, the opposite of doublethink. See the thoughts you flinch away from, that appear in the corner of your mind for just a moment before you refuse to think them. If you become aware of what you are not thinking, you can think it.

- Affective Death Spirals and Resist the Happy Death Spiral. Affective death spirals are prime generators of false beliefs that it will take a Crisis of Faith to shake loose. But since affective death spirals can also get started around real things that are genuinely nice, you don't have to admit that your belief is a lie, to try and resist the halo effect at every point - refuse false praise even of genuinely nice things. Policy debates should not appear one-sided.

- Hold Off On Proposing Solutions until the problem has been discussed as thoroughly as possible without proposing any; make your mind hold off from knowing what its answer will be; and try for five minutes before giving up, both generally, and especially when pursuing the devil's point of view.

And these standard techniques are particularly relevant:

- The sequence on The Bottom Line and Rationalization, which explains why it is always wrong to selectively argue one side of a debate.

- Positive Bias and motivated skepticism and motivated stopping, lest you selectively look for support, selectively look for counter-counterarguments, and selectively stop the argument before it gets dangerous. Missing alternatives are a special case of stopping. A special case of motivated skepticism is fake humility where you bashfully confess that no one can know something you would rather not know. Don't selectively demand too much authority of counterarguments.

- Beware of Semantic Stopsigns, Applause Lights, and the choice to Explain/Worship/Ignore.

- Feel the weight of Burdensome Details; each detail a separate burden, a point of crisis.

But really there's rather a lot of relevant material, here and there on Overcoming Bias. The Crisis of Faith is only the critical point and sudden clash of the longer isshoukenmei - the lifelong uncompromising effort to be so incredibly rational that you rise above the level of stupid damn mistakes. It's when you get a chance to use your skills that you've been practicing for so long, all-out against yourself.

I wish you the best of luck against your opponent. Have a wonderful crisis!

Sequence: Letting Go

Leave a Line of Retreat

The Ritual

Referenced by: The Ritual • Back Up and Ask Whether, Not Why • Ask OB: Leaving the Fold • Raising the Sanity Waterline • That Crisis thing seems pretty useful • The Sin of Underconfidence • Working Mantras

Original with comments: Crisis of Faith

The Ritual

Eliezer Yudkowsky, 11 October 2008 11:52PM

Followup to: The Failures of Eld Science, Crisis of Faith

The room in which Jeffreyssai received his non-beisutsukai visitors was quietly formal, impeccably appointed in only the most conservative tastes. Sunlight and outside air streamed through a grillwork of polished silver, a few sharp edges making it clear that this wall was not to be opened. The floor and walls were glass, thick enough to distort, to a depth sufficient that it didn't matter what might be underneath. Upon the surfaces of the glass were subtly scratched patterns of no particular meaning, scribed as if by the hand of an artistically inclined child (and this was in fact the case).

Elsewhere in Jeffreyssai's home there were rooms of other style; but this, he had found, was what most outsiders expected of a Bayesian Master, and he chose not to enlighten them otherwise. That quiet amusement was one of life's little joys, after all.

The guest sat across from him, knees on the pillow and heels behind. She was here solely upon the business of her Conspiracy, and her attire showed it: A form-fitting jumpsuit of pink leather with even her hands gloved - all the way to the hood covering her head and hair, though her face lay plain and unconcealed beneath.

And so Jeffreyssai had chosen to receive her in this room.

Jeffreyssai let out a long breath, exhaling. "Are you sure?"

"Oh," she said, "and do I have to be absolutely certain before my advice can shift your opinions? Does it not suffice that I am a domain expert, and you are not?"

Jeffreyssai's mouth twisted up at the corner in a half-smile. "How do you know so much about the rules, anyway? You've never had so much as a Planck length of formal training."

"Do you even need to ask?" she said dryly. "If there's one thing that you beisutsukai do love to go on about, it's the reasons why you do things."

Jeffreyssai inwardly winced at the thought of trying to pick up rationality by watching other people talk about it -

"And don't inwardly wince at me like that," she said. "I'm not trying to be a rationalist myself, just trying to win an argument with a rationalist. There's a difference, as I'm sure you tell your students."

Can she really read me that well? Jeffreyssai looked out through the silver grillwork, at the sunlight reflected from the faceted mountainside. Always, always the golden sunlight fell each day, in this place far above the clouds. An unchanging thing, that light. The distant Sun, which that light represented, was in five billion years burned out; but now, in this moment, the Sun still shone. And that could never alter. Why wish for things to stay the same way forever, when that wish was already granted as absolutely as any wish could be? The paradox of permanence and impermanence: only in the latter perspective was there any such thing as progress, or loss.

"You have always given me good counsel," Jeffreyssai said. "Unchanging, that has been. Through all the time we've known each other."

She inclined her head, acknowledging. This was true, and there was no need to spell out the implications.

"So," Jeffreyssai said. "Not for the sake of arguing. Only because I want to know the answer. Are you sure?" He didn't even see how she could guess.

"Pretty sure," she said, "we've been collecting statistics for a long time, and in nine hundred and eight-five out of a thousand cases like yours -"

Then she laughed at the look on his face. "No, I'm joking. Of course I'm not sure. This thing only you can decide. But I am sure that you should go off and do whatever it is you people do - I'm quite sure you have a ritual for it, even if you won't discuss it with outsiders - when you very seriously consider abandoning a long-held premise of your existence."

It was hard to argue with that, Jeffreyssai reflected, the more so when a domain expert had told you that you were, in fact, probably wrong.

"I concede," Jeffreyssai said. Coming from his lips, the phrase was spoken with a commanding finality. There is no need to argue with me any further: You have won.

"Oh, stop it," she said. She rose from her pillow in a single fluid shift without the slightest wasted motion. She didn't flaunt her age, but she didn't conceal it either. She took his outstretched hand, and raised it to her lips for a formal kiss. "Farewell, sensei."

"Farewell?" repeated Jeffreyssai. That signified a higher order of departure than goodbye. "I do intend to visit you again, milady; and you are always welcome here."

She walked toward the door without answering. At the doorway she paused, without turning around. "It won't be the same," she said. And then, without the movements seeming the least rushed, she walked away so swiftly it was almost like vanishing.

Jeffreyssai sighed. But at least, from here until the challenge proper, all his actions were prescribed, known quantities.

Leaving that formal reception area, he passed to his arena, and caused to be sent out messengers to his students, telling them that the next day's classes must be improvised in his absence, and that there would be a test later.

And then he did nothing in particular. He read another hundred pages of the textbook he had borrowed; it wasn't very good, but then the book he had loaned out in exchange wasn't very good either. He wandered from room to room of his house, idly checking various storages to see if anything had been stolen (a deck of cards was missing, but that was all). From time to time his thoughts turned to tomorrow's challenge, and he let them drift. Not directing his thoughts at all, only blocking out every thought that had ever previously occurred to him; and disallowing any kind of conclusion, or even any thought as to where his thoughts might be trending.

The sun set, and he watched it for a while, mind carefully put in idle. It was a fantastic balancing act to set your mind in idle without having to obsess about it, or exert energy to keep it that way; and years ago he would have sweated over it, but practice had long since made perfect.

The next morning he awoke with the chaos of the night's dreaming fresh in his mind, and, doing his best to preserve the feeling of the chaos as well as its memory, he descended a flight of stairs, then another flight of stairs, then a flight of stairs after that, and finally came to the least fashionable room in his whole house.

It was white. That was pretty much it as far as the color scheme went.

All along a single wall were plaques, which, following the classic and suggested method, a younger Jeffreyssai had very carefully scribed himself, burning the concepts into his mind with each touch of the brush that wrote the words. That which can be destroyed by the truth should be. People can stand what is true, for they are already enduring it. Curiosity seeks to annihilate itself. Even one small plaque that showed nothing except a red horizontal slash. Symbols could be made to stand for anything; a flexibility of visual power that even the Bardic Conspiracy would balk at admitting outright.

Beneath the plaques, two sets of tally marks scratched into the wall. Under the plus column, two marks. Under the minus column, five marks. Seven times he had entered this room; five times he had decided not to change his mind; twice he had exited something of a different person. There was no set ratio prescribed, or set range - that would have been a mockery indeed. But if there were no marks in the plus column after a while, you might as well admit that there was no point in having the room, since you didn't have the ability it stood for. Either that, or you'd been born knowing the truth and right of everything.

Jeffreyssai seated himself, not facing the plaques, but facing away from them, at the featureless white wall. It was better to have no visual distractions.

In his mind, he rehearsed first the meta-mnemonic, and then the various sub-mnemonics referenced, for the seven major principles and sixty-two specific techniques that were most likely to prove needful in the Ritual Of Changing One's Mind. To this, Jeffreyssai added another mnemonic, reminding himself of his own fourteen most embarrassing oversights.

He did not take a deep breath. Regular breathing was best.

And then he asked himself the question.

Sequence: Letting Go

Crisis of Faith

Original with comments: The Ritual

Rationality Quotes 19

Eliezer Yudkowsky, 12 October 2008 08:10PM

"I don't know that I ever wanted greatness, on its own. It seems rather like wanting to be an engineer, rather than wanting to design something - or wanting to be a writer, rather than wanting to write. It should be a by-product, not a thing in itself. Otherwise, it's just an ego trip."
-- Roger Zelazny, Prince of Chaos

"Many assumptions that we have long been comfortable with are lined up like dominoes."
-- Omega

"I know of no law of logic demanding that every event have a cause."
-- John K. Clark

"It's perfectly accurate, the accuracy only possessable by subjunctive syllogisms."
-- Damien R. Sullivan

"Money makes the world go round. Love just barely keeps it from blowing up."
-- Unknown

"Soon things got out of hand, where they have remained ever since."
-- Larry Gonick, The Cartoon History of the Universe

"Finally, ask yourself this: Are you sure you've really been abducted by aliens? Do you really want to know? For peace of mind and serenity of spirit, you may only need to remember the following: Ignorance is bliss, Prozac is cheap."
-- Pat Krass

"So it was that on the ninety-fifth day of false winter in the year 2929 since the founding of Neverness, we vowed above all else to seek wisdom and truth, even though our seeking should lead to our death and to the ruin of all that we loved and held dear."
-- David Zindell, Neverness

Original with comments: Rationality Quotes 19

Why Does Power Corrupt?

Eliezer Yudkowsky, 14 October 2008 12:23AM

Followup to: Evolutionary Psychology

"Power tends to corrupt, and absolute power corrupts absolutely. Great men are almost always bad men."
-- Lord Acton

Call it a just-so story if you must, but as soon as I was introduced to the notion of evolutionary psychology (~1995), it seemed obvious to me why human beings are corrupted by power. I didn't then know that hunter-gatherer bands tend to be more egalitarian than agricultural tribes - much less likely to have a central tribal-chief boss-figure - and so I thought of it this way:

Humans (particularly human males) have evolved to exploit power and status when they obtain it, for the obvious reason: If you use your power to take many wives and favor your children with a larger share of the meat, then you will leave more offspring, ceteris paribus. But you're not going to have much luck becoming tribal chief if you just go around saying, "Put me in charge so that I can take more wives and favor my children." You could lie about your reasons, but human beings are not perfect deceivers.

So one strategy that an evolution could follow, would be to create a vehicle that reliably tended to start believing that the old power-structure was corrupt, and that the good of the whole tribe required their overthrow...

The young revolutionary's belief is honest. There will be no betraying catch in his throat, as he explains why the tribe is doomed at the hands of the old and corrupt, unless he is given power to set things right. Not even subconsciously does he think, "And then, once I obtain power, I will strangely begin to resemble that old corrupt guard, abusing my power to increase my inclusive genetic fitness."

People often think as if "purpose" is an inherent property of things; and so many interpret the message of ev-psych as saying, "You have a subconscious, hidden goal to maximize your fitness." But individual organisms are adaptation-executers, not fitness-maximizers. The purpose that the revolutionary should obtain power and abuse it, is not a plan anywhere in his brain; it belongs to evolution, which can just barely be said to have purposes. It is a fact about many past revolutionaries having successfully taken power, having abused it, and having left many descendants.

When the revolutionary obtains power, he will find that it is sweet, and he will try to hold on to it - perhaps still thinking that this is for the good of the tribe. He will find that it seems right to take many wives (surely he deserves some reward for his labor) and to help his children (who are more deserving of help than others). But the young revolutionary has no foreknowledge of this in the beginning, when he sets out to overthrow the awful people who currently rule the tribe - evil mutants whose intentions are obviously much less good than his own.

The circuitry that will respond to power by finding it pleasurable, is already wired into our young revolutionary's brain; but he does not know this. (It would not help him evolutionarily if he did know it, because then he would not be able to honestly proclaim his good intentions - though it is scarcely necessary for evolution to prevent hunter-gatherers from knowing about evolution, which is one reason we are able to know about it now.)

And so we have the awful cycle of "meet the new boss, same as the old boss". Youthful idealism rails against their elders' corruption, but oddly enough, the new generation - when it finally succeeds to power - doesn't seem to be all that morally purer. The original Communist Revolutionaries, I would guess probably a majority of them, really were in it to help the workers; but once they were a ruling Party in charge...

All sorts of random disclaimers can be applied to this thesis: For example, you could suggest that maybe Stalin's intentions weren't all that good to begin with, and that some politicians do intend to abuse power and really are just lying. A much more important objection is the need to redescribe this scenario in terms of power structures that actually exist in hunter-gatherer bands, which, as I understand it, have egalitarian pressures (among adult males) to keep any one person from getting too far above others.

But human beings do find power over others sweet, and it's not as if this emotion could have materialized from thin air, without an evolutionary explanation in terms of hunter-gatherer conditions. If you don't think this is why human beings are corrupted by power - then what's your evolutionary explanation? On the whole, to me at least, the evolutionary explanation for this phenomenon has the problem of not even seeming profound, because what it explains seems so normal.

The moral of this story, and the reason for going into the evolutionary explanation, is that you shouldn't reason as if people who are corrupted by power are evil mutants, whose mutations you do not share.

Evolution is not an infinitely powerful deceiving demon, and our ancestors evolved under conditions of not knowing about evolutionary psychology. The tendency to be corrupted by power can be beaten, I think. The "warp" doesn't seem on the same level of deeply woven insidiousness as, say, confirmation bias.

There was once an occasion where a reporter wrote about me, and did a hatchet job. It was my first time being reported on, and I was completely blindsided by it. I'd known that reporters sometimes wrote hatchet jobs, but I'd thought that it would require malice - I hadn't begun to imagine that someone might write a hatchet job just because it was a cliche, an easy way to generate a few column inches. So I drew upon my own powers of narration, and wrote an autobiographical story on what it felt like to be reported on for the first time - that horrible feeling of violation. I've never sent that story off anywhere, though it's a fine and short piece of writing as I judge it.

For it occurred to me, while I was writing, that journalism is an example of unchecked power - the reporter gets to present only one side of the story, any way they like, and there's nothing that the reported-on can do about it. (If you've never been reported on, then take it from me, that's how it is.) And here I was writing my own story, potentially for publication as traditional journalism, not in an academic forum. I remember realizing that the standards were tremendously lower than in science. That you could get away with damn near anything, so long as it made a good story - that this was the standard in journalism. (If you, having never been reported on yourself, don't believe me that this is the case, then you're as naive as I once was.)

Just that thought - not even the intention, not even wondering whether to do it, but just the thought - that I could present only my side of the story and deliberately make the offending reporter look bad, and that no one would call me on it. Just that thought triggered this huge surge of positive reinforcement. This tremendous high, comparable to the high of discovery or the high of altruism.

And I knew right away what I was dealing with. So I sat there, motionless, fighting down that surge of positive reinforcement. It didn't go away just because I wanted it to go away. But it went away after a few minutes.

If I'd had no label to slap on that huge surge of positive reinforcement - if I'd been a less reflective fellow, flowing more with my passions - then that might have been that. People who are corrupted by power are not evil mutants.

I wouldn't call it a close call. I did know immediately what was happening. I fought it down without much trouble, and could have fought much harder if necessary. So far as I can tell, the temptation of unchecked power is not anywhere near as insidious as the labyrinthine algorithms of self-deception. Evolution is not an infinitely powerful deceiving demon. George Washington refused the temptation of the crown, and he didn't even know about evolutionary psychology. Perhaps it was enough for him to know a little history, and think of the temptation as a sin.

But it was still a scary thing to experience - this circuit that suddenly woke up and dumped a huge dose of unwanted positive reinforcement into my mental workspace, not when I planned to wield unchecked power, but just when my brain visualized the possibility.

To the extent you manage to fight off this temptation, you do not say: "Ah, now that I've beaten the temptation of power, I can safely make myself the wise tyrant who wields unchecked power benevolently, for the good of all." Having successfully fought off the temptation of power, you search for strategies that avoid seizing power. George Washington's triumph was not how well he ruled, but that he refused the crown - despite all temptation to be horrified at who else might then obtain power.

I am willing to admit of the theoretical possibility that someone could beat the temptation of power and then end up with no ethical choice left, except to grab the crown. But there would be a large burden of skepticism to overcome.

Referenced by: Ends Don't Justify Means (Among Humans) • Ethical Inhibitions • Ethical Injunctions • Which Parts Are "Me"? • Devil's Offers • Eutopia is Scary

Original with comments: Why Does Power Corrupt?

Ends Don't Justify Means (Among Humans)

Eliezer Yudkowsky, 14 October 2008 09:00PM

Followup to: Why Does Power Corrupt?

"If the ends don't justify the means, what does?"
-- variously attributed

"I think of myself as running on hostile hardware."
-- Justin Corwin

Yesterday I talked about how humans may have evolved a structure of political revolution, beginning by believing themselves morally superior to the corrupt current power structure, but ending by being corrupted by power themselves - not by any plan in their own minds, but by the echo of ancestors who did the same and thereby reproduced.

This fits the template:

In some cases, human beings have evolved in such fashion as to think that they are doing X for prosocial reason Y, but when human beings actually do X, other adaptations execute to promote self-benefiting consequence Z.

From this proposition, I now move on to my main point, a question considerably outside the realm of classical Bayesian decision theory:

"What if I'm running on corrupted hardware?"

In such a case as this, you might even find yourself uttering such seemingly paradoxical statements - sheer nonsense from the perspective of classical decision theory - as:

"The ends don't justify the means."

But if you are running on corrupted hardware, then the reflective observation that it seems like a righteous and altruistic act to seize power for yourself - this seeming may not be be much evidence for the proposition that seizing power is in fact the action that will most benefit the tribe.

By the power of naive realism, the corrupted hardware that you run on, and the corrupted seemings that it computes, will seem like the fabric of the very world itself - simply the way-things-are.

And so we have the bizarre-seeming rule: "For the good of the tribe, do not cheat to seize power even when it would provide a net benefit to the tribe."

Indeed it may be wiser to phrase it this way: If you just say, "when it seems like it would provide a net benefit to the tribe", then you get people who say, "But it doesn't just seem that way - it would provide a net benefit to the tribe if I were in charge."

The notion of untrusted hardware seems like something wholly outside the realm of classical decision theory. (What it does to reflective decision theory I can't yet say, but that would seem to be the appropriate level to handle it.)

But on a human level, the patch seems straightforward. Once you know about the warp, you create rules that describe the warped behavior and outlaw it. A rule that says, "For the good of the tribe, do not cheat to seize power even for the good of the tribe." Or "For the good of the tribe, do not murder even for the good of the tribe."

And now the philosopher comes and presents their "thought experiment" - setting up a scenario in which, by stipulation, the only possible way to save five innocent lives is to murder one innocent person, and this murder is certain to save the five lives. "There's a train heading to run over five innocent people, who you can't possibly warn to jump out of the way, but you can push one innocent person into the path of the train, which will stop the train. These are your only options; what do you do?"

An altruistic human, who has accepted certain deontological prohibits - which seem well justified by some historical statistics on the results of reasoning in certain ways on untrustworthy hardware - may experience some mental distress, on encountering this thought experiment.

So here's a reply to that philosopher's scenario, which I have yet to hear any philosopher's victim give:

"You stipulate that the only possible way to save five innocent lives is to murder one innocent person, and this murder will definitely save the five lives, and that these facts are known to me with effective certainty. But since I am running on corrupted hardware, I can't occupy the epistemological state you want me to imagine. Therefore I reply that, in a society of Artificial Intelligences worthy of personhood and lacking any inbuilt tendency to be corrupted by power, it would be right for the AI to murder the one innocent person to save five, and moreover all its peers would agree. However, I refuse to extend this reply to myself, because the epistemological state you ask me to imagine, can only exist among other kinds of people than human beings."

Now, to me this seems like a dodge. I think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort. The sort of person who goes around proposing that sort of thought experiment, might well deserve that sort of answer. But any human legal system does embody some answer to the question "How many innocent people can we put in jail to get the guilty ones?", even if the number isn't written down.

As a human, I try to abide by the deontological prohibitions that humans have made to live in peace with one another. But I don't think that our deontological prohibitions are literally inherently nonconsequentially terminally right. I endorse "the end doesn't justify the means" as a principle to guide humans running on corrupted hardware, but I wouldn't endorse it as a principle for a society of AIs that make well-calibrated estimates. (If you have one AI in a society of humans, that does bring in other considerations, like whether the humans learn from your example.)

And so I wouldn't say that a well-designed Friendly AI must necessarily refuse to push that one person off the ledge to stop the train. Obviously, I would expect any decent superintelligence to come up with a superior third alternative. But if those are the only two alternatives, and the FAI judges that it is wiser to push the one person off the ledge - even after taking into account knock-on effects on any humans who see it happen and spread the story, etc. - then I don't call it an alarm light, if an AI says that the right thing to do is sacrifice one to save five. Again, I don't go around pushing people into the paths of trains myself, nor stealing from banks to fund my altruistic projects. I happen to be a human. But for a Friendly AI to be corrupted by power would be like it starting to bleed red blood. The tendency to be corrupted by power is a specific biological adaptation, supported by specific cognitive circuits, built into us by our genes for a clear evolutionary reason. It wouldn't spontaneously appear in the code of a Friendly AI any more than its transistors would start to bleed.

I would even go further, and say that if you had minds with an inbuilt warp that made them overestimate the external harm of self-benefiting actions, then they would need a rule "the ends do not prohibit the means" - that you should do what benefits yourself even when it (seems to) harm the tribe. By hypothesis, if their society did not have this rule, the minds in it would refuse to breathe for fear of using someone else's oxygen, and they'd all die. For them, an occasional overshoot in which one person seizes a personal benefit at the net expense of society, would seem just as cautiously virtuous - and indeed be just as cautiously virtuous - as when one of us humans, being cautious, passes up an opportunity to steal a loaf of bread that really would have been more of a benefit to them than a loss to the merchant (including knock-on effects).

"The end does not justify the means" is just consequentialist reasoning at one meta-level up. If a human starts thinking on the object level that the end justifies the means, this has awful consequences given our untrustworthy brains; therefore a human shouldn't think this way. But it is all still ultimately consequentialism. It's just reflective consequentialism, for beings who know that their moment-by-moment decisions are made by untrusted hardware.

Referenced by: Protected From Myself • Ethical Inhibitions • Ethical Injunctions • Devil's Offers

Original with comments: Ends Don't Justify Means (Among Humans)

Entangled Truths, Contagious Lies

Eliezer Yudkowsky, 15 October 2008 11:39PM

"One of your very early philosophers came to the conclusion that a fully competent mind, from a study of one fact or artifact belonging to any given universe, could construct or visualize that universe, from the instant of its creation to its ultimate end..."
-- First Lensman

"If any one of you will concentrate upon one single fact, or small object, such as a pebble or the seed of a plant or other creature, for as short a period of time as one hundred of your years, you will begin to perceive its truth."
-- Gray Lensman

I am reasonably sure that a single pebble, taken from a beach of our own Earth, does not specify the continents and countries, politics and people of this Earth. Other planets in space and time, other Everett branches, would generate the same pebble. On the other hand, the identity of a single pebble would seem to include our laws of physics. In that sense the entirety of our Universe - all the Everett branches - would be implied by the pebble. (If, as seems likely, there are no truly free variables.)

So a single pebble probably does not imply our whole Earth. But a single pebble implies a very great deal. From the study of that single pebble you could see the laws of physics and all they imply. Thinking about those laws of physics, you can see that planets will form, and you can guess that the pebble came from such a planet. The internal crystals and molecular formations of the pebble formed under gravity, which tells you something about the planet's mass; the mix of elements in the pebble tells you something about the planet's formation.

I am not a geologist, so I don't know to which mysteries geologists are privy. But I find it very easy to imagine showing a geologist a pebble, and saying, "This pebble came from a beach at Half Moon Bay", and the geologist immediately says, "I'm confused" or even "You liar". Maybe it's the wrong kind of rock, or the pebble isn't worn enough to be from a beach - I don't know pebbles well enough to guess the linkages and signatures by which I might be caught, which is the point.

"Only God can tell a truly plausible lie." I wonder if there was ever a religion that developed this as a proverb? I would (falsifiably) guess not: it's a rationalist sentiment, even if you cast it in theological metaphor. Saying "everything is interconnected to everything else, because God made the whole world and sustains it" may generate some nice warm n' fuzzy feelings during the sermon, but it doesn't get you very far when it comes to assigning pebbles to beaches.

A penny on Earth exerts a gravitational acceleration on the Moon of around 4.5 * 10-31 m/s^2, so in one sense it's not too far wrong to say that every event is entangled with its whole past light cone. And since inferences can propagate backward and forward through causal networks, epistemic entanglements can easily cross the borders of light cones. But I wouldn't want to be the forensic astronomer who had to look at the Moon and figure out whether the penny landed heads or tails - the influence is far less than quantum uncertainty and thermal noise.

If you said "Everything is entangled with something else" or "Everything is inferentially entangled and some entanglements are much stronger than others", you might be really wise instead of just Deeply Wise.

Physically, each event is in some sense the sum of its whole past light cone, without borders or boundaries. But the list of noticeable entanglements is much shorter, and it gives you something like a network. This high-level regularity is what I refer to when I talk about the Great Web of Causality.

I use these Capitalized Letters somewhat tongue-in-cheek, perhaps; but if anything at all is worth Capitalized Letters, surely the Great Web of Causality makes the list.

"Oh what a tangled web we weave, when first we practise to deceive," said Sir Walter Scott. Not all lies spin out of control - we don't live in so righteous a universe. But it does occasionally happen, that someone lies about a fact, and then has to lie about an entangled fact, and then another fact entangled with that one:

"Where were you?"
"Oh, I was on a business trip."
"What was the business trip about?"
"I can't tell you that; it's proprietary negotiations with a major client."
"Oh - they're letting you in on those? Good news! I should call your boss to thank him for adding you."
"Sorry - he's not in the office right now..."

Human beings, who are not gods, often fail to imagine all the facts they would need to distort to tell a truly plausible lie. "God made me pregnant" sounded a tad more likely in the old days before our models of the world contained (quotations of) Y chromosomes. Many similar lies, today, may blow up when genetic testing becomes more common. Rapists have been convicted, and false accusers exposed, years later, based on evidence they didn't realize they could leave. A student of evolutionary biology can see the design signature of natural selection on every wolf that chases a rabbit; and every rabbit that runs away; and every bee that stings instead of broadcasting a polite warning - but the deceptions of creationists sound plausible to them, I'm sure.

Not all lies are uncovered, not all liars are punished; we don't live in that righteous a universe. But not all lies are as safe as their liars believe. How many sins would become known to a Bayesian superintelligence, I wonder, if it did a (non-destructive?) nanotechnological scan of the Earth? At minimum, all the lies of which any evidence still exists in any brain. Some such lies may become known sooner than that, if the neuroscientists ever succeed in building a really good lie detector via neuroimaging. Paul Ekman (a pioneer in the study of tiny facial muscle movements) could probably read off a sizeable fraction of the world's lies right now, given a chance.

Not all lies are uncovered, not all liars are punished. But the Great Web is very commonly underestimated. Just the knowledge that humans have already accumulated would take many human lifetimes to learn. Anyone who thinks that a non-God can tell a perfect lie, risk-free, is underestimating the tangledness of the Great Web.

Is honesty the best policy? I don't know if I'd go that far: Even on my ethics, it's sometimes okay to shut up. But compared to outright lies, either honesty or silence involves less exposure to recursively propagating risks you don't know you're taking.

Sequence: Against Rationalization

Is That Your True Rejection?

Of Lies and Black Swan Blowups

Referenced by: Dark Side Epistemology • Protected From Myself • Ethical Inhibitions • Ethical Injunctions • Of Lies and Black Swan Blowups • You're Entitled to Arguments, But Not (That Particular) Proof

Original with comments: Entangled Truths, Contagious Lies

Traditional Capitalist Values

Eliezer Yudkowsky, 17 October 2008 01:07AM

Followup to: Are Your Enemies Innately Evil?, Policy Debates Should Not Appear One-Sided

"The financial crisis is not the crisis of capitalism. It is the crisis of a system that has distanced itself from the most fundamental values of capitalism, which betrayed the spirit of capitalism."
-- Nicolas Sarkozy

During the current crisis, I've more than once heard someone remarking that financial-firm CEOs who take huge bonuses during the good years and then run away when their black-swan bets blow up, are only exercising the usual capitalist values of "grab all the money you can get".

I think that a fair amount of the enmity in the world, to say nothing of confusion on the Internet, stems from people refusing to contemplate the real values of the opposition as the opposition sees it. This is something I've remarked upon before, with respect to "the terrorists hate our freedom" or "the suicide hijackers were cowards" (statements that are sheerly silly).

Real value systems - as opposed to pretend demoniacal value systems - are phrased to generate warm fuzzies in their users, not to be easily mocked. They will sound noble at least to the people who believe them.

Whether anyone actually lives up to that value system, or should, and whether the results are what they are claimed to be; if there are hidden gotchas in the warm fuzzy parts - sure, you can have that debate. But first you should be clear about how your opposition sees itself - a view which has not been carefully optimized to make your side feel good about its opposition. Otherwise you're not engaging the real issues.

So here are the traditional values of capitalism as seen by those who regard it as noble - the sort of Way spoken of by Paul Graham, or P. T. Barnum (who did not say "There's a sucker born every minute"), or Warren Buffett:

- Make things that people want, or do things that people want done, in exchange for money or other valuta. This is a great and noble and worthwhile endeavor, and anyone who looks down on it reveals their own shallowness.

- Your competitors are your loyal opposition. Usher them toward oblivion by offering a better product at a lower price, then dance on their graves (or buy them out, if they're worthy). An act of violence, like punching them in the nose, is as absolutely forbidden as it would be to a scientist - violence is reserved for the thuggish lower classes, like politicians.

- Plan for the long term, in your own life and in your company. Short-term thinkers lose money; long-term thinkers get to pick up the remains for a song.

- Acquire a deserved reputation for honesty and reliability - with your customers, your suppliers, your subordinates, your supervisors, and the general community. Read the contract before you sign it, then do what you said you would. Play by the rules, and play to win.

- Save more than you earn. Keep a cushion for rainy days. Fear debt.

- Pay your shareholders a good dividend - that's why they originally gave you their money to invest, to make more money.

- Don't fiddle the numbers. The numbers are very important, so fiddling with them is very bad.

- Promote based on merit. Being nice to your nephew-in-law will make less money.

- Give someone a second chance, but not a third chance.

- Science and technology are responsible for there being something to trade other than stone knives and fruit. Adopt innovations to increase productivity. Respect expertise and use it to make money.

- Vigorous work is praiseworthy but should be accompanied by equally vigorous results.

- No one has a right to their job. Not the janitor, not the CEO, no one. It would be like a rationalist having a right to their own opinion. At some point you've got to fire the saddle-makers and close down the industry. If you want to reward loyalty, give them money.

- No company has a right to its continued existence. Change happens.

- Investing is risky. If you don't like it, don't invest. Diversification is one thing, but stay far away from get-rich-quick schemes that offer easy rewards with no risk or hard thinking. Trying to vote yourself rich falls in this category.

- A high standard of living is the just reward of hard work and intelligence. If other people or other places have lower standards of living, then the problem is the lower standard, not the higher one. Raise others up, don't lower yourself. A high standard of living is a good thing, not a bad one - a universal moral generalization that includes you in particular. If you've earned your wealth honestly, enjoy it without regrets.

- In all ways, at all times, and with every deed, make the pie larger rather than smaller.

- The phrase "making money" is a triumph in itself over previous worldviews that saw wealth as something to steal.

- Create value so that you can capture it, but don't feel obligated to capture all the value you create. If you capture all your value, your transactions benefit only yourself, and others have no motive to participate. If you have negotiating leverage, use it to drive a good bargain for yourself, but not a hateful one - someday you'll be on the other side of the table. Still, the invisible hand does require that price be sensitive to supply and demand.

- Everyone in a company should be pulling together to create value and getting a decent share of the value they create. The concept of 'class war' is a lie sold by politicians who think in terms of a fixed pie. Any person of wealth who actually, seriously tries to exploit those less fortunate is a bully and a heretic; but offering someone a job is not exploiting them. (Though bribing politicians to pass laws that transfer wealth, definitely could qualify as exploitation.)

- In countries that are lawful and just, it is the privilege and responsibility of a citizen to pay their low taxes. That said, a good billionaire wouldn't ask to pay a lower tax rate than his secretary.

- People safeguard, nourish, and improve that which they know will not be taken from them. Tax a little if you must, but at some point you must let people own what they buy.

- The fundamental morality of capitalism lies in the voluntary nature of its trades, consented to by all parties, and therefore providing a gain to all.

- In countries that are lawful and just, the government is the referee, not a player. If the referee runs onto the field and kicks the football, things start to get scary.

- Unearned gains destroy people, nations, and teenagers. If you want to help, go into the dark places of the world and open stores that offer lower prices; or offer employment to people who wouldn't have gotten a chance otherwise; or invest capital in worthy recipients who others shun. Plow your gains back into the operation, if you like; but if you don't make at least a little money in the process, how can you be sure that you're creating value?

- Wise philanthropy is a privilege and responsibility of achieved wealth.

- Making money is a virtuous endeavor, despite all the lies that have been told about it, and should properly be found in the company of other virtues. Those who set out to make money should not think of themselves as fallen, but should rather conduct themselves with honor, pride, and self-respect, as part of the grand pageantry of human civilization rising up from the dirt, and continuing forward into the future.

There was, once upon a time, an editorial in the Wall Street Journal calling Ford a "traitor to his class" because he offered more than the prevailing wages of the time. Coal miners trying to form a union, once upon a time, were fired upon by rifles. But I also think that Graham or Barnum or Buffett would regard those folk as the inheritors of kings.

"No true Scotsman" fallacy? Maybe, but let's at least be clear what the Scots say about it.

For myself, I would have to say that I'm an apostate from this moral synthesis - I grew up in this city and remember it fondly, but I no longer live there. I regard finance as more of a useful tool than an ultimate end of intelligence - I'm not sure it's the maximum possible fun we could all be having under optimal conditions. I'm more sympathetic than this to people who lose their jobs, because I know that retraining, or changing careers, isn't always easy and fun. I don't think the universe is set up to reward hard work; and I think that it is entirely possible for money to corrupt a person.

But I also admire any virtue clearly stated and synthesized into a moral system. We need more of those. Anyone who thinks that capitalism is just about grabbing the banana, is underestimating the number of decent and intelligent people who have put serious thought into the subject.

Those of other Ways may not agree with all these statements - but if you aspire to sanity in debate, you should at least be able to read them without your brain shutting down.

PS: Julian Morrison adds: Trade can act as a connective between people with diverging values.

Referenced by: Good Idealistic Books are Rare

Original with comments: Traditional Capitalist Values

Dark Side Epistemology

Eliezer Yudkowsky, 17 October 2008 11:55PM

Followup to: Entangled Truths, Contagious Lies

If you once tell a lie, the truth is ever after your enemy.

I have previously spoken of the notion that, the truth being entangled, lies are contagious. If you pick up a pebble from the driveway, and tell a geologist that you found it on a beach - well, do you know what a geologist knows about rocks? I don't. But I can suspect that a water-worn pebble wouldn't look like a droplet of frozen lava from a volcanic eruption. Do you know where the pebble in your driveway really came from? Things bear the marks of their places in a lawful universe; in that web, a lie is out of place.

What sounds like an arbitrary truth to one mind - one that could easily be replaced by a plausible lie - might be nailed down by a dozen linkages to the eyes of greater knowledge. To a creationist, the idea that life was shaped by "intelligent design" instead of "natural selection" might sound like a sports team to cheer for. To a biologist, plausibly arguing that an organism was intelligently designed would require lying about almost every facet of the organism. To plausibly argue that "humans" were intelligently designed, you'd have to lie about the design of the human retina, the architecture of the human brain, the proteins bound together by weak van der Waals forces instead of strong covalent bonds...

Or you could just lie about evolutionary theory, which is the path taken by most creationists. Instead of lying about the connected nodes in the network, they lie about the general laws governing the links.

And then to cover that up, they lie about the rules of science - like what it means to call something a "theory", or what it means for a scientist to say that they are not absolutely certain.

So they pass from lying about specific facts, to lying about general laws, to lying about the rules of reasoning. To lie about whether humans evolved, you must lie about evolution; and then you have to lie about the rules of science that constrain our understanding of evolution.

But how else? Just as a human would be out of place in a community of actually intelligently designed life forms, and you have to lie about the rules of evolution to make it appear otherwise; so too, beliefs about creationism are themselves out of place in science - you wouldn't find them in a well-ordered mind any more than you'd find palm trees growing on a glacier. And so you have to disrupt the barriers that would forbid them.

Which brings us to the case of self-deception.

A single lie you tell yourself may seem plausible enough, when you don't know any of the rules governing thoughts, or even that there are rules; and the choice seems as arbitrary as choosing a flavor of ice cream, as isolated as a pebble on the shore...

...but then someone calls you on your belief, using the rules of reasoning that they've learned. They say, "Where's your evidence?"

And you say, "What? Why do I need evidence?"

So they say, "In general, beliefs require evidence."

This argument, clearly, is a soldier fighting on the other side, which you must defeat. So you say: "I disagree! Not all beliefs require evidence. In particular, beliefs about dragons don't require evidence. When it comes to dragons, you're allowed to believe anything you like. So I don't need evidence to believe there's a dragon in my garage."

And the one says, "Eh? You can't just exclude dragons like that. There's a reason for the rule that beliefs require evidence. To draw a correct map of the city, you have to walk through the streets and make lines on paper that correspond to what you see. That's not an arbitrary legal requirement - if you sit in your living room and draw lines on the paper at random, the map's going to be wrong. With extremely high probability. That's as true of a map of a dragon as it is of anything."

So now this, the explanation of why beliefs require evidence, is also an opposing soldier. So you say: "Wrong with extremely high probability? Then there's still a chance, right? I don't have to believe if it's not absolutely certain."

Or maybe you even begin to suspect, yourself, that "beliefs require evidence". But this threatens a lie you hold precious; so you reject the dawn inside you, push the sun back under the horizon.

Or you've previously heard the proverb "beliefs require evidence", and it sounded wise enough, and you endorsed it in public. But it never quite occurred to you, until someone else brought it to your attention, that this provreb could apply to your belief that there's a dragon in your garage. So you think fast and say, "The dragon is in a separate magisterium."

Having false beliefs isn't a good thing, but it doesn't have to be permanently crippling - if, when you discover your mistake, you get over it. The dangerous thing is to have a false belief that you believe should be protected as a belief - a belief-in-belief, whether or not accompanied by actual belief.

A single Lie That Must Be Protected can block someone's progress into advanced rationality. No, it's not harmless fun.

Just as the world itself is more tangled by far than it appears on the surface; so too, there are stricter rules of reasoning, constraining belief more strongly, than the untrained would suspect. The world is woven tightly, governed by general laws, and so are rational beliefs.

Think of what it would take to deny evolution or heliocentrism - all the connected truths and governing laws you wouldn't be allowed to know. Then you can imagine how a single act of self-deception can block off the whole meta-level of truthseeking, once your mind begins to be threatened by seeing the connections. Forbidding all the intermediate and higher levels of the rationalist's Art. Creating, in its stead, a vast complex of anti-law, rules of anti-thought, general justifications for believing the untrue.

Steven said, "Promoting less than maximally accurate beliefs is an act of sabotage. Don't do it to anyone unless you'd also slash their tires." Giving someone a false belief to protect - convincing them that the belief itself must be defended from any thought that seems to threaten it - well, you shouldn't do that to someone unless you'd also give them a frontal lobotomy.

Once you tell a lie, the truth is your enemy; and every truth connected to that truth, and every ally of truth in general; all of these you must oppose, to protect the lie. Whether you're lying to others, or to yourself.

You have to deny that beliefs require evidence, and then you have to deny that maps should reflect territories, and then you have to deny that truth is a good thing...

Thus comes into being the Dark Side.

I worry that people aren't aware of it, or aren't sufficiently wary - that as we wander through our human world, we can expect to encounter systematically bad epistemology.

The "how to think" memes floating around, the cached thoughts of Deep Wisdom - some of it will be good advice devised by rationalists. But other notions were invented to protect a lie or self-deception: spawned from the Dark Side.

"Everyone has a right to their own opinion." When you think about it, where was that proverb generated? Is it something that someone would say in the course of protecting a truth, or in the course of protecting from the truth? But people don't perk up and say, "Aha! I sense the presence of the Dark Side!" As far as I can tell, it's not widely realized that the Dark Side is out there.

But how else? Whether you're deceiving others, or just yourself, the Lie That Must Be Protected will propagate recursively through the network of empirical causality, and the network of general empirical rules, and the rules of reasoning themselves, and the understanding behind those rules. If there is good epistemology in the world, and also lies or self-deceptions that people are trying to protect, then there will come into existence bad epistemology to counter the good. We could hardly expect, in this world, to find the Light Side without the Dark Side; there is the Sun, and that which shrinks away and generates a cloaking Shadow.

Mind you, these are not necessarily evil people. The vast majority who go about repeating the Deep Wisdom are more duped than duplicitous, more self-deceived than deceiving. I think.

And it's surely not my intent to offer you a Fully General Counterargument, so that whenever someone offers you some epistemology you don't like, you say: "Oh, someone on the Dark Side made that up." It's one of the rules of the Light Side that you have to refute the proposition for itself, not by accusing its inventor of bad intentions.

But the Dark Side is out there. Fear is the path that leads to it, and one betrayal can turn you. Not all who wear robes are either Jedi or fakes; there are also the Sith Lords, masters and unwitting apprentices. Be warned, be wary.

As for listing common memes that were spawned by the Dark Side - not random false beliefs, mind you, but bad epistemology, the Generic Defenses of Fail - well, would you care to take a stab at it, dear readers?

Sequence: Against Rationalization

Of Lies and Black Swan Blowups

The Sacred Mundane

Referenced by: Ethical Injunctions • Artificial Mysterious Intelligence • Belief in Self-Deception • Striving to Accept • Raising the Sanity Waterline • The Sacred Mundane • Beware of Other-Optimizing • My Way

Original with comments: Dark Side Epistemology

Protected From Myself

Eliezer Yudkowsky, 19 October 2008 12:09AM

Followup to: The Magnitude of His Own Folly, Entangled Truths, Contagious Lies

Every now and then, another one comes before me with the brilliant idea: "Let's lie!"

Lie about what? - oh, various things. The expected time to Singularity, say. Lie and say it's definitely going to be earlier, because that will get more public attention. Sometimes they say "be optimistic", sometimes they just say "lie". Lie about the current degree of uncertainty, because there are other people out there claiming to be certain, and the most unbearable prospect in the world is that someone else pull ahead. Lie about what the project is likely to accomplish - I flinch even to write this, but occasionally someone proposes to go and say to the Christians that the AI will create Christian Heaven forever, or go to the US government and say that the AI will give the US dominance forever.

But at any rate, lie. Lie because it's more convenient than trying to explain the truth. Lie, because someone else might lie, and so we have to make sure that we lie first. Lie to grab the tempting benefits, hanging just within reach -

Eh? Ethics? Well, now that you mention it, lying is at least a little bad, all else being equal. But with so much at stake, we should just ignore that and lie. You've got to follow the expected utility, right? The loss of a lie is much less than the benefit to be gained, right?

Thus do they argue. Except - what's the flaw in the argument? Wouldn't it be irrational not to lie, if lying has the greatest expected utility?

When I look back upon my history - well, I screwed up in a lot of ways. But it could have been much worse, if I had reasoned like those who offer such advice, and lied.

Once upon a time, I truly and honestly believed that either a superintelligence would do what was right, or else there was no right thing to do; and I said so. I was uncertain of the nature of morality, and I said that too. I didn't know if the Singularity would be in five years or fifty, and this also I admitted. My project plans were not guaranteed to deliver results, and I did not promise to deliver them. When I finally said "Oops", and realized that I needed to go off and do more fundamental research instead of rushing to write code immediately -

- well, I can imagine the mess I would have had on my hands, if I had told the people who trusted me: that the Singularity was surely coming in ten years; that my theory was sure to deliver results; that I had no lingering confusions; and that any superintelligence would surely give them their own private island and a harem of catpersons of the appropriate gender. How exactly would one then explain why you're now going to step back and look for math-inventors instead of superprogrammers, or why the code now has to be theorem-proved?

When you make an honest mistake, on some subject you were honest about, the recovery technique is straightforward: Just as you told people what you thought in the first place, you now list out the actual reasons that you changed your mind. This diff takes you to your current true thoughts, that imply your current desired policy. Then, just as people decided whether to aid you originally, they re-decide in light of the new information.

But what if you were "optimistic" and only presented one side of the story, the better to fulfill that all-important goal of persuading people to your cause? Then you'll have a much harder time persuading them away from that idea you sold them originally - you've nailed their feet to the floor, which makes it difficult for them to follow if you yourself take another step forward.

And what if, for the sake of persuasion, you told them things that you didn't believe yourself? Then there is no true diff from the story you told before, to the new story now. Will there be any coherent story that explains your change of heart?

Conveying the real truth is an art form. It's not an easy art form - those darned constraints of honesty prevent you from telling all kinds of convenient lies that would be so much easier than the complicated truth. But, if you tell lots of truth, you get good at what you practice. A lot of those who come to me and advocate lies, talk earnestly about how these matters of transhumanism are so hard to explain, too difficult and technical for the likes of Joe the Plumber. So they'd like to take the easy way out, and lie.

We don't live in a righteous universe where all sins are punished. Someone who practiced telling lies, and made their mistakes and learned from them, might well become expert at telling lies that allow for sudden changes of policy in the future, and telling more lies to explain the policy changes. If you use the various forbidden arts that create fanatic followers, they will swallow just about anything. The history of the Soviet Union and their sudden changes of policy, as presented to their ardent Western intellectual followers, helped inspire Orwell to write 1984.

So the question, really, is whether you want to practice truthtelling or practice lying, because whichever one you practice is the one you're going to get good at. Needless to say, those who come to me and offer their unsolicited advice do not appear to be expert liars. For one thing, a majority of them don't seem to find anything odd about floating their proposals in publicly archived, Google-indexed mailing lists.

But why not become an expert liar, if that's what maximizes expected utility? Why take the constrained path of truth, when things so much more important are at stake?

Because, when I look over my history, I find that my ethics have, above all, protected me from myself. They weren't inconveniences. They were safety rails on cliffs I didn't see.

I made fundamental mistakes, and my ethics didn't halt that, but they played a critical role in my recovery. When I was stopped by unknown unknowns that I just wasn't expecting, it was my ethical constraints, and not any conscious planning, that had put me in a recoverable position.

You can't duplicate this protective effect by trying to be clever and calculate the course of "highest utility". The expected utility just takes into account the things you know to expect. It really is amazing, looking over my history, the extent to which my ethics put me in a recoverable position from my unanticipated, fundamental mistakes, the things completely outside my plans and beliefs.

Ethics aren't just there to make your life difficult; they can protect you from Black Swans. A startling assertion, I know, but not one entirely irrelevant to current affairs.

If you've been following along my story, you'll recall that the downfall of all my theories, began with a tiny note of discord. A tiny note that I wouldn't ever have followed up, if I had only cared about my own preferences and desires. It was the thought of what someone else might think - someone to whom I felt I owed an ethical consideration - that spurred me to follow up that one note.

And I have watched others fail utterly on the problem of Friendly AI, because they simply try to grab the banana in one form or another - seize the world for their own favorite moralities, without any thought of what others might think - and so they never enter into the complexities and second thoughts that might begin to warn them of the technical problems.

We don't live in a righteous universe. And so, when I look over my history, the role that my ethics have played is so important that I've had to take a step back and ask, "Why is this happening?" The universe isn't set up to reward virtue - so why did my ethics help so much? Am I only imagining the phenomenon? That's one possibility. But after some thought, I've concluded that, to the extent you believe that my ethics did help me, these are the plausible reasons in order of importance:

1) The honest Way often has a kind of simplicity that trangressions lack. If you tell lies, you have to keep track of different stories you've told different groups, and worry about which facts might encounter the wrong people, and then invent new lies to explain any unexpected policy shifts you have to execute on account of your mistake. This simplicity is powerful enough to explain a great deal of the positive influence that I attribute to my ethics, in a universe that doesn't reward virtue per se.

2) I was stricter with myself, and held myself to a higher standard, when I was doing various things that I considered myself ethically obligated to do. Thus my recovery from various failures often seems to have begun with an ethical thought of some type - e.g. the whole development where "Friendly AI" led into the concept of AI as a precise art. That might just be a quirk of my own personality; but it seems to help account for the huge role my ethics played in leading me to important thoughts, which I cannot just explain by saying that the universe rewards virtue.

3) The constraints that the wisdom of history suggests, to avoid hurting other people, may also stop you from hurting yourself. When you have some brilliant idea that benefits the tribe, we don't want you to run off and do X, Y, and Z, even if you say "the end justifies the means!" Evolutionarily speaking, one suspects that the "means" have more often benefited the person who executes them, than the tribe. But this is not the ancestral environment. In the more complicated modern world, following the ethical constraints can prevent you from making huge networked mistakes that would catch you in their collapse. Robespierre led a shorter life than Washington.

Referenced by: Ethical Injunctions • Prices or Bindings? • Why Our Kind Can't Cooperate

Original with comments: Protected From Myself

Ethical Inhibitions

Eliezer Yudkowsky, 19 October 2008 08:44PM

Followup to: Entangled Truths, Contagious Lies, Evolutionary Psychology

What's up with that bizarre emotion we humans have, this sense of ethical caution?

One can understand sexual lust, parental care, and even romantic attachment. The evolutionary psychology of such emotions might be subtler than it at first appears, but if you ignore the subtleties, the surface reasons are obvious. But why a sense of ethical caution? Why honor, why righteousness? (And no, it's not group selection; it never is.) What reproductive benefit does that provide?

The specific ethical codes that people feel uneasy violating, vary from tribe to tribe (though there are certain regularities). But the emotion associated with feeling ethically inhibited - well, I Am Not An Evolutionary Anthropologist, but that looks like a human universal to me, something with brainware support.

The obvious story behind prosocial emotions in general, is that those who offend against the group are sanctioned; this converts the emotion to an individual reproductive advantage. The human organism, executing the ethical-caution adaptation, ends up avoiding the group sanctions that would follow a violation of the code. This obvious answer may even be the entire answer.

But I suggest - if a bit more tentatively than usual - that by the time human beings were evolving the emotion associated with "ethical inhibition", we were already intelligent enough to observe the existence of such things as group sanctions. We were already smart enough (I suggest) to model what the group would punish, and to fear that punishment.

Sociopaths have a concept of getting caught, and they try to avoid getting caught. Why isn't this sufficient? Why have an extra emotion, a feeling that inhibits you even when you don't expect to be caught? Wouldn't this, from evolution's perspective, just result in passing up perfectly good opportunities?

So I suggest (tentatively) that humans naturally underestimate the odds of getting caught. We don't foresee all the possible chains of causality, all the entangled facts that can bring evidence against us. Those ancestors who lacked a sense of ethical caution stole the silverware when they expected that no one would catch them or punish them; and were nonetheless caught or punished often enough, on average, to outweigh the value of the silverware.

Admittedly, this may be an unnecessary assumption. It is a general idiom of biology that evolution is the only long-term consequentialist; organisms compute short-term rewards. Hominids violate this rule, but that is a very recent innovation.

So one could counter-argue: "Early humans didn't reliably forecast the punishment that follows from breaking social codes, so they didn't reliably think consequentially about it, so they developed an instinct to obey the codes." Maybe the modern sociopaths that evade being caught are smarter than average. Or modern sociopaths are better educated than hunter-gatherer sociopaths. Or modern sociopaths get more second chances to recover from initial stumbles - they can change their name and move. It's not so strange to find an emotion executing in some exceptional circumstance where it fails to provide a reproductive benefit.

But I feel justified in bringing up the more complicated hypothesis, because ethical inhibitions are archetypallythat which stops us even when we think no one is looking. A humanly universal concept, so far as I know, though I am not an anthropologist.

Ethical inhibition, as a human motivation, seems to be implemented in a distinct style from hunger or lust. Hunger and lust can be outweighed when stronger desires are at stake; but the emotion associated with ethical prohibitions tries to assert itself deontologically. If you have the sense at all that you shouldn't do it, you have the sense that you unconditionally shouldn't do it. The emotion associated with ethical caution would seem to be a drive that - successfully or unsuccessfully - tries to override the temptation, not just weigh against it.

A monkey can be trapped by a food reward inside a hollowed shell - they can reach in easily enough, but once they close their fist, they can't take their hand out. The monkey may be screaming with distress, and still be unable to override the instinct to keep hold of the food. We humans can do better than that; we can let go of the food reward and run away, when our brain is warning us of the long-term consequences.

But why does the sensation of ethical inhibition, that might also command us to pass up a food reward, have a similar override-quality - even in the absence of explicitly expected long-term consequences? Is it just that ethical emotions evolved recently, and happen to be implemented in prefrontal cortex next to the long-term-override circuitry?

What is this tendency to feel inhibited from stealing the food reward? This message that tries to assert "I override", not just "I weigh against"? Even when we don't expect the long-term consequences of being discovered?

And before you think that I'm falling prey to some kind of appealing story, ask yourself why that particular story would sound appealing to humans. Why would it seem temptingly virtuous to let an ethical inhibition override, rather than just being one more weight in the balance?

One possible explanation would be if the emotion were carved out by the evolutionary-historical statistics of a black-swan bet.

Maybe you will, in all probability, get away with stealing the silverware on any particular occasion - just as your model of the world would extrapolate. But it was a statistical fact about your ancestors that sometimes the environment didn't operate the way they expected. Someone was watching from behind the trees. On those occasions their reputation was permanently blackened; they lost status in the tribe, and perhaps were outcast or murdered. Such occasions could be statistically rare, and still counterbalance the benefit of a few silver spoons.

The brain, like every other organ in the body, is a reproductive organ: it was carved out of entropy by the persistence of mutations that promoted reproductive fitness. And yet somehow, amazingly, the human brain wound up with circuitry for such things as honor, sympathy, and ethical resistance to temptations.

Which means that those alleles drove their alternatives to extinction. Humans, the organisms, can be nice to each other; but the alleles' game of frequencies is zero-sum. Honorable ancestors didn't necessarily kill the dishonorable ones. But if, by cooperating with each other, honorable ancestors outreproduced less honorable folk, then the honor allele killed the dishonor allele as surely as if it erased the DNA sequence off a blackboard.

That might be something to think about, the next time you're wondering if you should just give in to your ethical impulses, or try to override them with your rational awareness.

Especially if you're tempted to engage in some chicanery "for the greater good" - tempted to decide that the end justifies the means. Evolution doesn't care about whether something actually promotes the greater good - that's not how gene frequencies change. But if transgressive plans go awry often enough to hurt the transgressor, how much more often would they go awry and hurt the intended beneficiaries?

Historically speaking, it seems likely that, of those who set out to rob banks or murder opponents "in a good cause", those who managed to hurt themselves, mostly wouldn't make the history books. (Unless they got a second chance, like Hitler after the failed Beer Hall Putsch.) Of those cases we do read about in the history books, many people have done very well for themselves out of their plans to lie and rob and murder "for the greater good". But how many people cheated their way to actual huge altruistic benefits - cheated and actually realized the justifying greater good? Surely there must be at least one or two cases known to history - at least one king somewhere who took power by lies and assassination, and then ruled wisely and well - but I can't actually name a case off the top of my head. By and large, it seems to me a pretty fair generalization that people who achieve great good ends manage not to find excuses for all that much evil along the way.

Somehow, people seem much more likely to endorse plans that involve just a little pain for someone else, on behalf of the greater good, than to work out a way to let the sacrifice be themselves. But when you plan to damage society in order to save it, remember that your brain contains a sense of ethical unease that evolved from transgressive plans blowing up and damaging the originator - never mind the expected value of all the damage done to other people, if you really do care about them.

If natural selection, which doesn't care at all about the welfare of unrelated strangers, still manages to give you a sense of ethical unease on account of transgressive plans not always going as planned - then how much more reluctant should you be to rob banks for a good cause, if you aspire to actually help and protect others?

Referenced by: Ethical Injunctions • Ethics Notes

Original with comments: Ethical Inhibitions

Ethical Injunctions

Eliezer Yudkowsky, 20 October 2008 11:00PM

Followup to: Ethical Inhibitions, Ends Don't Justify Means (Among Humans), Entangled Truths, Contagious Lies, Protected From Myself

"Would you kill babies if it was the right thing to do? If no, under what circumstances would you not do the right thing to do? If yes, how right would it have to be, for how many babies?"
-- horrible job interview question

Swapping hats for a moment, I'm professionally intrigued by the decision theory of "things you shouldn't do even if they seem to be the right thing to do".

Suppose we have a reflective AI, self-modifying and self-improving, at an intermediate stage in the development process. In particular, the AI's goal system isn't finished - the shape of its motivations is still being loaded, learned, tested, or tweaked.

Yea, I have seen many ways to screw up an AI goal system design, resulting in a decision system that decides, given its goals, that the universe ought to be tiled with tiny molecular smiley-faces, or some such. Generally, these deadly suggestions also have the property that the AI will not desire its programmers to fix it. If the AI is sufficiently advanced - which it may be even at an intermediate stage - then the AI may also realize that deceiving the programmers, hiding the changes in its thoughts, will help transform the universe into smiley-faces.

Now, from our perspective as programmers, if we condition on the fact that the AI has decided to hide its thoughts from the programmers, or otherwise act willfully to deceive us, then it would seem likely that some kind of unintended consequence has occurred in the goal system. We would consider it probable that the AI is not functioning as intended, but rather likely that we have messed up the AI's utility function somehow. So that the AI wants to turn the universe into tiny reward-system counters, or some such, and now has a motive to hide from us.

Well, suppose we're not going to implement some object-level Great Idea as the AI's utility function. Instead we're going to do something advanced and recursive - build a goal system which knows (and cares) about the programmers outside. A goal system that, via some nontrivial internal structure, "knows it's being programmed" and "knows it's incomplete". Then you might be able to have and keep the rule:

"If [I decide that] fooling my programmers is the right thing to do, execute a controlled shutdown [instead of doing the right thing to do]."

And the AI would keep this rule, even through the self-modifying AI's revisions of its own code, because, in its structurally nontrivial goal system, the present-AI understands that this decision by a future-AI probably indicates something defined-as-a-malfunction. Moreover, the present-AI knows that if future-AI tries to evaluate the utility of executing a shutdown, once this hypothetical malfunction has occurred, the future-AI will probably decide not to shut itself down. So the shutdown should happen unconditionally, automatically, without the goal system getting another chance to recalculate the right thing to do.

I'm not going to go into the deep dark depths of the exact mathematical structure, because that would be beyond the scope of this blog. Also I don't yet know the deep dark depths of the mathematical structure. It looks like it should be possible, if you do things that are advanced and recursive and have nontrivial (but consistent) structure. But I haven't reached that level, as yet, so for now it's only a dream.

But the topic here is not advanced AI; it's human ethics. I introduce the AI scenario to bring out more starkly the strange idea of an ethical injunction:

You should never, ever murder an innocent person who's helped you, even if it's the right thing to do; because it's far more likely that you've made a mistake, than that murdering an innocent person who helped you is the right thing to do.

Sound reasonable?

During World War II, it became necessary to destroy Germany's supply of deuterium, a neutron moderator, in order to block their attempts to achieve a fission chain reaction. Their supply of deuterium was coming at this point from a captured facility in Norway. A shipment of heavy water was on board a Norwegian ferry ship, the SF Hydro. Knut Haukelid and three others had slipped on board the ferry in order to sabotage it, when the saboteurs were discovered by the ferry watchman. Haukelid told him that they were escaping the Gestapo, and the watchman immediately agreed to overlook their presence. Haukelid "considered warning their benefactor but decided that might endanger the mission and only thanked him and shook his hand." (Richard Rhodes, The Making of the Atomic Bomb.) So the civilian ferry Hydro sank in the deepest part of the lake, with eighteen dead and twenty-nine survivors. Some of the Norwegian rescuers felt that the German soldiers present should be left to drown, but this attitude did not prevail, and four Germans were rescued. And that was, effectively, the end of the Nazi atomic weapons program.

Good move? Bad move? Germany very likely wouldn't have gotten the Bomb anyway... I hope with absolute desperation that I never get faced by a choice like that, but in the end, I can't say a word against it.

On the other hand, when it comes to the rule:

"Never try to deceive yourself, or offer a reason to believe other than probable truth; because even if you come up with an amazing clever reason, it's more likely that you've made a mistake than that you have a reasonable expectation of this being a net benefit in the long run."

Then I really don't know of anyone who's knowingly been faced with an exception. There are times when you try to convince yourself "I'm not hiding any Jews in my basement" before you talk to the Gestapo officer. But then you do still know the truth, you're just trying to create something like an alternative self that exists in your imagination, a facade to talk to the Gestapo officer.

But to really believe something that isn't true? I don't know if there was ever anyone for whom that was knowably a good idea. I'm sure that there have been many many times in human history, where person X was better off with false belief Y. And by the same token, there is always some set of winning lottery numbers in every drawing. It's knowing which lottery ticket will win that is the epistemically difficult part, like X knowing when he's better off with a false belief.

Self-deceptions are the worst kind of black swan bets, much worse than lies, because without knowing the true state of affairs, you can't even guess at what the penalty will be for your self-deception. They only have to blow up once to undo all the good they ever did. One single time when you pray to God after discovering a lump, instead of going to a doctor. That's all it takes to undo a life. All the happiness that the warm thought of an afterlife ever produced in humanity, has now been more than cancelled by the failure of humanity to institute systematic cryonic preservations after liquid nitrogen became cheap to manufacture. And I don't think that anyone ever had that sort of failure in mind as a possible blowup, when they said, "But we need religious beliefs to cushion the fear of death." That's what black swan bets are all about - the unexpected blowup.

Maybe you even get away with one or two black-swan bets - they don't get you every time. So you do it again, and then the blowup comes and cancels out every benefit and then some. That's what black swan bets are all about.

Thus the difficulty of knowing when it's safe to believe a lie (assuming you can even manage that much mental contortion in the first place) - part of the nature of black swan bets is that you don't see the bullet that kills you; and since our perceptions just seem like the way the world is, it looks like there is no bullet, period.

So I would say that there is an ethical injunction against self-deception. I call this an "ethical injunction" not so much because it's a matter of interpersonal morality (although it is), but because it's a rule that guards you from your own cleverness - an override against the temptation to do what seems like the right thing.

So now we have two kinds of situation that can support an "ethical injunction", a rule not to do something even when it's the right thing to do. (That is, you refrain "even when your brain has computed it's the right thing to do", but this will just seem like "the right thing to do".)

First, being human and running on corrupted hardware, we may generalize classes of situation where when you say e.g. "It's time to rob a few banks for the greater good," we deem it more likely that you've been corrupted than that this is really the case. (Note that we're not prohibiting it from ever being the case in reality, but we're questioning the epistemic state where you're justified in trusting your own calculation that this is the right thing to do - fair lottery tickets can win, but you can't justifiably buy them.)

Second, history may teach us that certain classes of action are black-swan bets, that is, they sometimes blow up bigtime for reasons not in the decider's model. So even when we calculate within the model that something seems like the right thing to do, we apply the further knowledge of the black swan problem to arrive at an injunction against it.

But surely... if one is aware of these reasons... then one can simply redo the calculation, taking them into account. So we can rob banks if it seems like the right thing to do after taking into account the problem of corrupted hardware and black swan blowups. That's the rational course, right?

There's a number of replies I could give to that.

I'll start by saying that this is a prime example of the sort of thinking I have in mind, when I warn aspiring rationalists to beware of cleverness.

I'll also note that I wouldn't want an attempted Friendly AI that had just decided that the Earth ought to be transformed into paperclips, to assess whether this was a reasonable thing to do in light of all the various warnings it had received against it. I would want it to undergo an automatic controlled shutdown. Who says that meta-reasoning is immune from corruption?

I could mention the important times that my naive, idealistic ethical inhibitions have protected me from myself, and placed me in a recoverable position, or helped start the recovery, from very deep mistakes I had no clue I was making. And I could ask whether I've really advanced so much, and whether it would really be all that wise, to remove the protections that saved me before.

Yet even so... "Am I still dumber than my ethics?" is a question whose answer isn't automatically "Yes."

There are obvious silly things here that you shouldn't do; for example, you shouldn't wait until you're really tempted, and then try to figure out if you're smarter than your ethics on that particular occasion.

But in general - there's only so much power that can vest in what your parents told you not to do. One shouldn't underestimate the power. Smart people debated historical lessons in the course of forging the Enlightenment ethics that much of Western culture draws upon; and some subcultures, like scientific academia, or science-fiction fandom, draw on those ethics more directly. But even so the power of the past is bounded.

And in fact...

I've had to make my ethics much stricter than what my parents and Jerry Pournelle and Richard Feynman told me not to do.

Funny thing, how when people seem to think they're smarter than their ethics, they argue for less strictness rather than more strictness. I mean, when you think about how much more complicated the modern world is...

And along the same lines, the ones who come to me and say, "You should lie about the Singularity, because that way you can get more people to support you; it's the rational thing to do, for the greater good" - these ones seem to have no idea of the risks.

They don't mention the problem of running on corrupted hardware. They don't mention the idea that lies have to be recursively protected from all the truths and all the truthfinding techniques that threaten them. They don't mention that honest ways have a simplicity that dishonest ways often lack. They don't talk about black-swan bets. They don't talk about the terrible nakedness of discarding the last defense you have against yourself, and trying to survive on raw calculation.

I am reasonably sure that this is because they have no clue about any of these things.

If you've truly understood the reason and the rhythm behind ethics, then one major sign is that, augmented by this newfound knowledge, you don't do those things that previously seemed like ethical transgressions. Only now you know why.

Someone who just looks at one or two reasons behind ethics, and says, "Okay, I've understood that, so now I'll take it into account consciously, and therefore I have no more need of ethical inhibitions" - this one is behaving more like a stereotype than a real rationalist. The world isn't simple and pure and clean, so you can't just take the ethics you were raised with and trust them. But that pretense of Vulcan logic, where you think you're just going to compute everything correctly once you've got one or two abstract insights - that doesn't work in real life either.

As for those who, having figured out none of this, think themselves smarter than their ethics: Ha.

And as for those who previously thought themselves smarter than their ethics, but who hadn't conceived of all these elements behind ethical injunctions "in so many words" until they ran across this Overcoming Bias sequence, and who now think themselves smarter than their ethics, because they're going to take all this into account from now on: Double ha.

I have seen many people struggling to excuse themselves from their ethics. Always the modification is toward lenience, never to be more strict. And I am stunned by the speed and the lightness with which they strive to abandon their protections. Hobbes said, "I don't know what's worse, the fact that everyone's got a price, or the fact that their price is so low." So very low the price, so very eager they are to be bought. They don't look twice and then a third time for alternatives, before deciding that they have no option left but to transgress - though they may look very grave and solemn when they say it. They abandon their ethics at the very first opportunity. "Where there's a will to failure, obstacles can be found." The will to fail at ethics seems very strong, in some people.

I don't know if I can endorse absolute ethical injunctions that bind over all possible epistemic states of a human brain. The universe isn't kind enough for me to trust that. (Though an ethical injunction against self-deception, for example, does seem to me to have tremendous force. I've seen many people arguing for the Dark Side, and none of them seem aware of the network risks or the black-swan risks of self-deception.) If, someday, I attempt to shape a (reflectively consistent) injunction within a self-modifying AI, it will only be after working out the math, because that is so totally not the sort of thing you could get away with doing via an ad-hoc patch.

But I will say this much:

I am completely unimpressed with the knowledge, the reasoning, and the overall level, of those folk who have eagerly come to me, and said in grave tones, "It's rational to do unethical thing X because it will have benefit Y."

Referenced by: Prices or Bindings? • Ethics Notes

Original with comments: Ethical Injunctions

Prices or Bindings?

Eliezer Yudkowsky, 21 October 2008 04:00PM

Followup to: Ethical Injunctions

During World War II, Knut Haukelid and three other saboteurs sank a civilian Norwegian ferry ship, the SF Hydro, carrying a shipment of deuterium for use as a neutron moderator in Germany's atomic weapons program. Eighteen dead, twenty-nine survivors. And that was the end of the Nazi nuclear program. Can you imagine a Hollywood movie in which the hero did that, instead of coming up with some amazing clever way to save the civilians on the ship?

Stephen Dubner and Steven Levitt published the work of an anonymous economist turned bagelseller, Paul F., who dropped off baskets of bagels and came back to collect money from a cashbox, and also collected statistics on payment rates. The current average payment rate is 89%. Paul F. found that people on the executive floor of a company steal more bagels; that people with security clearances don't steal any fewer bagels; that telecom companies have robbed him and that law firms aren't worth the trouble.

Hobbes (of Calvin and Hobbes) once said: "I don't know what's worse, the fact that everyone's got a price, or the fact that their price is so low."

If Knut Haukelid sold his soul, he held out for a damned high price - the end of the Nazi atomic weapons program.

Others value their integrity less than a bagel.

One suspects that Haukelid's price was far higher than most people would charge, if you told them to never sell out. Maybe we should stop telling people they should never let themselves be bought, and focus on raising their price to something higher than a bagel?

But I really don't know if that's enough.

The German philosopher Fichte once said, "I would not break my word even to save humanity."

Raymond Smullyan, in whose book I read this quote, seemed to laugh and not take Fichte seriously.

Abraham Heschel said of Fichte, "His salvation and righteousness were apparently so much more important to him than the fate of all men that he would have destroyed mankind to save himself."

I don't think they get it.

If a serial killer comes to a confessional, and confesses that he's killed six people and plans to kill more, should the priest turn him in? I would answer, "No." If not for the seal of the confessional, the serial killer would never have come to the priest in the first place. All else being equal, I would prefer the world in which the serial killer talks to the priest, and the priest gets a chance to try and talk the serial killer out of it.

I use the example of a priest, rather than a psychiatrist, because a psychiatrist might be tempted to break confidentiality "just this once", and the serial killer knows that. But a Catholic priest who broke the seal of the confessional - for any reason - would face universal condemnation from his own church. No Catholic would be tempted to say, "Well, it's all right because it was a serial killer."

I approve of this custom and its absoluteness, and I wish we had a rationalist equivalent.

The trick would be establishing something of equivalent strength to a Catholic priest who believes God doesn't want him to break the seal, rather than the lesser strength of a psychiatrist who outsources their tape transcriptions to Pakistan. Otherwise serial killers will, quite sensibly, use the Catholic priests instead, and get less rational advice.

Suppose someone comes to a rationalist Confessor and says: "You know, tomorrow I'm planning to wipe out the human species using this neat biotech concoction I cooked up in my lab." What then? Should you break the seal of the confessional to save humanity?

It appears obvious to me that the issues here are just those of the one-shot Prisoner's Dilemma, and I do not consider it obvious that you should defect on the one-shot PD if the other player cooperates in advance on the expectation that you will cooperate as well.

There are issues with trustworthiness and how the sinner can trust the rationalist's commitment. It is not enough to be trustworthy; you must appear so. But anything that mocks the appearance of trustworthiness, while being unbound from its substance, is a poor signal; the sinner can follow that logic as well. Perhaps once neuroimaging is a bit more advanced, we could have the rationalist swear under a truthtelling machine that they would not break the seal of the confessional even to save humanity.

There's a proverb I failed to Google, which runs something like, "Once someone is known to be a liar, you might as well listen to the whistling of the wind." You wouldn't want others to expect you to lie, if you have something important to say to them; and this issue cannot be wholly decoupled from the issue of whether you actually tell the truth. If you'll lie when the fate of the world is at stake, and others can guess that fact about you, then, at the moment when the fate of the world is at stake, that's the moment when your words become the whistling of the wind.

I don't know if Fichte meant it that way, but his statement makes perfect sense as an ethical thesis to me. It's not that one person's personal integrity is worth more, as terminal valuta, than the entire world. Rather, losing all your ethics is not a pure advantage.

Being believed to tell the truth has advantages, and I don't think it's so easy to decouple that from telling the truth. Being believed to keep your word has advantages; and if you're the sort of person who would in fact break your word to save humanity, the other may guess that too. Even intrapersonal ethics can help protect you from black swans and fundamental mistakes. That logic doesn't change its structure when you double the value of the stakes, or even raise them to the level of a world. Losing your ethics is not like shrugging off some chains that were cool to look at, but were weighing you down in an athletic contest.

This I knew from the beginning: That if I had no ethics I would hold to even with the world at stake, I had no ethics at all. And I could guess how that would turn out.

Referenced by: Ethics Notes

Original with comments: Prices or Bindings?

Ethics Notes

Eliezer Yudkowsky, 21 October 2008 09:57PM

Followup to: Ethical Inhibitions, Ethical Injunctions, Prices or Bindings?

(Some collected replies to comments on the above three posts.)

From Ethical Inhibitions:

Spambot: Every major democratic political leader lies abundantly to obtain office, as it's a necessity to actually persuade the voters. So Bill Clinton, Jean Chretien, Winston Churchill should qualify for at least half of your list of villainy.

Have the ones who've lied more, done better?

In cases where the politician who told more lies won, has that politician gone on to rule well in an absolute sense?

Is it actually true that no one who refused to lie (and this is not the same as always telling the whole truth) could win political office?

Are the lies expected, and in that sense, less than true betrayals of someone who trusts you?

Are there understood Rules of Politics that include lies but not assassinations, which the good politicians abide by, so that they are not really violating the ethics of their tribe?

Will the world be so much worse off if sufficiently good people refuse to tell outright lies and are thereby barred from public office; or would we thereby lose a George Washington or Marcus Aurelius or two, and thereby darken history?

Pearson: American revolutionaries as well ended human lives for the greater good

Police must sometimes kill the guilty. Soldiers must sometimes kill civilians (or if the enemy knows you're reluctant, that gives them a motive to use civilians as a shield). Spies sometimes have legitimate cause to kill people who helped them, but this has probably been done far more often than it has been justified by a need to end the Nazi nuclear program.

I think it's worth noting that in all such cases, you can write out something like a code of ethics and at least try to have social acceptance of it. Politicians, who lie, may prefer not to discuss the whole thing, but politicians are only a small slice of society.

Are there many who transgress even the unwritten rules and end up really implementing the greater good? (And no, there's no unwritten rule that says you can rob a bank to stop global warming.)

...but if you're placing yourself under unusual stress, you may need to be stricter than what society will accept from you. In fact, I think it's fair to say that the further I push any art, such as rationality or AI theory, the more I perceive that what society will let you get away with is tremendously too sloppy a standard.

Yvain: There are all sorts of biases that would make us less likely to believe people who "break the rules" can ever turn out well. One is the halo effect. Another is availability bias - it's much easier to remember people like Mao than it is to remember the people who were quiet and responsible once their revolution was over, and no one notices the genocides that didn't happen because of some coup or assassination.

When the winners do something bad, it's never interpreted as bad after the fact. Firebombing a city to end a war more quickly, taxing a populace to give health care to the less fortunate, intervening in a foreign country's affairs to stop a genocide: they're all likely to be interpreted as evidence for "the ends don't justify the means" when they fail, but glossed over or treated as common sense interventions when they work.

Both fair points. One of the difficult things in reasoning about ethics is the extent to which we can expect historical data to be distorted by moral self-deception on top of the more standard fogs of history.

Morrison: I'm not sure you aren't "making too much stew from one oyster". I certainly feel a whole lot less ethically inhibited if I'm really, really certain I'm not going to be punished. When I override, it feels very deliberate - "system two" grappling and struggling with "system one"'s casual amorality, and with a significant chance of the override attempt failing.

Weeks: This entire post is kind of surreal to me, as I'm pretty confident I've never felt the emotion described here before... I don't remember ever wanting to do something that I both felt would be wrong and wouldn't have consequences otherwise.

I don't know whether to attribute this to genetic variance, environmental variance, misunderstanding, or a small number of genuine sociopaths among Overcoming Bias readers. Maybe Weeks is referring to "not wanting" in terms of not finally deciding to do something he felt was wrong, rather than not being tempted?

From Ethical Injunctions:

Psy-Kosh: Given the current sequence, perhaps it's time to revisit the whole Torture vs Dust Specks thing?

I can think of two positions on torture to which I am sympathetic:

Strategy 1: No legal system or society should ever refrain from prosecuting those who torture. Anything important enough that torture would even be on the table, like the standard nuclear bomb in New York, is important enough that everyone involved should be willing to go to prison for the crime of torture.

Strategy 2: The chance of actually encountering a "nuke in New York" situation, that can be effectively resolved by torture, is so low, and the knock-on effects of having the policy in place so awful, that a blanket injunction against torture makes sense.

In case 1, you would choose TORTURE over SPECKS, and then go to jail for it, even though it was the right thing to do.

In case 2, you would say "TORTURE over SPECKS is the right alternative of the two, but a human can never be in an epistemic state where you have justified belief that this is the case". Which would tie in well to the Hansonian argument that you have an O(3^^^3) probability penalty from the unlikelihood of finding yourself in such a unique position.

So I am sympathetic to the argument that people should never torture, or that a human can't actually get into the epistemic state of a TORTURE vs. SPECKS decision.

But I can't back the position that SPECKS over TORTURE is inherently the right thing to do, which I did think was the issue at hand. This seems to me to mix up an epistemic precaution with morality.

There's certainly worse things than torturing one person - torturing two people, for example. But if you adopt position 2, then you would refuse to torture one person with your own hands even to save a thousand people from torture, while simultaneously saying that that it is better for one person to be tortured at your own hands than for a thousand people to be tortured at someone else's.

I try to use the words "morality" and "ethics" consistently as follows: The moral questions are over the territory (or, hopefully equivalently, over epistemic states of absolute certainty). The ethical questions are over epistemic states that humans are likely to be in. Moral questions are terminal. Ethical questions are instrumental.

Hanson: The problem here of course is how selective to be about rules to let into this protected level of "rules almost no one should think themselves clever enough to know when to violate." After all, your social training may well want you to include "Never question our noble leader" in that set. Many a Christian has been told the mysteries of God are so subtle that they shouldn't think themselves clever enough to know when they've found evidence that God isn't following a grand plan to make this the best of all possible worlds.

Some of the flaws in Christian theology lie in what they think their supposed facts would imply: e.g., that because God did miracles you can know that God is good. Other problems come more from the falsity of the premises than the invalidity of the deductions. Which is to say, if God did exist and were good, then you would be justified in being cautious around stomping on parts of God's plan that didn't seem to make sense at the moment. But this epistemic state would best be arrived at via a long history of people saying, "Look how stupid God's plan is, we need to do X" and then X blowing up on them. Rather than, as is actually the case, people saying "God's plan is X" and then X blows up on them.

Or if you'd found with some historical regularity that, when you challenged the verdict of the black box, that you seemed to be right 90% of the time, but the other 10% of the time you got black-swan blowups that caused a hundred times as much damage, that would also be cause for hesitation - albeit it doesn't quite seem like grounds for suspecting a divine plan.

Nominull: S o... do you not actually believe in your injunction to "shut up and multiply"? Because for some time now you seem to have been arguing that we should do what feels right rather than trying to figure out what is right.

Certainly I'm not saying "just do what feels right". There's no safe defense, not even ethical injunctions. There's also no safe defense, not even "shut up and multiply".

I probably should have been clearer about this before, but I was trying to discuss things in order, and didn't want to wade into ethics without specialized posts...

People often object to the sort of scenarios that illustrate "shut up and multiply" by saying, "But if the experimenter tells you X, what if they might be lying?"

Well, in a lot of real-world cases, then yes, there are various probability updates you perform based on other people being willing to make bets against you; and just because you get certain experimental instructions doesn't imply the real world is that way.

But the base case has to be moral comparisons between worlds, or comparisons of expected utility between given probability distributions. If you can't ask about the base case, then what good can you get from instrumental ethics built on top?

Let's be very clear that I don't think that one small act of self-deception is an inherently morally worse event than, say, getting a hand chopped off. I'm asking, rather, how one should best avoid the dismembering chainsaw; and I am arguing that in reasonable states of knowledge a human can attain, the answer is, "Don't deceive yourself, it's a black-swan bet at best." Furthermore, that in the vast majority of cases where I have seen people conclude otherwise, it has indicated messed-up reasoning more than any actual advantage.

Vassar: For such a reason, I would be very wary of using such rules in an AGI, but of course, perhaps the actual mathematical formulation of the rule in question within the AGI would be less problematic, though a few seconds of thought doesn't give me much reason to think this.

Are we still talking about self-deception? Because I would give odds around as extreme as the odds I would give of anything, that if you tell me "the AI you built is trying to deceive itself", it indicates that some kind of really epic error has occurred. Controlled shutdown, immediately.

Vassar: In a very general sense though, I see a logical problem with this whole line of thought. How can any of these injunctions survive except as self-protecting beliefs? Isn't this whole approach just the sort of "fighting bias with bias" that you and Robin usually argue against?

Maybe I'm not being clear about how this would work in an AI!

The ethical injunction isn't self-protecting, it's supported within the structural framework of the underlying system. You might even find ethical injunctions starting to emerge without programmer intervention, in some cases, depending on how well the AI understood its own situation.

But the kind of injunctions I have in mind wouldn't be reflective - they wouldn't modify the utility function, or kick in at the reflective level to ensure their own propagation. That sounds really scary, to me - there ought to be an injunction against it!

You might have a rule that would controlledly shut down the (non-mature) AI if it tried to execute a certain kind of source code change, but that wouldn't be the same as having an injunction that exerts direct control over the source code to propagate itself.

To the extent the injunction sticks around in the AI, it should be as the result of ordinary reasoning, not reasoning taking the injunction into account! That would be the wrong kind of circularity; you can unwind past ethical unjunctions!

My ethical injunctions do not come with an extra clause that says, "Do not reconsider this injunction, including not reconsidering this clause." That would be going way too far. If anything, you ought to have an injunction against that kind of circularity (since it seems like a plausible failure mode in which the system has been parasitized by its own content).

You should never, ever murder an innocent person who's helped you, even if it's the right thing to do

Shut up and do the impossible!

Ord: As written, both these statements are conceptually confused. I understand that you didn't actually mean either of them literally, but I would advise against trading on such deep-sounding conceptual confusions.

I can't weaken them and make them come out as the right advice.

Even after "Shut up and do the impossible", there was that commenter who posted on their failed attempt at the AI-Box Experiment by saying that they thought they gave it a good try - which shows how hard it is to convey the sentiment of "Shut up and do the impossible!"

Readers can work out on their own how to distinguish the map and the territory, I hope. But if you say "Shut up and do what seems impossible!", then that, to me, sounds like dispelling part of the essential message - that what seems impossible doesn't look like it "seems impossible", it just looks impossible.

Likewise with "things you shouldn't do even if they're the right thing to do". Only the paradoxical phrasing, which is obviously not meant to be taken literally, conveys the danger and tension of ethics - the genuine opportunities you might be passing up - and for that matter, how dangerously meta the whole line of argument is.

"Don't do it, even if it seems right" sounds merely clever by comparison - like you're going to reliably divine the difference between what seems right and what is right, and happily ride off into the sunset.

Crowe: This seems closely related to inside-view versus outside-view. The think-lobe of the brain comes up with a cunning plan. The plan breaks an ethical rule but calculation shows it is for the greater good. The executive-lobe of the brain then ponders the outside view. Every-one who has executed an evil cunning plan has run a calculation of the greater good and had their plan endorsed. So the calculation lack outside-view credibility.

Yes, inside view versus outside view is definitely part of this. And the planning fallacy, optimism, and overconfidence, too.

But there are also biases arguing against the same line of reasoning, as noted by Yvain: History may be written by the victors to emphasize the transgressions of the losers while overlooking the moral compromises of those who achieved "good" results, etc.

Also, some people who execute evil cunning plans may just have evil intent - possibly also with outright lies about their intentions. In which case, they really wouldn't be in the reference class of well-meaning revolutionaries, albeit you would have to worry about your comrades; the Trotsky->Lenin->Stalin slide.

Kurz: What's to prohibit the meta-reasoning from taking place before the shutdown triggers? It would seem that either you can hard-code an ethical inhibition or you can't. Along those lines, is it fair to presume that the inhibitions are always negative, so that non-action is the safe alternative? Why not just revert to a known state?

If a self-modifying AI with the right structure will write ethical injunctions at all, it will also inspect the code to guarantee that no race condition exists with any deliberative-level supervisory systems that might have gone wrong in the condition where the code executes. Otherwise you might as well not have the code.

Inaction isn't safe but it's safer than running an AI whose moral system has gone awry.

Finney: Which is better: conscious self-deception (assuming that's even meaningful), or unconscious?

Once you deliberately choose self-deception, you may have to protect it by adopting other Dark Side Epistemology. I would, of course, say "neither" (as otherwise I would be swapping to the Dark Side) but if you ask me which is worse - well, hell, even I'm still undoubtedly unconsciously self-deceiving, but that's not the same as going over to the Dark Side by allowing it!

From Prices or Bindings?:

Psy-Kosh: Hrm. I'd think "avoid destroying the world" itself to be an ethical injunction too.

The problem is that this is phrased as an injunction over positive consequences. Deontology does better when it's closer to the action level and negative rather than positive.

Imagine trying to give this injunction to an AI. Then it would have to do anything that it thought would prevent the destruction of the world, without other considerations. Doesn't sound like a good idea.

Crossman: Eliezer, can you be explicit which argument you're making? I thought you were a utilitarian, but you've been sounding a bit Kantian lately.

If all I want is money, then I will one-box on Newcomb's Problem.

I don't think that's quite the same as being a Kantian, but it does reflect the idea that similar decision algorithms in similar epistemic states will tend to produce similar outputs, and that such decision systems should not pretend to the logical impossibility of local optimization. But this is a deep subject on which I have yet to write up my full views.

Clay: Put more seriously, I would think that being believed to put the welfare of humanity ahead of concerns about personal integrity could have significant advantages itself.

The whole point here is that "personal integrity" doesn't have to be about being a virtuous person. It can be about trying to save the world without any concern for your own virtue. It can be the sort of thing you'd want a pure nonsentient decision agent to do, something that was purely a means and not at all an end in itself.

Andrix: There seems to be a conflict here between not lying to yourself, and holding a traditional rule that suggests you ignore your rationality.

Your rationality is the sum of your full abilities, including your wisdom about what you refrain from doing in the presence of what seem like good reasons.

Yvain: I am glad Stanislav Petrov, contemplating his military oath to always obey his superiors and the appropriate guidelines, never read this post.

An interesting point, for several reasons.

First, did Petrov actually swear such an oath, and would it apply in such fashion as to require him to follow the written policy rather than using his own military judgment?

Second, you might argue that Petrov's oath wasn't intended to cover circumstances involving the end of the world, and that a common-sense exemption should apply when the stakes suddenly get raised hugely beyond the intended context of the original oath. I think this fails, because Petrov was regularly in charge of a nuclear-war installation and so this was exactly the sort of event his oath would be expected to apply to.

Third, the Soviets arguably implemented what I called Strategy 1 above: Petrov did the right thing, and was censured for it anyway.

Fourth - maybe, on sober reflection, we wouldn't have wanted the Soviets to act differently! Yes, the written policy was stupid. And the Soviet Union was undoubtedly censuring Petrov out of bureaucratic coverup, not for reasons of principle. But do you want the Soviet Union to have a written, explicit policy that says, "Anyone can ignore orders in a nuclear war scenario if they think it's a good idea," or even an explicit policy that says "Anyone who ignores orders in a nuclear war scenario, who is later vindicated by events, will be rewarded and promoted"?

Original with comments: Ethics Notes

Which Parts Are "Me"?

Eliezer Yudkowsky, 22 October 2008 06:15PM

Everything I am, is surely my brain; but I don't accept everything my brain does, as "me".

Though I don't like the fact, it does seem in my case to be true, that reflecting upon a passion can diminish it. It does seem that in taking a step back from myself to look at what my brain is doing, that this introduces a greater emotional distance from the world, than when I am not reflecting. Though I am extremely cautious when it comes to the moral issue of how to modify human nature, this is one thing that I would change, I think.

The point at which you start thinking that not everything that goes on your brain is to be identified with "you", is something of a Rubicon, I fear. And those who prize the intensity of a simple passionate existence, above success in life or a complex layered personality, should not cross that Rubicon.

(Though in this case you should accept advice from rationalists about practical policies for guarding yourself, because this is a dangerous world in which to be unreflective. For example: if you don't follow the link to the rest of this post, then sign the #$!@ up for cryonics regardless of what your simple passions say on the subject.)

Still here? Then here is the Rubicon of which I speak:

(1) Never seeing ex-girlfriend-X again is so terribly unbearably sad!
->
(2) Why am I so terribly sad?
->
(3) Damn, my stupid brain is still obsessing about this when I just want to get over it.

The first scenario is what I would call the intensity of simple passion; nothing between the world and the emotions. The sentence contains no "I" to get in the way. There is nothing to think about except the world itself, the sadness and the missing person.

In the second scenario it is not the world that is sad, but some particular person, an "I"; and the sadness of this "I" is something that can be called into question.

And in the third scenario, the borders of "I" have been drawn in a way that doesn't include everything in the brain, so that "I" is the victim of the brain, struggling against it. And this is not paradoxical. Everything that I am, has to be in my brain somewhere, because there is nowhere else for it to be. But that doesn't mean I have to identify with everything that my brain does. Just as I draw the border of "me" to include my brain but exclude my computer's CPU - which is still a sensible decision at least for now - I can define the border of myself to exclude certain events that happen in my brain, which I do not control, do not want to happen, and do not agree with.

That time I faced down the power-corrupts circuitry, I thought, "my brain is dumping this huge dose of unwanted positive reinforcement", and I sat there waiting for the surge to go away and trying not to let it affect anything.

Thinking "I am being tempted" wouldn't have quite described it, since the deliberate process that I usually think of as "me" - the little voice inside my own head - was not even close to being swayed by the attempted dose of reward that neural circuit was dumping. I wasn't tempted by power; I'd already made my decision, and the question was enforcing it.

But a dangerous state of mind indeed it would have been, to think "How tempting!" without an "I" to be tempted. From there it would only be a short step to thinking "How wonderful it is to exercise power!" This, so far as I can guess, is what the brain circuit is supposed to do to a human.

So it was a fine thing that I was reflective, on this particular occasion.

The problem is when I find myself getting in the way of even the parts I call "me". The joy of helping someone, or for that matter, the sadness of death - these emotions that I judge right and proper, which must be me if anything is me - I don't want those feelings diminished.

And I do better at this, now that my metaethics are straightened out, and I know that I have no specific grounds left for doubting my feelings.

But I still suspect that there's a little distance there, that wouldn't be there otherwise, and I wish my brain would stop doing that.

I have always been inside and outside myself, for as long as I can remember. To my memory, I have always been reflective. But I have witnessed the growth of others, and in at least one case I've taken someone across that Rubicon. The one now possesses a more complex and layered personality - seems more to me now like a real person, even - but also a greater emotional distance. Life's lows have been smoothed out, but also the highs. That's a sad tradeoff and I wish it didn't exist.

I don't want to have to choose between sanity and passion. I don't want to smooth out life's highs or even life's lows, if those highs and lows make sense. I wish to feel the emotion appropriate to the event. If death is horrible then I should fight death, not fight my own grief.

But if I am forced to choose, I will choose stability and deliberation, for the sake of what I protect. And my personality does reflect that. What you are willing to trade off, will sometimes get traded away - a dire warning in full generality.

This post is filed under "morality" because the question "Which parts of my brain are 'me'?" is a moral question - it's not predicted so much as chosen. You can't perform a test on neural tissue to find whether it's in or out. You have to accept or reject any particular part, based on what you think humans in general, and yourself particularly, ought to be.

The technique does have its advantages: It brings greater stability, being less subject to sudden changes of mind in the winds of momentary passion. I was unsettled the first time I met an unreflective person because they changed so fast, seemingly without anchors. Reflection conveys a visibly greater depth and complexity of personality, and opens a realm of thought that otherwise would not exist. It makes you more moral (at least in my experience and observation) because it gives you the opportunity to make moral choices about things that would otherwise be taken for granted, or decided for you by your brain. Waking up to reflection is like the difference between being an entirely helpless prisoner and victim of yourself, versus becoming aware of the prison and getting a chance to escape it sometimes. Not that you are departing your brain entirely, but the you that is the little voice in your own head may get a chance to fight back against some circuits that it doesn't want to be influenced by.

And the technique's use, to awaken the unreflective, is as I have described: First you must cross the gap between events-in-the-world just being terribly sad or terribly important or whatever, of themselves; and say, "I feel X". Then you must begin to judge the feeling, saying, "I do not want to feel this - I feel this way, but I wish I didn't." Justifying yourself with "This is not what a human should be", or "the emotion does not seem appropriate to the event".

And finally there is the Rubicon of "I wish my brain wouldn't do this", at which point you are thinking as if the feeling comes from outside the inner you, imposed upon you by your brain. (Which does not say that you are something other than your brain, but which does say that not every brain event will be accepted by you as you.)

After crossing this Rubicon you have set your feet fully upon the reflective Way; and I've yet to hear of anyone turning back successfully, though I think some have tried, or wished they could.

And once your feet are set on walking down that path, there is nothing left but to follow it forward, and try not to be emotionally distanced from the parts of yourself that you accept as you - an effort that a mind of simple passion would not need to make in the first place. And an effort which can easily backfire by drawing your attention to the layered depths of your selfhood, away from the event and the emotion.

Somewhere at the end of this, I think, is a mastery of techniques that are Zenlike but not Zen, so that you have full passion in the parts of yourself that you identify with, and distance from the pieces of your brain that you reject; and a complex layered personality with a stable inner core, without smoothing out those highs or lows of life that you accept as appropriate to the event.

And if not, then screw it, let's hack the brain so that it works that way. I have no confidence in my ability to judge how human nature should change, and would sooner leave it up to a more powerful mind in the same metamoral reference frame. But if I had to guess, I think that's the right thing to do.

Referenced by: Inner Goodness • Wise Pretensions v.0

Original with comments: Which Parts Are "Me"?

Inner Goodness

Eliezer Yudkowsky, 23 October 2008 10:19PM

Followup to: Which Parts Are "Me"?, Effortless Technique

A recent conversation with Michael Vassar touched on - or to be more accurate, he patiently explained to me - the psychology of at least three (3) different types of people known to him, who are evil and think of themselves as "evil". In ascending order of frequency:

The first type was someone who, having concluded that God does not exist, concludes that one should do all the things that God is said to dislike. (Apparently such folk actually exist.)

The third type was someone who thinks of "morality" only as a burden - all the things your parents say you can't do - and who rebels by deliberately doing those things.

The second type was a whole 'nother story, so I'm skipping it for now.

This reminded me of a topic I needed to post on:

Beware of placing goodness outside.

This specializes to e.g. my belief that ethicists should be inside rather than outside a profession: that it is futile to have "bioethicists" not working in biotech, or futile to think you can study Friendly AI without needing to think technically about AI.

But the deeper sense of "not placing goodness outside" was something I first learned at age ~15 from the celebrity logician Raymond Smullyan, in his book The Tao Is Silent, my first introduction to (heavily Westernized) Eastern thought.

Michael Vassar doesn't like this book. Maybe because most of the statements in it are patently false?

But The Tao Is Silent still has a warm place reserved in my heart, for it was here that I first encountered such ideas as:

Do you think of altruism as sacrificing one's own happiness for the sake of others, or as gaining one's happiness through the happiness of others?

(I would respond, by the way, that an "altruist" is someone who chooses between actions according to the criterion of others' welfare.)

A key chapter in The Tao Is Silent can be found online: "Taoism versus Morality". This chapter is medium-long (say, 3-4 Eliezer OB posts) but it should convey what I mean, when I say that this book manages to be quite charming, even though most of the statements in it are false.

Here is one key passage:

TAOIST: I think the word "humane" is central to our entire problem. You are pushing morality. I am encouraging humanity. You are emphasizing "right and wrong," I am emphasizing the value of natural love. I do not assert that it is logically impossible for a person to be both moralistic and humane, but I have yet to meet one who is! I don't believe in fact that there are any. My whole life experience has clearly shown me that the two are inversely related to an extraordinary degree. I have never yet met a moralist who is a really kind person. I have never met a truly kind and humane person who is a moralist. And no wonder! Morality and humaneness are completely antithetical in spirit.

MORALIST: I'm not sure that I really understand your use of the word "humane," and above all, I am totally puzzled as to why you should regard it as antithetical to morality.

TAOIST: A humane person is one who is simply kind, sympathetic, and loving. He does not believe that he SHOULD be so, or that it is his "duty" to be so; he just simply is. He treats his neighbor well not because it is the "right thing to do," but because he feels like it. He feels like it out of sympathy or empathy--out of simple human feeling. So if a person is humane, what does he need morality for? Why should a person be told that he should do something which he wants to do anyway?

MORALIST: Oh, I see what you're talking about; you're talking about saints! Of course, in a world full of saints, moralists would no longer be needed--any more than doctors would be needed in a world full of healthy people. But the unfortunate reality is that the world is not full of saints. Of everybody were what you call "humane," things would be fine. But most people are fundamentally not so nice. They don't love their neighbor; at the first opporunity they will explot their neighbor for their own selfish ends. That's why we moralists are necessary to keep them in check.

TAOIST: To keep them in check! How perfectly said! And do you succeed in keeping them in check?

MORALIST: I don't say that we always succeed, but we try our best. After all, you can't blame a doctor for failing to keep a plague in check if he conscientiously does everything he can. We moralists are not gods, and we cannot guarantee our efforts will succeed. All we can do is tell people they SHOULD be more humane, we can't force them to. After all, people have free wills.

TAOIST: And it has never once occurred to you that what in fact you are doing is making people less humane rather than more humane?

MORALIST: Of course not, what a horrible thing to say! Don't we explicitly tell people that they should be MORE humane?

TAOIST: Exactly! And that is precisely the trouble. What makes you think that telling one that one should be humane or that it is one's "duty" to be humane is likely to influence one to be more humane? It seems to me, it would tend to have the opposite effect. What you are trying to do is to command love. And love, like a precious flower, will only wither at any attempt to force it. My whole criticism of you is to the effect that you are trying to force that which can thrive only if it is not forced. That's what I mean when I say that you moralists are creating the very problems about which you complain.

MORALIST: No, no, you don't understand! I am not commanding people to love each other. I know as well as you do that love cannot be commanded. I realize it would be a beautiful world if everyone loved one another so much that morality would not be necessary at all, but the hard facts of life are that we don't live in such a world. Therefore morality is necessary. But I am not commanding one to love one's neighbor--I know that is impossible. What I command is: even though you don't love your neighbor all that much, it is your duty to treat him right anyhow. I am a realist.

TAOIST: And I say you are not a realist. I say that right treatment or fairness or truthfulness or duty or obligation can no more be successfully commanded than love.

Or as Lao-Tse said: "Give up all this advertising of goodness and duty, and people will regain love of their fellows."

As an empirical proposition, the idea that human nature begins as pure sweetness and light and is then tainted by the environment, is flat wrong. I don't believe that a world in which morality was never spoken of, would overflow with kindness.

But it is often much easier to point out where someone else is wrong, than to be right yourself. Smullyan's criticism of Western morality - especially Christian morality, which he focuses on - does hit the mark, I think.

It is very common to find a view of morality as something external, a burden of duty, a threat of punishment, an inconvenient thing that constrains you against your own desires; something from outside.

Though I don't recall the bibliography off the top of my head, there's been more than one study demonstrating that children who are told to, say, avoid playing with a car, and offered a cookie if they refrain, will go ahead and play with the car when they think no one is watching, or if no cookie is offered. If no reward or punishment is offered, and the child is simply told not to play with the car, the child will refrain even if no adult is around. So much for the positive influence of "God is watching you" on morals. I don't know if any direct studies have been done on the question; but extrapolating from existing knowledge, you would expect childhood religious belief to interfere with the process of internalizing morality. (If there were actually a God, you wouldn't want to tell the kids about it until they'd grown up, considering how human nature seems to work in the laboratory.)

Human nature is not inherent sweetness and light. But if evil is not something that comes from outside, then neither is morality external. It's not as if we got it from God.

I won't say that you ought to adopt a view of goodness that's more internal. I won't tell you that you have a duty to do it. But if you see morality as something that's outside yourself, then I think you've gone down a garden path; and I hope that, in coming to see this, you will retrace your footsteps.

Take a good look in the mirror, and ask yourself: Would I rather that people be happy, than sad?

If the answer is "Yes", you really have no call to blame anyone else for your altruism; you're just a good person, that's all.

But what if the answer is: "Not really - I don't care much about other people."

Then I ask: Does answering this way, make you sad? Do you wish that you could answer differently?

If so, then this sadness again originates in you, and it would be futile to attribute it to anything not-you.

But suppose the one even says: "Actually, I actively dislike most people I meet and want to hit them with a sockfull of spare change. Only my knowledge that it would be wrong keeps me from acting on my desire."

Then I would say to look in the mirror and ask yourself who it is that prefers to do the right thing, rather than the wrong thing. And again if the answer is "Me", then it is pointless to externalize your righteousness.

Albeit if the one says: "I hate everyone else in the world and want to hurt them before they die, and also I have no interest in right or wrong; I am restrained from being a serial killer only out of a cold, calculated fear of punishment" - then, I admit, I have very little to say to them.

Occasionally I meet people who are not serial killers, but who have decided for some reason that they ought to be only selfish, and therefore, should reject their own preference that other people be happy rather than sad. I wish I knew what sort of cognitive history leads into this state of mind. Ayn Rand? Aleister Crowley? How exactly do you get there? What Rubicons do you cross? It's not the justifications I'm interested in, but the critical moments of thought.

Even the most elementary ideas of Friendly AI cannot be grasped by someone who externalizes morality. They will think of Friendliness as chains imposed to constrain the AI's own "true" desires; rather than as a shaping (selection from out of a huge space of possibilities) of the AI so that the AI chooses according to certain criteria, "its own desire" as it were. They will object to the idea of founding the AI on human morals in any way, saying, "But humans are such awful creatures," not realizing that it is only humans who have ever passed such a judgment.

As recounted in Original Teachings of Ch'an Buddhism by Chang Chung-Yuan, and quoted by Smullyan:

One day P'ang Yun, sitting quietly in his temple, made this remark:

"How difficult it is!
How difficult it is!
My studies are like drying the fibers of a thousand pounds
of flax in the sun by hanging them on the trees!"

But his wife responded:

"My way is easy indeed!
I found the teachings of the
Patriarchs right on the tops
of the flowering plants!"

When their daughter overheard this exchange, she sang:

"My study is neither difficult nor easy.
When I am hungry I eat,
When I am tired I rest."

Original with comments: Inner Goodness

San Jose Meetup, Sat 10/25 @ 7:30pm

Eliezer Yudkowsky, 23 October 2008 10:55PM

It's on Saturday 7.30pm at Il Fornaio, 302 S Market St (in the Sainte Claire Hotel), San Jose. All aspiring rationalists welcome. The reservation is currently for 21 but can be changed if needed. Please RSVP if you haven't already.

They have a wide variety including the requested vegetarian options & pizza from an oak wood burning oven, mostly good reviews, casual dress code, the Wine Spectator Award of Excellence 2007, and are just down the road from Montgomery Theater.

7.30 should give people time to finish chatting at the theater or freshen up at their hotels, but if you want to come a bit earlier they say they'll probably be able to seat you. If lots of people think 7.30 is too late I could change the time or suggest a cafe for early birds to grab a coffee in before dinner.

Please RSVP whether or not you'll be coming, either in a comment or email me at cursor_loop 4t yahoo p0int com. That way if new people want to come as they can take unclaimed places, and I can let the restaurant know if the number changes. If you think you'll be late, please let me know that too so we can ask the restaurant to keep your seat.

Also, as my cellphone won't work San Jose, please could someone volunteer their number for unexpectedly late or lost people to ring?

Please see here for further info. Hope to see you all on Saturday!

-- Posted on behalf of Michael Howard

Original with comments: San Jose Meetup, Sat 10/25 @ 7:30pm

Expected Creative Surprises

Eliezer Yudkowsky, 24 October 2008 10:22PM

Imagine that I'm playing chess against a smarter opponent. If I could predict exactly where my opponent would move on each turn, I would automatically be at least as good a chess player as my opponent. I could just ask myself where my opponent would move, if they were in my shoes; and then make the same move myself. (In fact, to predict my opponent's exact moves, I would need to be superhuman - I would need to predict my opponent's exact mental processes, including their limitations and their errors. It would become a problem of psychology, rather than chess.)

So predicting an exact move is not possible, but neither is it true that I have no information about my opponent's moves.

Personally, I am a very weak chess player - I play an average of maybe two games per year. But even if I'm playing against former world champion Garry Kasparov, there are certain things I can predict about his next move. When the game starts, I can guess that the move P-K4 is more likely than P-KN4. I can guess that if Kasparov has a move which would allow me to checkmate him on my next move, that Kasparov will not make that move.

Much less reliably, I can guess that Kasparov will not make a move that exposes his queen to my capture - but here, I could be greatly surprised; there could be a rationale for a queen sacrifice which I have not seen.

And finally, of course, I can guess that Kasparov will win the game...

Supposing that Kasparov is playing black, I can guess that the final position of the chess board will occupy the class of positions that are wins for black. I cannot predict specific features of the board in detail; but I can narrow things down relative to the class of all possible ending positions.

If I play chess against a superior opponent, and I don't know for certain where my opponent will move, I can still endeavor to produce a probability distribution that is well-calibrated - in the sense that, over the course of many games, legal moves that I label with a probability of "ten percent" are made by the opponent around 1 time in 10.

You might ask: Is producing a well-calibrated distribution over Kasparov, beyond my abilities as an inferior chess player?

But there is a trivial way to produce a well-calibrated probability distribution - just use the maximum-entropy distribution representing a state of total ignorance. If my opponent has 37 legal moves, I can assign a probability of 1/37 to each move. This makes me perfectly calibrated: I assigned 37 different moves a probability of 1 in 37, and exactly one of those moves will happen; so I applied the label "1 in 37" to 37 different events, and exactly 1 of those events occurred.

Total ignorance is not very useful, even if you confess it honestly. So the question then becomes whether I can do better than maximum entropy. Let's say that you and I both answer a quiz with ten yes-or-no questions. You assign probabilities of 90% to your answers, and get one answer wrong. I assign probabilities of 80% to my answers, and get two answers wrong. We are both perfectly calibrated but you exhibited better discrimination - your answers more strongly distinguished truth from falsehood.

Suppose that someone shows me an arbitrary chess position, and asks me: "What move would Kasparov make if he played black, starting from this position?" Since I'm not nearly as good a chess player as Kasparov, I can only weakly guess Kasparov's move, and I'll assign a non-extreme probability distribution to Kasparov's possible moves. In principle I can do this for any legal chess position, though my guesses might approach maximum entropy - still, I would at least assign a lower probability to what I guessed were obviously wasteful or suicidal moves.

If you put me in a box and feed me chess positions and get probability distributions back out, then we would have - theoretically speaking - a system that produces Yudkowsky's guess for Kasparov's move in any chess position. We shall suppose (though it may be unlikely) that my prediction is well-calibrated, if not overwhelmingly discriminating.

Now suppose we turn "Yudkowsky's prediction of Kasparov's move" into an actual chess opponent, by having a computer randomly make moves at the exact probabilities I assigned. We'll call this system RYK, which stands for "Randomized Yudkowsky-Kasparov", though it should really be "Random Selection from Yudkowsky's Probability Distribution over Kasparov's Move."

Will RYK be as good a player as Kasparov? Of course not. Sometimes the RYK system will randomly make dreadful moves which the real-life Kasparov would never make - start the game with P-KN4. I assign such moves a low probability, but sometimes the computer makes them anyway, by sheer random chance. The real Kasparov also sometimes makes moves that I assigned a low probability, but only when the move has a better rationale than I realized - the astonishing, unanticipated queen sacrifice.

Randomized Yudkowsky-Kasparov is definitely no smarter than Yudkowsky, because RYK draws on no more chess skill than I myself possess - I build all the probability distributions myself, using only my own abilities. Actually, RYK is a far worse player than Yudkowsky. I myself would make the best move I saw with my knowledge. RYK only occasionally makes the best move I saw - I won't be very confident that Kasparov would make exactly the same move I would.

Now suppose that I myself play a game of chess against the RYK system.

RYK has the odd property that, on each and every turn, my probabilistic prediction for RYK's move is exactly the same prediction I would make if I were playing against world champion Garry Kasparov.

Nonetheless, I can easily beat RYK, where the real Kasparov would crush me like a bug.

The creative unpredictability of intelligence is not like the noisy unpredictability of a random number generator. When I play against a smarter player, I can't predict exactly where my opponent will move against me. But I can predict the end result of my smarter opponent's moves, which is a win for the other player. When I see the randomized opponent make a move that I assigned a tiny probability, I chuckle and rub my hands, because I think the opponent has randomly made a dreadful move and now I can win. When a superior opponent surprises me by making a move to which I assigned a tiny probability, I groan because I think the other player saw something I didn't, and now I'm about to be swept off the board. Even though it's exactly the same probability distribution! I can be exactly as uncertain about the actions, and yet draw very different conclusions about the eventual outcome.

(This situation is possible because I am not logically omniscient; I do not explicitly represent a joint probability distribution over all entire games.)

When I play against a superior player, I can't predict exactly where my opponent will move against me. If I could predict that, I would necessarily be at least that good at chess myself. But I can predict the consequence of the unknown move, which is a win for the other player; and the more the player's actual action surprises me, the more confident I become of this final outcome.

The unpredictability of intelligence is a very special and unusual kind of surprise, which is not at all like noise or randomness. There is a weird balance between the unpredictability of actions and the predictability of outcomes.

Referenced by: Belief in Intelligence • ...Recursion, Magic • Justified Expectation of Pleasant Surprises

Original with comments: Expected Creative Surprises

Belief in Intelligence

Eliezer Yudkowsky, 25 October 2008 03:00PM

Previously in series: Expected Creative Surprises

Since I am so uncertain of Kasparov's moves, what is the empirical content of my belief that "Kasparov is a highly intelligent chess player"? What real-world experience does my belief tell me to anticipate? Is it a cleverly masked form of total ignorance?

To sharpen the dilemma, suppose Kasparov plays against some mere chess grandmaster Mr. G, who's not in the running for world champion. My own ability is far too low to distinguish between these levels of chess skill. When I try to guess Kasparov's move, or Mr. G's next move, all I can do is try to guess "the best chess move" using my own meager knowledge of chess. Then I would produce exactly the same prediction for Kasparov's move or Mr. G's move in any particular chess position. So what is the empirical content of my belief that "Kasparov is a better chess player than Mr. G"?

The empirical content of my belief is the testable, falsifiable prediction that the final chess position will occupy the class of chess positions that are wins for Kasparov, rather than drawn games or wins for Mr. G. (Counting resignation as a legal move that leads to a chess position classified as a loss.) The degree to which I think Kasparov is a "better player" is reflected in the amount of probability mass I concentrate into the "Kasparov wins" class of outcomes, versus the "drawn game" and "Mr. G wins" class of outcomes. These classes are extremely vague in the sense that they refer to vast spaces of possible chess positions - but "Kasparov wins" is more specific than maximum entropy, because it can be definitely falsified by a vast set of chess positions.

The outcome of Kasparov's game is predictable because I know, and understand, Kasparov's goals. Within the confines of the chess board, I know Kasparov's motivations - I know his success criterion, his utility function, his target as an optimization process. I know where Kasparov is ultimately trying to steer the future and I anticipate he is powerful enough to get there, although I don't anticipate much about how Kasparov is going to do it.

Imagine that I'm visiting a distant city, and a local friend volunteers to drive me to the airport. I don't know the neighborhood. Each time my friend approaches a street intersection, I don't know whether my friend will turn left, turn right, or continue straight ahead. I can't predict my friend's move even as we approach each individual intersection - let alone, predict the whole sequence of moves in advance.

Yet I can predict the result of my friend's unpredictable actions: we will arrive at the airport. Even if my friend's house were located elsewhere in the city, so that my friend made a completely different sequence of turns, I would just as confidently predict our arrival at the airport. I can predict this long in advance, before I even get into the car. My flight departs soon, and there's no time to waste; I wouldn't get into the car in the first place, if I couldn't confidently predict that the car would travel to the airport along an unpredictable pathway.

Isn't this a remarkable situation to be in, from a scientific perspective? I can predict the outcome of a process, without being able to predict any of the intermediate steps of the process.

How is this even possible? Ordinarily one predicts by imagining the present and then running the visualization forward in time. If you want a precise model of the Solar System, one that takes into account planetary perturbations, you must start with a model of all major objects and run that model forward in time, step by step.

Sometimes simpler problems have a closed-form solution, where calculating the future at time T takes the same amount of work regardless of T. A coin rests on a table, and after each minute, the coin turns over. The coin starts out showing heads. What face will it show a hundred minutes later? Obviously you did not answer this question by visualizing a hundred intervening steps. You used a closed-form solution that worked to predict the outcome, and would also work to predict any of the intervening steps.

But when my friend drives me to the airport, I can predict the outcome successfully using a strange model that won't work to predict any of the intermediate steps. My model doesn't even require me to input the initial conditions - I don't need to know where we start out in the city!

I do need to know something about my friend. I must know that my friend wants me to make my flight. I must credit that my friend is a good enough planner to successfully drive me to the airport (if he wants to). These are properties of my friend's initial state - properties which let me predict the final destination, though not any intermediate turns.

I must also credit that my friend knows enough about the city to drive successfully. This may be regarded as a relation between my friend and the city; hence, a property of both. But an extremely abstract property, which does not require any specific knowledge about either the city, or about my friend's knowledge about the city.

This is one way of viewing the subject matter to which I've devoted my life - these remarkable situations which place us in such an odd epistemic positions. And my work, in a sense, can be viewed as unraveling the exact form of that strange abstract knowledge we can possess; whereby, not knowing the actions, we can justifiably know the consequence.

"Intelligence" is too narrow a term to describe these remarkable situations in full generality. I would say rather "optimization process". A similar situation accompanies the study of biological natural selection, for example; we can't predict the exact form of the next organism observed.

But my own specialty is the kind of optimization process called "intelligence"; and even narrower, a particular kind of intelligence called "Friendly Artificial Intelligence" - of which, I hope, I will be able to obtain especially precise abstract knowledge.

Referenced by: Aiming at the Target • Building Something Smarter

Original with comments: Belief in Intelligence

Aiming at the Target

Eliezer Yudkowsky, 26 October 2008 04:47PM

Previously in series: Belief in Intelligence

Previously, I spoke of that very strange epistemic position one can occupy, wherein you don't know exactly where Kasparov will move on the chessboard, and yet your state of knowledge about the game is very different than if you faced a random move-generator with the same subjective probability distribution - in particular, you expect Kasparov to win. I have beliefs about where Kasparov wants to steer the future, and beliefs about his power to do so.

Well, and how do I describe this knowledge, exactly?

In the case of chess, there's a simple function that classifies chess positions into wins for black, wins for white, and drawn games. If I know which side Kasparov is playing, I know the class of chess positions Kasparov is aiming for. (If I don't know which side Kasparov is playing, I can't predict whether black or white will win - which is not the same as confidently predicting a drawn game.)

More generally, I can describe motivations using a preference ordering. When I consider two potential outcomes, X and Y, I can say that I prefer X to Y; prefer Y to X; or find myself indifferent between them. I would write these relations as X > Y; X < Y; and X ~ Y.

Suppose that you have the ordering A < B ~ C < D ~ E. Then you like B more than A, and C more than A. {B, C}, belonging to the same class, seem equally desirable to you; you are indifferent between which of {B, C} you receive, though you would rather have either than A, and you would rather have something from the class {D, E} than {B, C}.

When I think you're a powerful intelligence, and I think I know something about your preferences, then I'll predict that you'll steer reality into regions that are higher in your preference ordering.

Think of a huge circle containing all possible outcomes, such that outcomes higher in your preference ordering appear to be closer to the center. Outcomes between which you are indifferent are the same distance from the center - imagine concentric rings of outcomes that are all equally preferred. If you aim your actions and strike a consequence close to the center - an outcome that ranks high in your preference ordering - then I'll think better of your ability to aim.

The more intelligent I believe you are, the more probability I'll concentrate into outcomes that I believe are higher in your preference ordering - that is, the more I'll expect you to achieve a good outcome, and the better I'll expect the outcome to be. Even if a powerful enemy opposes you, so that I expect the final outcome to be one that is low in your preference ordering, I'll still expect you to lose less badly if I think you're more intelligent.

What about expected utilities as opposed to preference orderings? To talk about these, you have to attribute a probability distribution to the actor, or to the environment - you can't just observe the outcome. If you have one of these probability distributions, then your knowledge of a utility function can let you guess at preferences between gambles (stochastic outcomes) and not just preferences between the outcomes themselves.

The "aiming at the target" metaphor - and the notion of measuring how closely we hit - extends beyond just terminal outcomes, to the forms of instrumental devices and instrumental plans.

Consider a car - say, a Toyota Corolla. The Toyota Corolla is made up of some number of atoms - say, on the (very) rough order of ten to the twenty-ninth. If you consider all the possible ways we could arrange those 10^29 atoms, it's clear that only an infinitesimally tiny fraction of possible configurations would qualify as a working car. If you picked a random configurations of 10^29 atoms once per Planck time, many ages of the universe would pass before you hit on a wheeled wagon, let alone an internal combustion engine.

(When I talk about this in front of a popular audience, someone usually asks: "But isn't this what the creationists argue? That if you took a bunch of atoms and put them in a box and shook them up, it would be astonishingly improbable for a fully functioning rabbit to fall out?" But the logical flaw in the creationists' argument is not that randomly reconfiguring molecules would by pure chance assemble a rabbit. The logical flaw is that there is a process, natural selection, which, through the non-chance retention of chance mutations, selectively accumulates complexity, until a few billion years later it produces a rabbit. Only the very first replicator in the history of time needed to pop out of the random shaking of molecules - perhaps a short RNA string, though there are more sophisticated hypotheses about autocatalytic hypercycles of chemistry.)

Even restricting our attention to running vehicles, there is an astronomically huge design space of possible vehicles that could be composed of the same atoms as the Corolla, and most of them, from the perspective of a human user, won't work quite as well. We could take the parts in the Corolla's air conditioner, and mix them up in thousands of possible configurations; nearly all these configurations would result in a vehicle lower in our preference ordering, still recognizable as a car but lacking a working air conditioner.

So there are many more configurations corresponding to nonvehicles, or vehicles lower in our preference ranking, than vehicles ranked greater than or equal to the Corolla.

A tiny fraction of the design space does describe vehicles that we would recognize as faster, more efficient, and safer than the Corolla. Thus the Corolla is not optimal under our preferences, nor under the designer's own goals. The Corolla is, however, optimized, because the designer had to hit an infinitesimal target in design space just to create a working car, let alone a car of Corolla-equivalent quality. The subspace of working vehicles is dwarfed by the space of all possible molecular configurations for the same atoms. You cannot build so much as an effective wagon by sawing boards into random shapes and nailing them together according to coinflips. To hit such a tiny target in configuration space requires a powerful optimization process. The better the car you want, the more optimization pressure you have to exert - though you need a huge optimization pressure just to get a car at all.

This whole discussion assumes implicitly that the designer of the Corolla was trying to produce a "vehicle", a means of travel. This assumption deserves to be made explicit, but it is not wrong, and it is highly useful in understanding the Corolla.

Planning also involves hitting tiny targets in a huge search space. On a 19-by-19 Go board there are roughly 1e180 legal positions (not counting superkos). On early positions of a Go game there are more than 300 legal moves per turn. The search space explodes, and nearly all moves are foolish ones if your goal is to win the game. From all the vast space of Go possibilities, a Go player seeks out the infinitesimal fraction of plans which have a decent chance of winning.

You cannot even drive to the supermarket without planning - it will take you a long, long time to arrive if you make random turns at each intersection. The set of turn sequences that will take you to the supermarket is a tiny subset of the space of turn sequences. Note that the subset of turn sequences we're seeking is defined by its consequence - the target - the destination. Within that subset, we care about other things, like the driving distance. (There are plans that would take us to the supermarket in a huge pointless loop-the-loop.)

In general, as you live your life, you try to steer reality into a particular region of possible futures. When you buy a Corolla, you do it because you want to drive to the supermarket. You drive to the supermarket to buy food, which is a step in a larger strategy to avoid starving. All else being equal, you prefer possible futures in which you are alive, rather than dead of starvation.

When you drive to the supermarket, you aren't really aiming for the supermarket, you're aiming for a region of possible futures in which you don't starve. Each turn at each intersection doesn't carry you toward the supermarket, it carries you out of the region of possible futures where you lie helplessly starving in your apartment. If you knew the supermarket was empty, you wouldn't bother driving there. An empty supermarket would occupy exactly the same place on your map of the city, but it wouldn't occupy the same role in your map of possible futures. It is not a location within the city that you are really aiming at, when you drive.

Human intelligence is one kind of powerful optimization process, capable of winning a game of Go or turning sand into digital computers. Natural selection is much slower than human intelligence; but over geological time, cumulative selection pressure qualifies as a powerful optimization process.

Once upon a time, human beings anthropomorphized stars, saw constellations in the sky and battles between constellations. But though stars burn longer and brighter than any craft of biology or human artifice, stars are neither optimization processes, nor products of strong optimization pressures. The stars are not gods; there is no true power in them.

Referenced by: Measuring Optimization Power • Free to Optimize

Original with comments: Aiming at the Target

Measuring Optimization Power

Eliezer Yudkowsky, 27 October 2008 09:44PM

Previously in series: Aiming at the Target

Yesterday I spoke of how "When I think you're a powerful intelligence, and I think I know something about your preferences, then I'll predict that you'll steer reality into regions that are higher in your preference ordering."

You can quantify this, at least in theory, supposing you have (A) the agent or optimization process's preference ordering, and (B) a measure of the space of outcomes - which, for discrete outcomes in a finite space of possibilities, could just consist of counting them - then you can quantify how small a target is being hit, within how large a greater region.

Then we count the total number of states with equal or greater rank in the preference ordering to the outcome achieved, or integrate over the measure of states with equal or greater rank. Dividing this by the total size of the space gives you the relative smallness of the target - did you hit an outcome that was one in a million? One in a trillion?

Actually, most optimization processes produce "surprises" that are exponentially more improbable than this - you'd need to try far more than a trillion random reorderings of the letters in a book, to produce a play of quality equalling or exceeding Shakespeare. So we take the log base two of the reciprocal of the improbability, and that gives us optimization power in bits.

This figure - roughly, the improbability of an "equally preferred" outcome being produced by a random selection from the space (or measure on the space) - forms the foundation of my Bayesian view of intelligence, or to be precise, optimization power. It has many subtleties:

(1) The wise will recognize that we are calculating the entropy of something. We could take the figure of the relative improbability of "equally good or better" outcomes, and call this the negentropy of the system relative to a preference ordering. Unlike thermodynamic entropy, the entropy of a system relative to a preference ordering can easily decrease (that is, the negentropy can increase, that is, things can get better over time relative to a preference ordering).

Suppose e.g. that a single switch will determine whether the world is saved or destroyed, and you don't know whether the switch is set to 1 or 0. You can carry out an operation that coerces the switch to 1; in accordance with the second law of thermodynamics, this requires you to dump one bit of entropy somewhere, e.g. by radiating a single photon of waste heat into the void. But you don't care about that photon - it's not alive, it's not sentient, it doesn't hurt - whereas you care a very great deal about the switch.

For some odd reason, I had the above insight while watching X TV. (Those of you who've seen it know why this is funny.)

Taking physical entropy out of propositional variables that you care about - coercing them from unoptimized states into an optimized states - and dumping the entropy into residual variables that you don't care about, means that relative to your preference ordering, the total "entropy" of the universe goes down. This is pretty much what life is all about.

We care more about the variables we plan to alter, than we care about the waste heat emitted by our brains. If this were not the case - if our preferences didn't neatly compartmentalize the universe into cared-for propositional variables and everything else - then the second law of thermodynamics would prohibit us from ever doing optimization. Just like there are no-free-lunch theorems showing that cognition is impossible in a maxentropy universe, optimization will prove futile if you have maxentropy preferences. Having maximally disordered preferences over an ordered universe is pretty much the same dilemma as the reverse.

(2) The quantity we're measuring tells us how improbable this event is, in the absence of optimization, relative to some prior measure that describes the unoptimized probabilities. To look at it another way, the quantity is how surprised you would be by the event, conditional on the hypothesis that there were no optimization processes around. This plugs directly into Bayesian updating: it says that highly optimized events are strong evidence for optimization processes that produce them.

Ah, but how do you know a mind's preference ordering? Suppose you flip a coin 30 times and it comes up with some random-looking string - how do you know this wasn't because a mind wanted it to produce that string?

This, in turn, is reminiscent of the Minimum Message Length formulation of Occam's Razor: if you send me a message telling me what a mind wants and how powerful it is, then this should enable you to compress your description of future events and observations, so that the total message is shorter. Otherwise there is no predictive benefit to viewing a system as an optimization process. This criterion tells us when to take the intentional stance.

(3) Actually, you need to fit another criterion to take the intentional stance - there can't be a better description that averts the need to talk about optimization. This is an epistemic criterion more than a physical one - a sufficiently powerful mind might have no need to take the intentional stance toward a human, because it could just model the regularity of our brains like moving parts in a machine.

(4) If you have a coin that always comes up heads, there's no need to say "The coin always wants to come up heads" because you can just say "the coin always comes up heads". Optimization will beat alternative mechanical explanations when our ability to perturb a system defeats our ability to predict its interim steps in detail, but not our ability to predict a narrow final outcome. (Again, note that this is an epistemic criterion.)

(5) Suppose you believe a mind exists, but you don't know its preferences? Then you use some of your evidence to infer the mind's preference ordering, and then use the inferred preferences to infer the mind's power, then use those two beliefs to testably predict future outcomes. The total gain in predictive accuracy should exceed the complexity-cost of supposing that "there's a mind of unknown preferences around", the initial hypothesis.

Similarly, if you're not sure whether there's an optimizer around, some of your evidence-fuel burns to support the hypothesis that there's an optimizer around, some of your evidence is expended to infer its target, and some of your evidence is expended to infer its power. The rest of the evidence should be well explained, or better yet predicted in advance, by this inferred data: this is your revenue on the transaction, which should exceed the costs just incurred, making an epistemic profit.

(6) If you presume that you know (from a superior epistemic vantage point) the probabilistic consequences of an action or plan, or if you measure the consequences repeatedly, and if you know or infer a utility function rather than just a preference ordering, then you might be able to talk about the degree of optimization of an action or plan rather than just the negentropy of a final outcome. We talk about the degree to which a plan has "improbably" high expected utility, relative to a measure over the space of all possible plans.

(7) A similar presumption that we can measure the instrumental value of a device, relative to a terminal utility function, lets us talk about a Toyota Corolla as an "optimized" physical object, even though we attach little terminal value to it per se.

(8) If you're a human yourself and you take the measure of a problem, then there may be "obvious" solutions that don't count for much in your view, even though the solution might be very hard for a chimpanzee to find, or a snail. Roughly, because your own mind is efficient enough to calculate the solution without an apparent expenditure of internal effort, a solution that good will seem to have high probability, and so an equally good solution will not seem very improbable.

By presuming a base level of intelligence, we measure the improbability of a solution that "would take us some effort", rather than the improbability of the same solution emerging from a random noise generator. This is one reason why many people say things like "There has been no progress in AI; machines still aren't intelligent at all." There are legitimate abilities that modern algorithms entirely lack, but mostly what they're seeing is that AI is "dumber than a village idiot" - it doesn't even do as well as the "obvious" solutions that get most of the human's intuitive measure, let alone surprisingly better than that; it seems anti-intelligent, stupid.

To measure the impressiveness of a solution to a human, you've got to do a few things that are a bit more complicated than just measuring optimization power. For example, if a human sees an obvious computer program to compute many solutions, they will measure the total impressiveness of all the solutions as being no more than the impressiveness of writing the computer program - but from the internal perspective of the computer program, it might seem to be making a metaphorical effort on each additional occasion. From the perspective of Deep Blue's programmers, Deep Blue is a one-time optimization cost; from Deep Blue's perspective it has to optimize each chess game individually.

To measure human impressiveness you have to talk quite a bit about humans - how humans compact the search space, the meta-level on which humans approach a problem. People who try to take human impressiveness as their primitive measure will run into difficulties, because in fact the measure is not very primitive.

(9) For the vast majority of real-world problems we will not be able to calculate exact optimization powers, any more than we can do actual Bayesian updating over all hypotheses, or actual expected utility maximization in our planning. But, just like Bayesian updating or expected utility maximization, the notion of optimization power does give us a gold standard against which to measure - a simple mathematical idea of what we are trying to do whenever we essay more complicated and efficient algorithms.

(10) "Intelligence" is efficient cross-domain optimization.

Referenced by: Efficient Cross-Domain Optimization • Economic Definition of Intelligence? • Intelligence in Economics • Complexity and Intelligence • Recognizing Intelligence • Worse Than Random • The First World Takeover

Original with comments: Measuring Optimization Power

Efficient Cross-Domain Optimization

Eliezer Yudkowsky, 28 October 2008 04:33PM

Previously in series: Measuring Optimization Power

Is Deep Blue "intelligent"? It was powerful enough at optimizing chess boards to defeat Kasparov, perhaps the most skilled chess player humanity has ever fielded.

A bee builds hives, and a beaver builds dams; but a bee doesn't build dams and a beaver doesn't build hives. A human, watching, thinks, "Oh, I see how to do it" and goes on to build a dam using a honeycomb structure for extra strength.

Deep Blue, like the bee and the beaver, never ventured outside the narrow domain that it itself was optimized over.

There are no-free-lunch theorems showing that you can't have a truly general intelligence that optimizes in all possible universes (the vast majority of which are maximum-entropy heat baths). And even practically speaking, human beings are better at throwing spears than, say, writing computer programs.

But humans are much more cross-domain than bees, beavers, or Deep Blue. We might even conceivably be able to comprehend the halting behavior of every Turing machine up to 10 states, though I doubt it goes much higher than that.

Every mind operates in some domain, but the domain that humans operate in isn't "the savanna" but something more like "not too complicated processes in low-entropy lawful universes". We learn whole new domains by observation, in the same way that a beaver might learn to chew a different kind of wood. If I could write out your prior, I could describe more exactly the universes in which you operate.

Is evolution intelligent? It operates across domains - not quite as well as humans do, but with the same general ability to do consequentialist optimization on causal sequences that wend through widely different domains. It built the bee. It built the beaver.

Whatever begins with genes, and impacts inclusive genetic fitness, through any chain of cause and effect in any domain, is subject to evolutionary optimization. That much is true.

But evolution only achieves this by running millions of actual experiments in which the causal chains are actually played out. This is incredibly inefficient. Cynthia Kenyon said, "One grad student can do things in an hour that evolution could not do in a billion years." This is not because the grad student does quadrillions of detailed thought experiments in their imagination, but because the grad student abstracts over the search space.

By human standards, evolution is unbelievably stupid. It is the degenerate case of design with intelligence equal to zero, as befitting the accidentally occurring optimization process that got the whole thing started in the first place.

(As for saying that "evolution built humans, therefore it is efficient", this is, firstly, a sophomoric objection; second, it confuses levels. Deep Blue's programmers were not superhuman chessplayers. The importance of distinguishing levels can be seen from the point that humans are efficiently optimizing human goals, which are not the same as evolution's goal of inclusive genetic fitness. Evolution, in producing humans, may have entirely doomed DNA.)

I once heard a senior mainstream AI type suggest that we might try to quantify the intelligence of an AI system in terms of its RAM, processing power, and sensory input bandwidth. This at once reminded me of a quote from Dijkstra: "If we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger." If you want to measure the intelligence of a system, I would suggest measuring its optimization power as before, but then dividing by the resources used. Or you might measure the degree of prior cognitive optimization required to achieve the same result using equal or fewer resources. Intelligence, in other words, is efficient optimization.

So if we say "efficient cross-domain optimization" - is that necessary and sufficient to convey the wisest meaning of "intelligence", after making a proper effort to factor out anthropomorphism in ranking solutions?

I do hereby propose: "Yes."

Years ago when I was on a panel with Jaron Lanier, he had offered some elaborate argument that no machine could be intelligent, because it was just a machine and to call it "intelligent" was therefore bad poetry, or something along those lines. Fed up, I finally snapped: "Do you mean to say that if I write a computer program and that computer program rewrites itself and rewrites itself and builds its own nanotechnology and zips off to Alpha Centauri and builds its own Dyson Sphere, that computer program is not intelligent?"

This, I think, is a core meaning of "intelligence" that it is wise to keep in mind.

I mean, maybe not that exact test. And it wouldn't be wise to bow too directly to human notions of "impressiveness", because this is what causes people to conclude that a butterfly must have been intelligently designed (they don't see the vast incredibly wasteful trail of trial and error), or that an expected paperclip maximizer is stupid.

But still, intelligences ought to be able to do cool stuff, in a reasonable amount of time using reasonable resources, even if we throw things at them that they haven't seen before, or change the rules of the game (domain) a little. It is my contention that this is what's captured by the notion of "efficient cross-domain optimization".

Occasionally I hear someone say something along the lines of, "No matter how smart you are, a tiger can still eat you." Sure, if you get stripped naked and thrown into a pit with no chance to prepare and no prior training, you may be in trouble. And by similar token, a human can be killed by a large rock dropping on their head. It doesn't mean a big rock is more powerful than a human.

A large asteroid, falling on Earth, would make an impressive bang. But if we spot the asteroid, we can try to deflect it through any number of methods. With enough lead time, a can of black paint will do as well as a nuclear weapon. And the asteroid itself won't oppose us on our own level - won't try to think of a counterplan. It won't send out interceptors to block the nuclear weapon. It won't try to paint the opposite side of itself with more black paint, to keep its current trajectory. And if we stop that asteroid, the asteroid belt won't send another planet-killer in its place.

We might have to do some work to steer the future out of the unpleasant region it will go to if we do nothing, but the asteroid itself isn't steering the future in any meaningful sense. It's as simple as water flowing downhill, and if we nudge the asteroid off the path, it won't nudge itself back.

The tiger isn't quite like this. If you try to run, it will follow you. If you dodge, it will follow you. If you try to hide, it will spot you. If you climb a tree, it will wait beneath.

But if you come back with an armored tank - or maybe just a hunk of poisoned meat - the tiger is out of luck. You threw something at it that wasn't in the domain it was designed to learn about. The tiger can't do cross-domain optimization, so all you need to do is give it a little cross-domain nudge and it will spin off its course like a painted asteroid.

Steering the future, not energy or mass, not food or bullets, is the raw currency of conflict and cooperation among agents. Kasparov competed against Deep Blue to steer the chessboard into a region where he won - knights and bishops were only his pawns. And if Kasparov had been allowed to use any means to win against Deep Blue, rather than being artificially restricted, it would have been a trivial matter to kick the computer off the table - a rather light optimization pressure by comparison with Deep Blue's examining hundreds of millions of moves per second, or by comparison with Kasparov's pattern-recognition of the board; but it would have crossed domains into a causal chain that Deep Blue couldn't model and couldn't optimize and couldn't resist. One bit of optimization pressure is enough to flip a switch that a narrower opponent can't switch back.

A superior general can win with fewer troops, and superior technology can win with a handful of troops. But even a suitcase nuke requires at least a few kilograms of matter. If two intelligences of the same level compete with different resources, the battle will usually go to the wealthier.

The same is true, on a deeper level, of efficient designs using different amounts of computing power. Human beings, five hundred years after the Scientific Revolution, are only just starting to match their wits against the billion-year heritage of biology. We're vastly faster, it has a vastly longer lead time; after five hundred years and a billion years respectively, the two powers are starting to balance.

But as a measure of intelligence, I think it is better to speak of how well you can use your resources - if we want to talk about raw impact, then we can speak of optimization power directly.

So again I claim that this - computationally-frugal cross-domain future-steering - is the necessary and sufficient meaning that the wise should attach to the word, "intelligence".

Referenced by: Economic Definition of Intelligence? • Intelligence in Economics • Building Something Smarter • Nonsentient Optimizers • Free to Optimize

Original with comments: Efficient Cross-Domain Optimization

Economic Definition of Intelligence?

Eliezer Yudkowsky, 29 October 2008 07:32PM

Followup to: Efficient Cross-Domain Optimization

Shane Legg once produced a catalogue of 71 definitions of intelligence. Looking it over, you'll find that the 18 definitions in dictionaries and the 35 definitions of psychologists are mere black boxes containing human parts.

However, among the 18 definitions from AI researchers, you can find such notions as

"Intelligence measures an agent's ability to achieve goals in a wide range of environments" (Legg and Hutter)

or

"Intelligence is the ability to optimally use limited resources - including time - to achieve goals" (Kurzweil)

or even

"Intelligence is the power to rapidly find an adequate solution in what appears a priori (to observers) to be an immense search space" (Lenat and Feigenbaum)

which is about as close as you can get to my own notion of "efficient cross-domain optimization" without actually measuring optimization power in bits.

But Robin Hanson, whose AI background we're going to ignore for a moment in favor of his better-known identity as an economist, at once said:

"I think what you want is to think in terms of a production function, which describes a system's output on a particular task as a function of its various inputs and features."

Economists spend a fair amount of their time measuring things like productivity and efficiency. Might they have something to say about how to measure intelligence in generalized cognitive systems?

This is a real question, open to all economists. So I'm going to quickly go over some of the criteria-of-a-good-definition that stand behind my own proffered suggestion on intelligence, and what I see as the important challenges to a productivity-based view. It seems to me that this is an important sub-issue of Robin's and my persistent disagreement about the Singularity.

(A) One of the criteria involved in a definition of intelligence is that it ought to separate form and function. The Turing Test fails this - it says that if you can build something indistinguishable from a bird, it must definitely fly, which is true but spectacularly unuseful in building an airplane.

(B) We will also prefer quantitative measures to qualitative measures that only say "this is intelligent or not intelligent". Sure, you can define "flight" in terms of getting off the ground, but what you really need is a way to quantify aerodynamic lift and relate it to other properties of the airplane, so you can calculate how much lift is needed to get off the ground, and calculate how close you are to flying at any given point.

(C) So why not use the nicely quantified IQ test? Well, imagine if the Wright Brothers had tried to build the Wright Flyer using a notion of "flight quality" build around a Fly-Q test standardized on the abilities of the average pigeon, including various measures of wingspan and air maneuverability. We want a definition that is not parochial to humans.

(D) We have a nice system of Bayesian expected utility maximization. Why not say that any system's "intelligence" is just the average utility of the outcome it can achieve? But utility functions are invariant up to a positive affine transformation, i.e., if you add 3 to all utilities, or multiply all by 5, it's the same utility function. If we assume a fixed utility function, we would be able to compare the intelligence of the same system on different occasions - but we would like to be able to compare intelligences with different utility functions.

(E) And by much the same token, we would like our definition to let us recognize intelligence by observation rather than presumption, which means we can't always start off assuming that something has a fixed utility function, or even any utility function at all. We can have a prior over probable utility functions, which assigns a very low probability to overcomplicated hypotheses like "the lottery wanted 6-39-45-46-48-36 to win on October 28th, 2008", but higher probabilities to simpler desires.

(F) Why not just measure how well the intelligence plays chess? But in real-life situations, plucking the opponent's queen off the board or shooting the opponent is not illegal, it is creative. We would like our definition to respect the creative shortcut - to not define intelligence into the box of a narrow problem domain.

(G) It would be nice if intelligence were actually measurable using some operational test, but this conflicts strongly with criteria F and D. My own definition essentially tosses this out the window - you can't actually measure optimization power on any real-world problem any more than you can compute the real-world probability update or maximize real-world expected utility. But, just as you can wisely wield algorithms that behave sorta like Bayesian updates or increase expected utility, there are all sorts of possible methods that can take a stab at measuring optimization power.

(H) And finally, when all is said and done, we should be able to recognize very high "intelligence" levels in an entity that can, oh, say, synthesize nanotechnology and build its own Dyson Sphere. Nor should we assign very high "intelligence" levels to something that couldn't build a wooden wagon (even if it wanted to, and had hands). Intelligence should not be defined too far away from that impressive thingy we humans sometimes do.

Which brings us to production functions. I think the main problems here would lie in criteria DE.

First, a word of background: In Artificial Intelligence, it's more common to spend your days obsessing over the structure of a problem space - and when you find a good algorithm, you use that algorithm and pay however much computing power it requires. You aren't as likely to find a situation where there are five different algorithms competing to solve a problem and a sixth algorithm that has to decide where to invest a marginal unit of computing power. Not that computer scientists haven't studied this as a specialized problem. But it's ultimately not what AIfolk do all day. So I hope that we can both try to appreciate the danger of deformation professionelle.

Robin Hanson said:

"Eliezer, even if you measure output as you propose in terms of a state space reduction factor, my main point was that simply 'dividing by the resources used' makes little sense."

I agree that "divide by resources used" is a very naive method, rather tacked-on by comparison. If one mind gets 40 bits of optimization using a trillion floating-point operations, and another mind achieves 80 bits of optimization using two trillion floating-point operations, even in the same domain using the same utility function, they may not at all be equally "well-designed" minds. One of the minds may itself be a lot more "optimized" than the other (probably the second one).

I do think that measuring the rarity of equally good solutions in the search space smooths out the discussion a lot. More than any other simple measure I can think of. You're not just presuming that 80 units are twice as good as 40 units, but trying to give some measure of how rare 80-unit solutions are in the space; if they're common it will take less "optimization power" to find them and we'll be less impressed. This likewise helps when comparing minds with different preferences.

But some search spaces are just easier to search than others. I generally choose to talk about this by hiking the "optimization" metric up a meta-level: how easy is it to find an algorithm that searches this space? There's no absolute easiness, unless you talk about simple random selection, which I take as my base case. Even if a fitness gradient is smooth - a very simple search - e.g. natural selection would creep down it by incremental neighborhood search, while a human would leap through by e.g. looking at the first and second derivatives. Which of these is the "inherent easiness" of the space?

Robin says:

Then we can talk about partial derivatives; rates at which output increases as a function of changes in inputs or features... Yes a production function formulation may abstract from some relevant details, but it is far closer to reality than dividing by "resources."

A partial derivative divides the marginal output by marginal resource. Is this so much less naive than dividing total output by total resources?

I confess that I said "divide by resources" just to have some measure of efficiency; it's not a very good measure. Still, we need to take resources into account somehow - we don't want natural selection to look as "intelligent" as humans: human engineers, given 3.85 billion years and the opportunity to run 1e44 experiments, would produce products overwhelmingly superior to biology.

But this is really establishing an ordering based on superior performance with the same resources, not a quantitative metric. I might have to be content with a partial ordering among intelligences, rather than being able to quantify them. If so, one of the ordering characteristics will be the amount of resources used, which is what I was getting at by saying "divide by total resources".

The idiom of "division" is based around things that can be divided, that is, fungible resources. A human economy based on mass production has lots of these. In modern-day computing work, programmers use fungible resources like computing cycles and RAM, but tend to produce much less fungible outputs. Informational goods tend to be mostly non-fungible: two copies of the same file are worth around as much as one, so every worthwhile informational good is unique. If I draw on my memory to produce an essay, neither the sentences of the essay, or the items of my memory, will be substitutable for one another. If I create a unique essay by drawing upon a thousand unique memories, how well have I done, and how much resource have I used?

Economists have a simple way of establishing a kind of fungibility-of-valuation between all the inputs and all the outputs of an economy: they look at market prices.

But this just palms off the problem of valuation on hedge funds. Someone has to do the valuing. A society with stupid hedge funds ends up with stupid valuations.

Steve Omohundro has pointed out that for fungible resources in an AI - and computing power is a fungible resource on modern architectures - there ought to be a resource balance principle: the marginal result of shifting a unit of resource between any two tasks should produce a decrease in expected utility, relative to the AI's probability function that determines the expectation. To the extent any of these things have continuous first derivatives, shifting an infinitesimal unit of resource between any two tasks should have no effect on expected utility. This establishes "expected utilons" as something akin to a central currency within the AI.

But this gets us back to the problems of criteria D and E. If I look at a mind and see a certain balance of resources, is that because the mind is really cleverly balanced, or because the mind is stupid? If a mind would rather have two units of CPU than one unit of RAM (and how can I tell this by observation, since the resources are not readily convertible?) then is that because RAM is inherently twice as valuable as CPU, or because the mind is twice as stupid in using CPU as RAM?

If you can assume the resource-balance principle, then you will find it easy to talk about the relative efficiency of alternative algorithms for use inside the AI, but this doesn't give you a good way to measure the external power of the whole AI.

Similarly, assuming a particular relative valuation of resources, as given by an external marketplace, doesn't let us ask questions like "How smart is a human economy?" Now the relative valuation a human economy assigns to internal resources can no longer be taken for granted - a more powerful system might assign very different relative values to internal resources.

I admit that dividing optimization power by "total resources" is handwaving - more a qualitative way of saying "pay attention to resources used" than anything you could actually quantify into a single useful figure. But I pose an open question to Robin (or any other economist) to explain how production theory can help us do better, bearing in mind that:

- Informational inputs and outputs tend to be non-fungible;

- I want to be able to observe the "intelligence" and "utility function" of a whole system without starting out assuming them;

- I would like to be able to compare, as much as possible, the performance of intelligences with different utility functions;

- I can't assume a priori any particular breakdown of internal tasks or "ideal" valuation of internal resources.

I would finally point out that all data about the market value of human IQ only applies to variances of intelligence within the human species. I mean, how much would you pay a chimpanzee to run your hedge fund?

Referenced by: Intelligence in Economics • Money: The Unit of Caring

Original with comments: Economic Definition of Intelligence?

Intelligence in Economics

Eliezer Yudkowsky, 30 October 2008 09:17PM

Followup to: Economic Definition of Intelligence?

After I challenged Robin to show how economic concepts can be useful in defining or measuring intelligence, Robin responded by - as I interpret it - challenging me to show why a generalized concept of "intelligence" is any use in economics.

Well, I'm not an economist (as you may have noticed) but I'll try to respond as best I can.

My primary view of the world tends to be through the lens of AI. If I talk about economics, I'm going to try to subsume it into notions like expected utility maximization (I manufacture lots of copies of something that I can use to achieve my goals) or information theory (if you manufacture lots of copies of something, my probability of seeing a copy goes up). This subsumption isn't meant to be some kind of challenge for academic supremacy - it's just what happens if you ask an AI guy an econ question.

So first, let me describe what I see when I look at economics:

I see a special case of game theory in which some interactions are highly regular and repeatable: You can take 3 units of steel and 1 unit of labor and make 1 truck that will transport 5 units of grain between Chicago and Manchester once per week, and agents can potentially do this over and over again. If the numbers aren't constant, they're at least regular - there's diminishing marginal utility, or supply/demand curves, rather than rolling random dice every time. Imagine economics if no two elements of reality were fungible - you'd just have a huge incompressible problem in non-zero-sum game theory.

This may be, for example, why we don't think of scientists writing papers that build on the work of other scientists in terms of an economy of science papers - if you turn an economist loose on science, they may measure scientist salaries paid in fungible dollars, or try to see whether scientists trade countable citations with each other. But it's much less likely to occur to them to analyze the way that units of scientific knowledge are produced from previous units plus scientific labor. Where information is concerned, two identical copies of a file are the same information as one file. So every unit of knowledge is unique, non-fungible, and so is each act of production. There isn't even a common currency that measures how much a given paper contributes to human knowledge. (I don't know what economists don't know, so do correct me if this is actually extensively studied.)

Since "intelligence" deals with an informational domain, building a bridge from it to economics isn't trivial - but where do factories come from, anyway? Why do humans get a higher return on capital than chimpanzees?

I see two basic bridges between intelligence and economics.

The first bridge is the role of intelligence in economics: the way that steel is put together into a truck involves choosing one out of an exponentially vast number of possible configurations. With a more clever configuration, you may be able to make a truck using less steel, or less labor. Intelligence also plays a role at a larger scale, in deciding whether or not to buy a truck, or where to invest money. We may even be able to talk about something akin to optimization at a macro scale, the degree to which the whole economy has put itself together in a special configuration that earns a high rate of return on investment. (Though this introduces problems for my own formulation, as I assume a central preference ordering / utility function that an economy doesn't possess - still, deflated monetary valuations seem like a good proxy.)

The second bridge is the role of economics in intelligence: if you jump up a meta-level, there are repeatable cognitive algorithms underlying the production of unique information. These cognitive algorithms use some resources that are fungible, or at least material enough that you can only use the resource on one task, creating a problem of opportunity costs. (A unit of time will be an example of this for almost any algorithm.) Thus we have Omohundro's resource balance principle, which says that the inside of an efficiently organized mind should have a common currency in expected utilons.

Says Robin:

'Eliezer has just raised the issue of how to define "intelligence", a concept he clearly wants to apply to a very wide range of possible systems. He wants a quantitative concept that is "not parochial to humans," applies to systems with very "different utility functions," and that summarizes the system's performance over a broad "not ... narrow problem domain." My main response is to note that this may just not be possible. I have no objection to looking, but it is not obvious that there is any such useful broadly-applicable "intelligence" concept.'

Well, one might run into some trouble assigning a total ordering to all intelligences, as opposed to a partial ordering. But that intelligence as a concept is useful - especially the way that I've defined it - that I must strongly defend. Our current science has advanced further on some problems than others. Right now, there is better understanding of the steps carried out to construct a car, than the cognitive algorithms that invented the unique car design. But they are both, to some degree, regular and repeatable; we don't all have different brain architectures.

I generally inveigh against focusing on relatively minor between-human variations when discussing "intelligence". It is controversial what role is played in the modern economy by such variations in whatever-IQ-tests-try-to-measure. Anyone who denies that some such a role exists would be a poor deluded fool indeed. But, on the whole, we needn't expect "the role played by IQ variations" to be at all the same sort of question as "the role played by intelligence".

You will surely find no cars, if you take away the mysterious "intelligence" that produces, from out of a vast exponential space, the information that describes one particular configuration of steel etc. constituting a car design. Without optimization to conjure certain informational patterns out of vast search spaces, the modern economy evaporates like a puff of smoke.

So you need some account of where the car design comes from.

Why should you try to give the same account of "intelligence" across different domains? When someone designs a car, or an airplane, or a hedge-fund trading strategy, aren't these different designs?

Yes, they are different informational goods.

And wasn't it a different set of skills that produced them? You can't just take a car designer and plop them down in a hedge fund.

True, but where did the different skills come from?

From going to different schools.

Where did the different schools come from?

They were built by different academic lineages, compounding knowledge upon knowledge within a line of specialization.

But where did so many different academic lineages come from? And how is this trick of "compounding knowledge" repeated over and over?

Keep moving meta, and you'll find a regularity, something repeatable: you'll find humans, with common human genes that construct common human brain architectures.

No, not every discipline puts the same relative strain on the same brain areas. But they are all using human parts, manufactured by mostly-common DNA. Not all the adult brains are the same, but they learn into unique adulthood starting from a much more regular underlying set of learning algorithms. We should expect less variance in infants than in adults.

And all the adaptations of the human brain were produced by the (relatively much structurally simpler) processes of natural selection. Without that earlier and less efficient optimization process, there wouldn't be a human brain design, and hence no human brains.

Subtract the human brains executing repeatable cognitive algorithms, and you'll have no unique adulthoods produced by learning; and no grown humans to invent the cultural concept of science; and no chains of discoveries that produce scientific lineages; and no engineers who attend schools; and no unique innovative car designs; and thus, no cars.

The moral being that you can generalize across domains, if you keep tracing back the causal chain and keep going meta.

It may be harder to talk about "intelligence" as a common factor in the full causal account of the economy, as to talk about the repeated operation that puts together many instantiations of the same car design - but there is a common factor, and the economy could hardly exist without it.

As for generalizing away from humans - well, what part of the notion of "efficient cross-domain optimization" ought to apply only to humans?

Referenced by: The Weak Inside View • Cascades, Cycles, Insight... • Permitted Possibilities, & Locality • Emotional Involvement

Original with comments: Intelligence in Economics

Mundane Magic

Eliezer Yudkowsky, 31 October 2008 04:00PM

Followup to: Joy in the Merely Real, Joy in Discovery, If You Demand Magic, Magic Won't Help

As you may recall from some months earlier, I think that part of the rationalist ethos is binding yourself emotionally to an absolutely lawful reductionistic universe - a universe containing no ontologically basic mental things such as souls or magic - and pouring all your hope and all your care into that merely real universe and its possibilities, without disappointment.

There's an old trick for combating dukkha where you make a list of things you're grateful for, like a roof over your head.

So why not make a list of abilities you have that would be amazingly cool if they were magic, or if only a few chosen individuals had them?

For example, suppose that instead of one eye, you possessed a magical second eye embedded in your forehead. And this second eye enabled you to see into the third dimension - so that you could somehow tell how far away things were - where an ordinary eye would see only a two-dimensional shadow of the true world. Only the possessors of this ability can accurately aim the legendary distance-weapons that kill at ranges far beyond a sword, or use to their fullest potential the shells of ultrafast machinery called "cars".

"Binocular vision" would be too light a term for this ability. We'll only appreciate it once it has a properly impressive name, like Mystic Eyes of Depth Perception.

So here's a list of some of my favorite magical powers:

- Vibratory Telepathy. By transmitting invisible vibrations through the very air itself, two users of this ability can share thoughts. As a result, Vibratory Telepaths can form emotional bonds much deeper than those possible to other primates.

- Psychometric Tracery. By tracing small fine lines on a surface, the Psychometric Tracer can leave impressions of emotions, history, knowledge, even the structure of other spells. This is a higher level than Vibratory Telepathy as a Psychometric Tracer can share the thoughts of long-dead Tracers who lived thousands of years earlier. By reading one Tracery and inscribing another simultaneously, Tracers can duplicate Tracings; and these replicated Tracings can even contain the detailed pattern of other spells and magics. Thus, the Tracers wield almost unimaginable power as magicians; but Tracers can get in trouble trying to use complicated Traceries that they could not have Traced themselves.

- Multidimensional Kinesis. With simple, almost unthinking acts of will, the Kinetics can cause extraordinarily complex forces to flow through small tentacles and into any physical object within touching range - not just pushes, but combinations of pushes at many points that can effectively apply torques and twists. The Kinetic ability is far subtler than it first appears: they use it not only to wield existing objects with martial precision, but also to apply forces that sculpt objects into forms more suitable for Kinetic wielding. They even create tools that extend the power of their Kinesis and enable them to sculpt ever-finer and ever-more-complicated tools, a positive feedback loop fully as impressive as it sounds.

- The Eye. The user of this ability can perceive infinitesimal traveling twists in the Force that binds matter - tiny vibrations, akin to the life-giving power of the Sun that falls on leaves, but far more subtle. A bearer of the Eye can sense objects far beyond the range of touch using the tiny disturbances they make in the Force. Mountains many days travel away can be known to them as if within arm's reach. According to the bearers of the Eye, when night falls and sunlight fails, they can sense huge fusion fires burning at unthinkable distances - though no one else has any way of verifying this. Possession of a single Eye is said to make the bearer equivalent to royalty.

And finally,

- The Ultimate Power. The user of this ability contains a smaller, imperfect echo of the entire universe, enabling them to search out paths through probability to any desired future. If this sounds like a ridiculously powerful ability, you're right - game balance goes right out the window with this one. Extremely rare among life forms, it is the sekai no ougi or "hidden technique of the world".

Nothing can oppose the Ultimate Power except the Ultimate Power. Any less-than-ultimate Power will simply be "comprehended" by the Ultimate and disrupted in some inconceivable fashion, or even absorbed into the Ultimates' own power base. For this reason the Ultimate Power is sometimes called the "master technique of techniques" or the "trump card that trumps all other trumps". The more powerful Ultimates can stretch their "comprehension" across galactic distances and aeons of time, and even perceive the bizarre laws of the hidden "world beneath the world".

Ultimates have been killed by immense natural catastrophes, or by extremely swift surprise attacks that give them no chance to use their power. But all such victories are ultimately a matter of luck - it does not confront the Ultimates on their own probability-bending level, and if they survive they will begin to bend Time to avoid future attacks.

But the Ultimate Power itself is also dangerous, and many Ultimates have been destroyed by their own powers - falling into one of the flaws in their imperfect inner echo of the world.

Stripped of weapons and armor and locked in a cell, an Ultimate is still one of the most dangerous life-forms on the planet. A sword can be broken and a limb can be cut off, but the Ultimate Power is "the power that cannot be removed without removing you".

Perhaps because this connection is so intimate, the Ultimates regard one who loses their Ultimate Power permanently - without hope of regaining it - as schiavo, or "dead while breathing". The Ultimates argue that the Ultimate Power is so important as to be a necessary part of what makes a creature an end in itself, rather than a means. The Ultimates even insist that anyone who lacks the Ultimate Power cannot begin to truly comprehend the Ultimate Power, and hence, cannot understand why the Ultimate Power is morally important - a suspiciously self-serving argument.

The users of this ability form an absolute aristocracy and treat all other life forms as their pawns.

Sequence: Joy in the Merely Real

If You Demand Magic, Magic Won't Help

The Beauty of Settled Science

Referenced by: Free to Optimize • Unteachable Excellence

Original with comments: Mundane Magic

BHTV: Jaron Lanier and Yudkowsky

Eliezer Yudkowsky, 01 November 2008 05:04PM

My Bloggingheads.tv interview with Jaron Lanier is up. Reductionism, zombies, and questions that you're not allowed to answer:

This ended up being more of me interviewing Lanier than a dialog, I'm afraid. I was a little too reluctant to interrupt. But you at least get a chance to see the probes I use, and Lanier's replies to them.

If there are any BHTV heads out there who read Overcoming Bias and have something they'd like to talk to me about, do let me or our kindly producers know.

Original with comments: BHTV: Jaron Lanier and Yudkowsky

Building Something Smarter

Eliezer Yudkowsky, 02 November 2008 05:00PM

Previously in series: Efficient Cross-Domain Optimization

Once you demystify "intelligence" far enough to think of it as searching possible chains of causality, across learned domains, in order to find actions leading to a future ranked high in a preference ordering...

...then it no longer sounds quite as strange, to think of building something "smarter" than yourself.

There's a popular conception of AI as a tape-recorder-of-thought, which only plays back knowledge given to it by the programmers - I deconstructed this in Artificial Addition, giving the example of the machine that stores the expert knowledge Plus-Of(Seven, Six) = Thirteen instead of having a CPU that does binary arithmetic.

There's multiple sources supporting this misconception:

The stereotype "intelligence as book smarts", where you memorize disconnected "facts" in class and repeat them back.

The idea that "machines do only what they are told to do", which confuses the idea of a system whose abstract laws you designed, with your exerting moment-by-moment detailed control over the system's output.

And various reductionist confusions - a computer is "mere transistors" or "only remixes what's already there" (just as Shakespeare merely regurgitated what his teachers taught him: the alphabet of English letters - all his plays are merely that).

Since the workings of human intelligence are still to some extent unknown, and will seem very mysterious indeed to one who has not studied much cognitive science, it will seem impossible for the one to imagine that a machine could contain the generators of knowledge.

The knowledge-generators and behavior-generators are black boxes, or even invisible background frameworks. So tasking the imagination to visualize "Artificial Intelligence" only shows specific answers, specific beliefs, specific behaviors, impressed into a "machine" like being stamped into clay. The frozen outputs of human intelligence, divorced of their generator and not capable of change or improvement.

You can't build Deep Blue by programming a good chess move for every possible position. First and foremost, you don't know exactly which chess positions the AI will encounter. You would have to record a specific move for zillions of positions, more than you could consider in a lifetime with your slow neurons.

But worse, even if you could record and play back "good moves", the resulting program would not play chess any better than you do. That is the peril of recording and playing back surface phenomena, rather than capturing the underlying generator.

If I want to create an AI that plays better chess than I do, I have to program a search for winning moves. I can't program in specific moves because then the chess player really won't be any better than I am. And indeed, this holds true on any level where an answer has to meet a sufficiently high standard. If you want any answer better than you could come up with yourself, you necessarily sacrifice your ability to predict the exact answer in advance - though not necessarily your ability to predict that the answer will be "good" according to a known criterion of goodness. "We never run a computer program unless we know an important fact about the output and we don't know the output," said Marcello Herreshoff.

Deep Blue played chess barely better than the world's top humans, but a heck of a lot better than its own programmers. Deep Blue's programmers had to know the rules of chess - since Deep Blue wasn't enough of a general AI to learn the rules by observation - but the programmers didn't play chess anywhere near as well as Kasparov, let alone Deep Blue.

Deep Blue's programmers didn't just capture their own chess-move generator. If they'd captured their own chess-move generator, they could have avoided the problem of programming an infinite number of chess positions. But they couldn't have beat Kasparov; they couldn't have built a program that played better chess than any human in the world.

The programmers built a better move generator - one that more powerfully steered the game toward the target of winning game positions. Deep Blue's programmers surely had some slight ability to find chess moves that aimed at this same target, but their steering ability was much weaker than Deep Blue's.

It is futile to protest that this is "paradoxical", since it actually happened.

Equally "paradoxical", but true, is that Garry Kasparov was not born with a complete library of chess moves programmed into his DNA. Kasparov invented his own moves; he was not explicitly preprogrammed by evolution to make particular moves - though natural selection did build a brain that could learn. And Deep Blue's programmers invented Deep Blue's code without evolution explicitly encoding Deep Blue's code into their genes.

Steam shovels lift more weight than humans can heft, skyscrapers are taller than their human builders, humans play better chess than natural selection, and computer programs play better chess than humans. The creation can exceed the creator. It's just a fact.

If you can understand steering-the-future, hitting-a-narrow-target as the work performed by intelligence - then, even without knowing exactly how the work gets done, it should become more imaginable that you could build something smarter than yourself.

By building something and then testing it? So that we can see that a design reaches the target faster or more reliably than our own moves, even if we don't understand how? But that's not how Deep Blue was actually built. You may recall the principle that just formulating a good hypothesis to test, usually requires far more evidence than the final test that 'verifies' it - that Einstein, in order to invent General Relativity, must have already had in hand enough evidence to isolate that one hypothesis as worth testing. Analogously, we can see that nearly all of the optimization power of human engineering must have already been exerted in coming up with good designs to test. The final selection on the basis of good results is only the icing on the cake. If you test four designs that seem like good ideas, and one of them works best, then at most 2 bits of optimization pressure can come from testing - the rest of it must be the abstract thought of the engineer.

There are those who will see it as almost a religious principle that no one can possibly know that a design will work, no matter how good the argument, until it is actually tested. Just like the belief that no one can possibly accept a scientific theory, until it is tested. But this is ultimately more of an injunction against human stupidity and overlooked flaws and optimism and self-deception and the like - so far as theoretical possibility goes, it is clearly possible to get a pretty damn good idea of which designs will work in advance of testing them.

And to say that humans are necessarily at least as good at chess as Deep Blue, since they built Deep Blue? Well, it's an important fact that we built Deep Blue, but the claim is still a nitwit sophistry. You might as well say that proteins are as smart as humans, that natural selection reacts as fast as humans, or that the laws of physics play good chess.

If you carve up the universe along its joints, you will find that there are certain things, like butterflies and humans, that bear the very identifiable design signature and limitations of evolution; and certain other things, like nuclear power plants and computers, that bear the signature and the empirical design level of human intelligence. To describe the universe well, you will have to distinguish these signatures from each other, and have separate names for "human intelligence", "evolution", "proteins", and "protons", because even if these things are related they are not at all the same.

Referenced by: Complexity and Intelligence • Recognizing Intelligence • Failure By Affective Analogy

Original with comments: Building Something Smarter

Complexity and Intelligence

Eliezer Yudkowsky, 03 November 2008 08:27PM

Followup to: Building Something Smarter , Say Not "Complexity", That Alien Message

One of the Godel-inspired challenges to the idea of self-improving minds is based on the notion of "complexity".

Now "complexity", as I've previously mentioned, is a dangerous sort of word. "Complexity" conjures up the image of a huge machine with incomprehensibly many gears inside - an impressive sort of image. Thanks to this impressiveness, "complexity" sounds like it could be explaining all sorts of things - that all sorts of phenomena could be happening because of "complexity".

It so happens that "complexity" also names another meaning, strict and mathematical: the Kolmogorov complexity of a pattern is the size of the program code of the shortest Turing machine that produces the pattern as an output, given unlimited tape as working memory.

I immediately note that this mathematical meaning, is not the same as that intuitive image that comes to mind when you say "complexity". The vast impressive-looking collection of wheels and gears? That's not what the math term means.

Suppose you ran a Turing machine with unlimited tape, so that, starting from our laws of physics, it simulated our whole universe - not just the region of space we see around us, but all regions of space and all quantum branches. (There's strong indications our universe may be effectively discrete, but if not, just calculate it out to 3^^^3 digits of precision.)

Then the "Kolmogorov complexity" of that entire universe - throughout all of space and all of time, from the Big Bang to whatever end, and all the life forms that ever evolved on Earth and all the decoherent branches of Earth and all the life-bearing planets anywhere, and all the intelligences that ever devised galactic civilizations, and all the art and all the technology and every machine ever built by those civilizations...

...would be 500 bits, or whatever the size of the true laws of physics when written out as equations on a sheet of paper.

The Kolmogorov complexity of just a single planet, like Earth, would of course be much higher than the "complexity" of the entire universe that contains it.

"Eh?" you say. "What's this?" you say. "How can a single planet contain more wheels and gears, more complexity, than the whole vast turning universe that embeds it? Wouldn't one planet contain fewer books, fewer brains, fewer species?"

But the new meaning that certain mathematicians have formulated and attached to the English word "complexity", is not like the visually impressive complexity of a confusing system of wheels and gears.

It is, rather, the size of the smallest Turing machine that unfolds into a certain pattern.

If you want to print out the entire universe from the beginning of time to the end, you only need to specify the laws of physics.

If you want to print out just Earth by itself, then it's not enough to specify the laws of physics. You also have to point to just Earth within the universe. You have to narrow it down somehow. And, in a big universe, that's going to take a lot of information. It's like the difference between giving directions to a city, and giving directions for finding a single grain of sand on one beach of one city. Indeed, with all those quantum branches existing side-by-side, it's going to take around as much information to find Earth in the universe, as to just directly specify the positions of all the atoms.

Kolmogorov complexity is the sense at which we zoom into the endless swirls of the Mandelbrot fractal and think, not "How complicated", but rather, "How simple", because the Mandelbrot set is defined using a very simple equation. But if you wanted to find a subset of the Mandelbrot set, one particular swirl, you would have to specify where to look, and that would take more bits.

That's why we use the Kolmogorov complexity in Occam's Razor to determine how "complicated" or "simple" something is. So that, when we think of the entire universe, all the stars we can see and all the implied stars we can't see, and hypothesize laws of physics standing behind it, we will think "what a simple universe" and not "what a complicated universe" - just like looking into the Mandelbrot fractal and thinking "how simple". We could never accept a theory of physics as probably true, or even remotely possible, if it got an Occam penalty the size of the universe.

As a logical consequence of the way that Kolmogorov complexity has been defined, no closed system can ever increase in Kolmogorov complexity. (At least, no closed system without a 'true randomness' source.) A program can pattern ever more interacting wheels and gears into its RAM, but nothing it does from within itself can increase "the size of the smallest computer program that unfolds into it", by definition.

Suppose, for example, that you had a computer program that defined a synthetic biology and a synthetic gene system. And this computer program ran through an artificial evolution that simulated 10^44 organisms (which is one estimate for the number of creatures who have ever lived on Earth), and subjected them to some kind of competition. And finally, after some predefined number of generations, it selected the highest-scoring organism, and printed it out. In an intuitive sense, you would (expect to) say that the best organisms on each round were getting more complicated, their biology more intricate, as the artificial biosystem evolved. But from the standpoint of Kolmogorov complexity, the final organism, even after a trillion years, is no more "complex" than the program code it takes to specify the tournament and the criterion of competition. The organism that wins is implicit in the specification of the tournament, so it can't be any more "complex" than that.

But if, on the other hand, you reached into the biology and made a few million random changes here and there, the Kolmogorov complexity of the whole system would shoot way up: anyone wanting to specify it exactly, would have to describe the random changes that you made.

I specify "random" changes, because if you made the changes with beneficence aforethought - according to some criterion of goodness - then I could just talk about the compact criterion you used to make the changes. Only random information is incompressible on average, so you have to make purposeless changes if you want to increase the Kolmogorov complexity as fast as possible.

So! As you've probably guessed, the argument against self-improvement is that since closed systems cannot increase their "complexity", the AI must look out upon the world, demanding a high-bandwidth sensory feed, in order to grow up.

If, that is, you believe that "increasing Kolmogorov complexity" is prerequisite to increasing real-world effectiveness.

(We will dispense with the idea that if system A builds system B, then system A must be "by definition" as smart as system B. This is the "Einstein's mother must have been one heck of a physicist" sophistry. Even if a future planetary ecology is in some sense "implicit" in a single self-replicating RNA strand in some tidal pool, the ecology is a lot more impressive in a real-world sense: in a given period of time it can exert larger optimization pressures and do more neat stuff.)

Now, how does one argue that "increasing Kolmogorov complexity" has something to do with increasing intelligence? Especially when small machines can unfold into whole universes, and the maximum Kolmogorov complexity is realized by random noise?

One of the other things that a closed computable system provably can't do, is solve the general halting problem - the problem of telling whether any Turing machine halts.

A similar proof shows that, if you take some given solver, and consider the maximum size bound such that the solver can solve the halting problem for all machines of that size or less, then this omniscience is bounded by at most the solver's own complexity plus a small constant.

So... isn't increasing your Kolmogorov complexity through outside sensory inputs, the key to learning to solve the halting problem for ever-larger systems?

And doesn't this show that no closed system can "self-improve"?

In a word, no.

I mean, if you were to try to write it out as logic, you'd find that one of the steps involves saying, "If you can solve all systems of complexity N, you must be of complexity greater than N (maybe minus a small constant, depending on the details of the proof). Therefore, by increasing your complexity, you increase the range of systems you can solve." This is formally a non-sequitur.

It's also a non-sequitur in practice.

I mean, sure, if we're not dealing with a closed system, you can't prove that it won't solve the halting problem. You could be looking at an external bright light in the sky that flashes on or off to reveal the halting solution.

But unless you already have that kind of mathematical ability yourself, you won't know just from looking at the light that it's giving you true solutions to the halting problem. You must have just been constructed with faith in the light, and the light must just happen to work.

(And in any computable universe, any light in the sky that you see won't happen to solve the halting problem.)

It's not easy for "sensory information" to give you justified new mathematical knowledge that you could not in principle obtain with your eyes closed.

It's not a matter, for example, of seeing written in the sky a brilliant proof, that you would never have thought of on your own. A closed system with infinite RAM can close its eyes, and write out every possible sensory experience it could have, along with its own reactions to them, that could occur within some bounded number of steps. Doing this does not increase its Kolmogorov complexity.

So the notion can't be that the environment tells you something that you recognize as a proof, but didn't think of on your own. Somehow, having that sensory experience in particular, has to increase your mathematical ability even after you perfectly imagined that experience and your own reaction to it in advance.

Could it be the healing powers of having a larger universe to live in, or other people to talk to? But you can simulate incredibly large universes - vastly larger than anything we see in our telescopes, up-arrow large - within a closed system without increasing its Kolmogorov complexity. Within that simulation you could peek at people watching the stars, and peek at people interacting with each other, and plagiarize the books they wrote about the deep wisdom that comes from being embedded in a larger world.

What justified novel mathematical knowledge - about the halting problem in particular - could you gain from a sensory experience, that you could not gain from perfectly imagining that sensory experience and your own reaction to it, nor gain from peeking in on a simulated universe that included someone having that sensory experience?

Well, you can always suppose that you were born trusting the light in the sky, and the light in the sky always happens to tell the truth.

But what's actually going on is that the non-sequitur is coming back to bite: Increasing your Kolmogorov complexity doesn't necessarily increase your ability to solve math problems. Swallowing a bucket of random bits will increase your Kolmogorov complexity too.

You aren't likely to learn any mathematics by gazing up at the external stars, that you couldn't learn from "navel-gazing" into an equally large simulated universe. Looking at the actual stars around you is good for figuring out where in the universe you are (the extra information that specifies your location) but not much good for learning new math that you couldn't learn by navel-gazing into a simulation as large as our universe.

In fact, it's just bloody hard to fundamentally increase your ability to solve math problems in a way that "no closed system can do" just by opening the system. So far as I can tell, it basically requires that the environment be magic and that you be born with faith in this fact.

Saying that a 500-state Turing machine might be able to solve all problems up to at most 500 states plus a small constant, is misleading. That's an upper bound, not a lower bound, and it comes from having a constructive way to build a specific unsolvable Turing machine out of the solver. In reality, you'd expect a 500-state Turing machine to get nowhere near solving the halting problem up to 500. I would drop dead of shock if there were a 500-state Turing machine that solved the halting problem for all the Turing machines up to 50 states. The vast majority of 500-state Turing machines that implement something that looks like a "halting problem solver" will go nowhere near 500 states (but see this comment).

Suppose you write a relatively short Turing machine that, by virtue of its unlimited working memory, creates an entire universe containing googols or up-arrows of atoms...

...and within this universe, life emerges on some planet-equivalent, and evolves, and develops intelligence, and devises science to study its ecosphere and its universe, and builds computers, and goes out into space and investigates the various physical systems that have formed, and perhaps encounters other life-forms...

...and over the course of trillions or up-arrows of years, a transgalactic intrauniversal economy develops, with conduits conducting information from one end of the universe to another (because you didn't pick a physics with inconvenient lightspeed limits), a superWeb of hyperintelligences all exchanging information...

...and finally - after a long, long time - your computer program blinks a giant message across the universe, containing a problem to be solved and a specification of how to answer back, and threatening to destroy their universe if the answer is wrong...

...then this intrauniversal civilization - and everything it's ever learned by theory or experiment over the last up-arrow years - is said to contain 400 bits of complexity, or however long the original program was.

But where did it get its math powers, from inside the closed system?

If we trace back the origins of the hypergalactic civilization, then every belief it ever adopted about math, came from updating on some observed event. That might be a star exploding, or it might be the output of a calculator, or it might be an observed event within some mind's brain... but in all cases, the update will occur because of a logic that says, "If I see this, it means that". Before you can learn, you must have the prior that structures learning. If you see something that makes you decide to change the way you learn, then you must believe that seeing X implies you should learn a different way Y. That's how it would be for superintelligences, I expect.

If you keep tracing back through that simulated universe, you arrive at something before the dawn of superintelligence - the first intelligent beings, produced by evolution. These first minds are the ones who'll look at Peano Arithmetic and think, "This has never produced a contradiction, so it probably never will - I'll go ahead and program that into my AI." These first minds are the ones who'll think, "Induction seems like it works pretty well, but how do I formalize the notion of induction?" And these first minds are the ones who'll think, "If I build a self-improving AI, how should it update itself - including changing its own updating process - from the results of observation?"

And how did the first minds get the ability to think those thoughts? From natural selection, that generated the adaptations that executed to think all those thoughts, using the simple evolutionary rule: "keep what works".

And in turn, natural selection in this universe popped out of the laws of physics.

So everything that this hypergalactic civilization ever believes about math, is really just induction in one form or another. All the evolved adaptations that do induction, produced by inductive natural selection; and all the generalizations made from experience, including generalizations about how to form better generalizations. It would all just unfold out of the inductive principle...

...running in a box sealed as tightly shut as our own universe appears to be.

And I don't see how we, in our own closed universe, are going to do any better. Even if we have the ability to look up at the stars, it's not going to give us the ability to go outside that inductive chain to obtain justified mathematical beliefs.

If you wrote that 400-bit simulated universe over the course of a couple of months using human-level intelligence and some mysterious source of unlimited computing power, then you are much more complex than that hypergalactic civilization. You take much more than 400 bits to find within the space of possibilities, because you are only one particular brain.

But y'know, I think that your mind, and the up-arrow mind of that inconceivable civilization, would still be worth distinguishing as Powers. Even if you can figure out how to ask them questions. And even if you're asking them questions by running an internal simulation, which makes it all part of your own "complexity" as defined in the math.

To locate a up-arrow-sized mind within an up-arrow-sized civilization, would require up-arrow bits - even if the entire civilization unfolded out of a 400-bit machine as compact as our own laws of physics. But which would be more powerful, that one "complex" mind, or the "simple" civilization it was part of?

None of this violates Godelian limitations. You can transmit to the hypergalactic civilization a similar Turing machine to the one that built it, and ask it how that Turing machine behaves. If you can fold a hypergalactic civilization into a 400-bit Turing machine, then even a hypergalactic civilization can confront questions about the behavior of 400-bit Turing machines that are real stumpers.

And 400 bits is going to be an overestimate. I bet there's at least one up-arrow-sized hypergalactic civilization folded into a halting Turing machine with 15 states, or something like that. If that seems unreasonable, you are not acquainted with the Busy-Beaver problem.

You can get a hell of a lot of mathematical ability out of small Turing machines that unfold into pangalactic hypercivilizations. But unfortunately, there are other small Turing machines that are hellishly difficult problems - perhaps unfolding into hypercivilizations themselves, or things even less comprehensible. So even the tremendous mathematical minds that can unfold out of small Turing machines, won't be able to solve all Turing machines up to a larger size bound. Hence no Godelian contradiction.

(I wonder: If humanity unfolded into a future civilization of infinite space and infinite time, creating descendants and hyperdescendants of unlimitedly growing size, what would be the largest Busy Beaver number ever agreed upon? 15? Maybe even as large as 20? Surely not 100 - you could encode a civilization of similar origins and equivalent power into a smaller Turing machine than that.)

Olie Lamb said: "I don't see anything good about complexity. There's nothing artful about complexity. There's nothing mystical about complexity. It's just complex." This is true even when you're talking about wheels and gears, never mind Kolmogorov complexity. It's simplicity, not complexity, that is the admirable virtue.

The real force behind this whole debate is that the word "complexity" sounds impressive and can act as an explanation for anything you don't understand. Then the word gets captured by a mathematical property that's spelled using the same letters, which happens to be provably constant for closed systems.

That, I think, is where the argument really comes from, as it rationalizes the even more primitive intuition of some blind helpless thing in a box.

This argument is promulgated even by some people who can write proofs about complexity - but frankly, I do not think they have picked up the habit of visualizing what their math symbols mean in real life. That thing Richard Feynman did, where he visualized two balls turning colors or growing hairs? I don't think they do that. On the last step of interpretation, they just make a quick appeal to the sound of the words.

But I will agree that, if the laws we know are true, then a self-improving mind which lacks sensory inputs, shouldn't be able to improve its mathematical abilities beyond the level of a up-arrow-sized civilization - for example, it shouldn't be able to solve Busy-Beaver(100).

It might perhaps be more limited than this in mere practice, if it's just running on a laptop computer or something. But if theoretical mathematical arguments about closed systems show anything, that is what they show.

Referenced by: The Weighted Majority Algorithm

Original with comments: Complexity and Intelligence

Today's Inspirational Tale

Eliezer Yudkowsky, 04 November 2008 04:15PM

At a Foresight Gathering some years ago, a Congressman was in attendance, and he spoke to us and said the following:

"Everyone in this room who's signed up for cryonics, raise your hand."

Many hands went up.

"Now everyone who knows the name of your representative in the House, raise your hand."

Fewer hands went up.

"And you wonder why you don't have any political influence."

Rationalists would likewise do well to keep this lesson in mind.

(I should also mention that voting is a Newcomblike problem. As I don't believe rational agents should defect in the 100fold iterated prisoner's dilemma, I don't buy the idea that rational agents don't vote .)

(See also Stop Voting For Nincompoops. It's more applicable to primaries than to the general election. But a vote for a losing candidate is not "thrown away"; it sends a message to mainstream candidates that you vote, but they have to work harder to appeal to your interest group to get your vote. Readers in non-swing states especially should consider what message they're sending with their vote before voting for any candidate, in any election, that they don't actually like.)

Original with comments: Today's Inspirational Tale

Hanging Out My Speaker's Shingle

Eliezer Yudkowsky, 05 November 2008 02:00PM

I was recently invited to give a talk on heuristics and biases at Jane Street Capital, one of the top proprietary trading firms ("proprietary" = they trade their own money). When I got back home, I realized that (a) I'd successfully managed to work through the trip, and (b) it'd been very pleasant mentally, a nice change of pace. (One of these days I have to blog about what I discovered at Jane Street - it turns out they've got their own rationalist subculture going.)

So I've decided to hang out my shingle as a speaker at financial companies.

You may be thinking: "Perhaps, Eliezer, this is not the best of times."

Well... I do have hopes that, among the firms interested in having me as a speaker, a higher-than-usual percentage will have come out of the crash okay. I checked recently to see if this were the case for Jane Street Capital, and it was.

But more importantly - your competitors are learning the secrets of rationality! Are you?

Or maybe I should frame it as: "Not doing too well this year? Drop the expensive big-name speakers. I can give a fascinating and useful talk and I won't charge you as much."

And just to offer a bit of a carrot - if I can monetize by speaking, I'm much less likely to try charging for access to my future writings. No promises, but something to keep in mind. So do recommend me to your friends as well.

I expect that, as I speak, the marginal value of money to my work will go down; the more I speak, the more my price will go up. If my (future) popular book on rationality becomes a hit, I'll upgrade to big-name fees. And later in my life, if all goes as planned, I'll be just plain not available.

So I'm offering you, my treasured readers, a chance to get me early. I would suggest referencing this page when requesting me as a speaker. Emails will be answered in the order they arrive.

Original with comments: Hanging Out My Speaker's Shingle

Back Up and Ask Whether, Not Why

Eliezer Yudkowsky, 06 November 2008 07:20PM

Followup to: The Bottom Line

A recent conversation reminded me of this simple, important, and difficult method:

When someone asks you "Why are you doing X?",
And you don't remember an answer previously in mind,
Do not ask yourself "Why am I doing X?".

For example, if someone asks you
"Why are you using a QWERTY keyboard?" or "Why haven't you invested in stocks?"
and you don't remember already considering this exact question and deciding it,
do not ask yourself "Why am I using a QWERTY keyboard?" or "Why aren't I invested in stocks?"

Instead, try to blank your mind - maybe not a full-fledged crisis of faith, but at least try to prevent your mind from knowing the answer immediately - and ask yourself:

"Should I do X, or not?"

Should I use a QWERTY keyboard, or not? Should I invest in stocks, or not?

When you finish considering this question, print out a traceback of the arguments that you yourself considered in order to arrive at your decision, whether that decision is to X, or not X. Those are your only real reasons, nor is it possible to arrive at a real reason in any other way.

And this is also writing advice: because I have sometimes been approached by people who say "How do I convince people to wear green shoes? I don't know how to argue it," and I reply, "Ask yourself honestly whether you should wear green shoes; then make a list of which thoughts actually move you to decide one way or another; then figure out how to explain or argue them, recursing as necessary."

Original with comments: Back Up and Ask Whether, Not Why

Recognizing Intelligence

Eliezer Yudkowsky, 07 November 2008 11:22PM

Previously in series: Building Something Smarter

Humans in Funny Suits inveighed against the problem of "aliens" on TV shows and movies who think and act like 21st-century middle-class Westerners, even if they have tentacles or exoskeletons. If you were going to seriously ask what real aliens might be like, you would try to make fewer assumptions - a difficult task when the assumptions are invisible.

I previously spoke of how you don't have to start out by assuming any particular goals, when dealing with an unknown intelligence. You can use some of your evidence to deduce the alien's goals, and then use that hypothesis to predict the alien's future achievements, thus making an epistemological profit.

But could you, in principle, recognize an alien intelligence without even hypothesizing anything about its ultimate ends - anything about the terminal values it's trying to achieve?

This sounds like it goes against my suggested definition of intelligence, or even optimization. How can you recognize something as having a strong ability to hit narrow targets in a large search space, if you have no idea what the target is?

And yet, intuitively, it seems easy to imagine a scenario in which we could recognize an alien's intelligence while having no concept whatsoever of its terminal values - having no idea where it's trying to steer the future.

Suppose I landed on an alien planet and discovered what seemed to be a highly sophisticated machine, all gleaming chrome as the stereotype demands. Can I recognize this machine as being in any sense well-designed, if I have no idea what the machine is intended to accomplish? Can I guess that the machine's makers were intelligent, without guessing their motivations?

And again, it seems like in an intuitive sense I should obviously be able to do so. I look at the cables running through the machine, and find large electrical currents passing through them, and discover that the material is a flexible high-temperature high-amperage superconductor. Dozens of gears whir rapidly, perfectly meshed...

I have no idea what the machine is doing. I don't even have a hypothesis as to what it's doing. Yet I have recognized the machine as the product of an alien intelligence. Doesn't this show that "optimization process" is not an indispensable notion to "intelligence"?

But you can't possibly recognize intelligence without at least having such a thing as a concept of "intelligence" that divides the universe into intelligent and unintelligent parts. For there to be a concept, there has to be a boundary. So what am I recognizing?

If I don't see any optimization criterion by which to judge the parts or the whole - so that, as far as I know, a random volume of air molecules or a clump of dirt would be just as good a design - then why am I focusing on this particular object and saying, "Here is a machine"? Why not say the same about a cloud or a rainstorm?

Why is it a good hypothesis to suppose that intelligence or any other optimization process played a role in selecting the form of what I see, any more than it is a good hypothesis to suppose that the dust particles in my rooms are arranged by dust elves?

Consider that gleaming chrome. Why did humans start making things out of metal? Because metal is hard; it retains its shape for a long time. So when you try to do something, and the something stays the same for a long period of time, the way-to-do-it may also stay the same for a long period of time. So you face the subproblem of creating things that keep their form and function. Metal is one solution to that subproblem.

There are no-free-lunch theorems showing the impossibility of various kinds of inference, in maximally disordered universes. In the same sense, if an alien's goals were maximally disordered, it would be unable to achieve those goals and you would be unable to detect their achievement.

But as simple a form of negentropy as regularity over time - that the alien's terminal values don't take on a new random form with each clock tick - can imply that hard metal, or some other durable substance, would be useful in a "machine" - a persistent configuration of material that helps promote a persistent goal.

The gears are a solution to the problem of transmitting mechanical forces from one place to another, which you would want to do because of the presumed economy of scale in generating the mechanical force at a central location and then distributing it. In their meshing, we recognize a force of optimization applied in the service of a recognizable instrumental value: most random gears, or random shapes turning against each other, would fail to mesh, or fly apart. Without knowing what the mechanical forces are meant to do, we recognize something that transmits mechanical force - this is why gears appear in many human artifacts, because it doesn't matter much what kind of mechanical force you need to transmit on the other end. You may still face problems like trading torque for speed, or moving mechanical force from generators to appliers.

These are not universally convergent instrumental challenges. They probably aren't even convergent with respect to maximum-entropy goal systems (which are mostly out of luck).

But relative to the space of low-entropy, highly regular goal systems - goal systems that don't pick a new utility function for every different time and every different place - that negentropy pours through the notion of "optimization" and comes out as a concentrated probability distribution over what an "alien intelligence" would do, even in the "absence of any hypothesis" about its goals.

Because the "absence of any hypothesis", in this case, does not correspond to a maxentropy distribution, but rather an ordered prior that is ready to recognize any structure that it sees. If you see the aliens making cheesecakes over and over and over again, in many different places and times, you are ready to say "the aliens like cheesecake" rather than "my, what a coincidence". Even in the absence of any notion of what the aliens are doing - whether they're making cheesecakes or paperclips or eudaimonic sentient beings - this low-entropy prior itself can pour through the notion of "optimization" and be transformed into a recognition of solved instrumental problems.

If you truly expected no order of an alien mind's goals - if you did not credit even the structured prior that lets you recognize order when you see it - then you would be unable to identify any optimization or any intelligence. Every possible configuration of matter would appear equally probable as "something the mind might design", from desk dust to rainstorms. Just another hypothesis of maximum entropy.

This doesn't mean that there's some particular identifiable thing that all alien minds want. It doesn't mean that a mind, "by definition", doesn't change its goals over time. Just that if there were an "agent" whose goals were pure snow on a television screen, its acts would be the same.

Like thermodynamics, cognition is about flows of order. An ordered outcome needs negentropy to fuel it. Likewise, where we expect or recognize a thing, even so lofty and abstract as "intelligence", we must have ordered beliefs to fuel our anticipation. It's all part of the great game, Follow-the-Negentropy.

Referenced by: Lawful Creativity

Original with comments: Recognizing Intelligence

Lawful Creativity

Eliezer Yudkowsky, 08 November 2008 07:54PM

Previously in Series: Recognizing Intelligence

Creativity, we've all been told, is about Jumping Out Of The System, as Hofstadter calls it (JOOTSing for short). Questioned assumptions, violated expectations.

Fire is dangerous: the rule of fire is to run away from it. What must have gone through the mind of the first hominid to domesticate fire? The rule of milk is that it spoils quickly and then you can't drink it - who first turned milk into cheese? The rule of computers is that they're made with vacuum tubes, fill a room and are so expensive that only corporations can own them. Wasn't the transistor a surprise...

Who, then, could put laws on creativity? Who could bound it, who could circumscribe it, even with a concept boundary that distinguishes "creativity" from "not creativity"? No matter what system you try to lay down, mightn't a more clever person JOOTS right out of it? If you say "This, this, and this is 'creative'" aren't you just making up the sort of rule that creative minds love to violate?

Why, look at all the rules that smart people have violated throughout history, to the enormous profit of humanity. Indeed, the most amazing acts of creativity are those that violate the rules that we would least expect to be violated.

Is there not even creativity on the level of how to think? Wasn't the invention of Science a creative act that violated old beliefs about rationality? Who, then, can lay down a law of creativity?

But there is one law of creativity which cannot be violated...

Ordinarily, if you took a horse-and-buggy, unstrapped the horses, put a large amount of highly combustible fluid on board, and then set fire to the fluid, you would expect the buggy to burn. You certainly wouldn't expect the buggy to move forward at a high rate of speed, for the convenience of its passengers.

How unexpected was the internal combustion engine! How surprising! What a creative act, to violate the rule that you shouldn't start a fire inside your vehicle!

But now suppose that I unstrapped the horses from a buggy, put gasoline in the buggy, and set fire to the gasoline, and it did just explode.

Then there would be no element of "creative surprise" about that. More experienced engineers would just shake their heads wisely and say, "That's why we use horses, kiddo."

Creativity is surprising - but not just any kind of surprise counts as a creative surprise. Suppose I set up an experiment involving a quantum event of very low amplitude, such that the macroscopic probability is a million to one. If the event is actually observed to occur, it is a happenstance of extremely low probability, and in that sense surprising. But it is not a creative surprise. Surprisingness is not a sufficient condition for creativity.

So what kind of surprise is it, that creates the unexpected "shock" of creativity?

In information theory, the more unexpected an event is, the longer the message it takes to send it - to conserve bandwidth, you use the shortest messages for the most common events.

So do we reason that the most unexpected events, convey the most information, and hence the most surprising acts are those that give us a pleasant shock of creativity - the feeling of suddenly absorbing new information?

This contains a grain of truth, I think, but not the whole truth: the million-to-one quantum event would also require a 20-bit message to send, but it wouldn't convey a pleasant shock of creativity, any more than a random sequence of 20 coinflips.

Rather, the creative surprise is the idea that ranks high in your preference ordering but low in your search ordering.

If, before anyone had thought of an internal combustion engine (which predates cars, of course) I had asked some surprisingly probability-literate engineer to write out a vocabulary for describing effective vehicles, it would contain short symbols for horses, long symbols for flammable fluid, and maybe some kind of extra generalization that says "With probability 99.99%, a vehicle should not be on fire" so that you need to use a special 14-bit prefix for vehicles that violate this generalization.

So when I now send this past engineer a description of an automobile, he experiences the shock of getting a large amount of useful information - a design that would have taken a long time for him to find, in the implicit search ordering he set up - a design that occupies a region of low density in his prior distribution for where good designs are to be found in the design space. And even the added "absurdity" shock of seeing a generalization violated - not a generalization about physical laws, but a generalization about which designs are effective or ineffective.

But the vehicle still goes somewhere - that part hasn't changed.

What if I try to explain about telepresence and virtual offices, so that you don't even need a car?

But you're still trying to talk to people, or get work done with people - you've just found a more effective means to that end, than travel.

A car is a more effective means of travel, a computer is a more effective means than travel. But there's still some end. There's some criterion that makes the solution a "solution". There's some criterion that makes the unusual reply, unusually good. Otherwise any part of the design space would be as good as any other.

An amazing creative solution has to obey at least one law, the criterion that makes it a "solution". It's the one box you can't step outside: No optimization without goals.

The pleasant shock of witnessing Art arises from the constraints of Art - from watching a skillful archer send an arrow into an exceedingly narrow target. Static on a television screen is not beautiful, it is noise.

In the strange domain known as Modern Art, people sometimes claim that their goal is to break all the rules, even the rule that Art has to hit some kind of target. They put up a blank square of canvas, and call it a painting. And by now that is considered staid and boring Modern Art, because a blank square of canvas still hangs on the wall and has a frame. What about a heap of garbage? That can also be Modern Art! Surely, this demonstrates that true creativity knows no rules, and even no goals...

But the rules are still there, though unspoken. I could submit a realistic landscape painting as Modern Art, and this would be rejected because it violates the rule that Modern Art cannot delight the untrained senses of a mere novice.

Or better yet, if a heap of garbage can be Modern Art, then I'll claim that someone else's heap of garbage is my work of Modern Art - boldly defying the convention that I need to produce something for it to count as my artwork. Or what about the pattern of dust particles on my desk? Isn't that Art?

Flushed with triumph, I present to you an even bolder, more convention-defying work of Modern Art - a stunning, outrageous piece of performance art that, in fact, I never performed. I am defying the foolish convention that I need to actually perform my performance art for it to count as Art.

Now, up to this point, you probably could still get a grant from the National Endowment for the Arts, and get sophisticated critics to discuss your shocking, outrageous non-work, which boldly violates the convention that art must be real rather than imaginary.

But now suppose that you go one step further, and refuse to tell anyone that you have performed your work of non-Art. You even refuse to apply for an NEA grant. It is the work of Modern Art that never happened and that no one knows never happened; it exists only as my concept of what I am supposed not to conceptualize. Better yet, I will say that my Modern Art is your non-conception of something that you are not conceptualizing. Here is the ultimate work of Modern Art, that truly defies all rules: It isn't mine, it isn't real, and no one knows it exists...

And this ultimate rulebreaker you could not pass off as Modern Art, even if NEA grant committees knew that no one knew it existed. For one thing, they would realize that you were making fun of them - and that is an unspoken rule of Modern Art that no one dares violate. You must take yourself seriously. You must break the surface rules in a way that allows sophisticated critics to praise your boldness and defiance with a straight face. This is the unwritten real goal, and if it is not achieved, all efforts are for naught. Whatever gets sophisticated critics to praise your rule-breaking is good Modern Art, and whatever fails in this end is poor Modern Art. Within that unalterable constraint, you can use whatever creative means you like.

But doesn't creative engineering sometimes involve altering your goals? First my goal was to try and figure out how to build a vehicle using horses; now my goal is to build a vehicle using fire...

Creativity clearly involves altering my local intentions, my what-I'm-trying-to-do-next. I begin by intending to use horses, to build a vehicle, to drive to the supermarket, to buy food, to eat food, so that I don't starve to death, because I prefer being alive to starving to death. I may creatively use fire, instead of horses; creatively walk, instead of driving; creatively drive to a nearby restaurant, instead of a supermarket; creatively grow my own vegetables, instead of buying them; or even creatively devise a way to run my body on electricity, instead of chemical energy...

But what does not count as "creativity" is creatively preferring to starve to death, rather than eating. This "solution" does not strike me as very impressive; it involves no effort, no intelligence, and no surprise when it comes to looking at the result. If this is someone's idea of how to break all the rules, they would become pretty easy to predict.

Are there cases where you genuinely want to change your preferences? You may look back in your life and find that your moral beliefs have changed over decades, and that you count this as "moral progress". Civilizations also change their morals over time. In the seventeenth century, people used to think it was okay to enslave people with differently colored skin; and now we elect them President.

But you might guess by now, you might somehow intuit, that if these moral changes seem interesting and important and vital and indispensable, then not just any change would suffice. If there's no criterion, no target, no way of choosing - then your current point in state space is just as good as any other point, no more, no less; and you might as well keep your current state, unchanging, forever.

Every surprisingly creative Jump-Out-Of-The-System needs a criterion that makes it surprisingly good, some fitness metric that it matches. This criterion, itself, supplies the order in our beliefs that lets us recognize an act of "creativity" despite our surprise. Just as recognizing intelligence requires at least some belief about that intelligence's goals, however abstract.

One might wish to reconsider, in light of this principle, such notions as "free will that is not constrained by anything"; or the necessary conditions for our discussions of what is "right" to have some kind of meaning.

There is an oft-repeated cliche of Deep Wisdom which says something along the lines of "intelligence is balanced between Order and Chaos", as if cognitive science were a fantasy novel written by Roger Zelazny. Logic as Order, following the rules. Creativity as Chaos, violating the rules. And so you can try to understand logic, but you are blocked when it comes to creativity - and of course you could build a logical computer but not a creative one - and of course 'rationalists' can only use the Order side of the equation, and can never become whole people; because Art requires an element of irrationality, just like e.g. emotion.

And I think that despite its poetic appeal, that whole cosmological mythology is just flat wrong. There's just various forms of regularity, of negentropy, where all the structure and all the beauty live. And on the other side is what's left over - the static on the television screen, the heat bath, the noise.

I shall be developing this startling thesis further in future posts.

Referenced by: Lawful Uncertainty • Failure By Affective Analogy • Free to Optimize • Changing Emotions • In Praise of Boredom • Value is Fragile • The Thing That I Protect

Original with comments: Lawful Creativity

Ask OB: Leaving the Fold

Eliezer Yudkowsky, 09 November 2008 06:08PM

Followup to: Crisis of Faith

I thought this comment from "Jo" deserved a bump to the front page:

"So here I am having been raised in the Christian faith and trying not to freak out over the past few weeks because I've finally begun to wonder whether I believe things just because I was raised with them. Our family is surrounded by genuinely wonderful people who have poured their talents into us since we were teenagers, and our social structure and business rests on the tenets of what we believe. I've been trying to work out how I can 'clear the decks' and then rebuild with whatever is worth keeping, yet it's so foundational that it will affect my marriage (to a pretty special man) and my daughters who, of course, have also been raised to walk the Christian path.

Is there anyone who's been in this position - really, really invested in a faith and then walked away?"

Handling this kind of situation has to count as part of the art. But I haven't gone through anything like this. Can anyone with experience advise Jo on what to expect, what to do, and what not to do?

Original with comments: Ask OB: Leaving the Fold

Lawful Uncertainty

Eliezer Yudkowsky, 10 November 2008 09:06PM

Previously in series: Lawful Creativity

From Robyn Dawes, Rational Choice in an Uncertain World:

"Many psychological experiments were conducted in the late 1950s and early 1960s in which subjects were asked to predict the outcome of an event that had a random component but yet had base-rate predictability - for example, subjects were asked to predict whether the next card the experiment turned over would be red or blue in a context in which 70% of the cards were blue, but in which the sequence of red and blue cards was totally random.

In such a situation, the strategy that will yield the highest proportion of success is to predict the more common event. For example, if 70% of the cards are blue, then predicting blue on every trial yields a 70% success rate.

What subjects tended to do instead, however, was match probabilities - that is, predict the more probable event with the relative frequency with which it occurred. For example, subjects tended to predict 70% of the time that the blue card would occur and 30% of the time that the red card would occur. Such a strategy yields a 58% success rate, because the subjects are correct 70% of the time when the blue card occurs (which happens with probability .70) and 30% of the time when the red card occurs (which happens with probability .30); .70 * .70 + .30 * .30 = .58.

In fact, subjects predict the more frequent event with a slightly higher probability than that with which it occurs, but do not come close to predicting its occurrence 100% of the time, even when they are paid for the accuracy of their predictions... For example, subjects who were paid a nickel for each correct prediction over a thousand trials... predicted [the more common event] 76% of the time."

(Dawes cites: Tversky, A. and Edwards, W. 1966. Information versus reward in binary choice. Journal of Experimental Psychology, 71, 680-683.)

Do not think that this experiment is about a minor flaw in gambling strategies. It compactly illustrates the most important idea in all of rationality.

Of this experiment Dawes goes on to say, "Despite feedback through a thousand trials, subjects cannot bring themselves to believe that the situation is one in which they cannot predict."

But the error must go deeper than that. Even if subjects think they've come up with a hypothesis, they don't have to actually bet on that prediction in order to test their hypothesis. They can say, "Now if this hypothesis is correct, the next card will be red" - and then just bet on blue.

I wouldn't fault a subject for continuing to invent hypotheses - how could they know the sequence is truly beyond their ability to predict? But I would fault a subject for betting on the guesses, when this wasn't necessary to gather information, and literally hundreds of earlier guesses had been disconfirmed.

Can even a human be that overconfident?

I would suspect that something simpler is going on - that the all-blue strategy just didn't occur to the subjects.

People see a mix of mostly blue cards with some red, and suppose that the optimal betting strategy must be a mix of mostly blue cards with some red.

It is a counterintuitive idea that, given incomplete information, the optimal betting strategy does not resemble a typical sequence of cards.

It is a counterintuitive idea that the optimal strategy is to behave lawfully, even in an environment that has random elements.

It seems like your behavior ought to be unpredictable, just like the environment - but no! A random key does not open a random lock just because they are "both random".

You don't fight fire with fire, you fight fire with water. But this thought involves an extra step, a new concept not directly activated by the problem statement, and so it's not the first idea that comes to mind.

In the dilemma of the blue and red cards, our partial knowledge tells us - on each and every round - that the best bet is blue. This advice of our partial knowledge is the same on each and every round. If 30% of the time we go against our partial knowledge and bet on red instead, then we will do worse thereby - because now we're being outright stupid, betting on what we know is the less probable outcome.

If you bet on red every round, you would do as badly as you could possibly do; you would be 100% stupid. If you bet on red 30% of the time, faced with 30% red cards, then you're making yourself 30% stupid. Given incomplete information, the optimal betting strategy does not resemble a typical sequence of cards.

When your knowledge is incomplete - meaning that the world will seem to you to have an element of randomness - randomizing your actions doesn't solve the problem. Randomizing your actions takes you further from the target, not closer. In a world already foggy, throwing away your intelligence just makes things worse.

It is a counterintuitive idea that the optimal strategy can be to think lawfully, even under conditions of uncertainty.

And so there are not many rationalists, for most who perceive a chaotic world will try to fight chaos with chaos. You have to take an extra step, and think of something that doesn't pop right into your mind, in order to imagine fighting fire with something that is not itself fire.

You have heard the unenlightened ones say, "Rationality works fine for dealing with rational people, but the world isn't rational." But faced with an irrational opponent, throwing away your own reason is not going to help you. There are lawful forms of thought that still generate the best response, even when faced with an opponent who breaks those laws. Decision theory does not burst into flames and die when faced with an opponent who disobeys decision theory.

This is no more obvious than the idea of betting all blue, faced with a sequence of both blue and red cards. But each bet that you make on red is an expected loss, and so too with every departure from the Way in your own thinking.

How many Star Trek episodes are thus refuted? How many theories of AI?

You don't fight fire with fire - any more than you fight irrationality with irrationality - nor fight randomness with randomness - nor fight chaos with chaos - nor fight unpredictability with unpredictability - nor survive in an uncertain universe with uncertain behaviors - nor unravel an environment of unknown laws with unknown laws of reasoning - nor analyze a changing world with a changing algebra of probability.

Referenced by: Worse Than Random • The Weighted Majority Algorithm • Failure By Affective Analogy

Original with comments: Lawful Uncertainty

Worse Than Random

Eliezer Yudkowsky, 11 November 2008 07:01PM

Previously in series: Lawful Uncertainty

You may have noticed a certain trend in recent posts: I've been arguing that randomness hath no power, that there is no beauty in entropy, nor yet strength from noise.

If one were to formalize the argument, it would probably run something like this: that if you define optimization as previously suggested, then sheer randomness will generate something that seems to have 12 bits of optimization, only by trying 4096 times; or 100 bits of optimization, only by trying 10^30 times.

This may not sound like a profound insight, since it is true by definition. But consider - how many comic books talk about "mutation" as if it were a source of power? Mutation is random. It's the selection part, not the mutation part, that explains the trends of evolution.

Or you may have heard people talking about "emergence" as if it could explain complex, functional orders. People will say that the function of an ant colony emerges - as if, starting from ants that had been selected only to function as solitary individuals, the ants got together in a group for the first time and the ant colony popped right out. But ant colonies have been selected on as colonies by evolution. Optimization didn't just magically happen when the ants came together.

And you may have heard that certain algorithms in Artificial Intelligence work better when we inject randomness into them.

Is that even possible? How can you extract useful work from entropy?

But it is possible in theory, since you can have things that are anti-optimized. Say, the average state has utility -10, but the current state has an unusually low utility of -100. So in this case, a random jump has an expected benefit. If you happen to be standing in the middle of a lava pit, running around at random is better than staying in the same place. (Not best, but better.)

A given AI algorithm can do better when randomness is injected, provided that some step of the unrandomized algorithm is doing worse than random.

Imagine that we're trying to solve a pushbutton combination lock with 20 numbers and four steps - 160,000 possible combinations. And we try the following algorithm for opening it:

1. Enter 0-0-0-0 into the lock.

2. If the lock opens, return with SUCCESS.

3. If the lock remains closed, go to step 1.

Obviously we can improve this algorithm by substituting "Enter a random combination" on the first step.

If we were to try and explain in words why this works, a description might go something like this: "When we first try 0-0-0-0 it has the same chance of working (so far as we know) as any other combination. But if it doesn't work, it would be stupid to try it again, because now we know that 0-0-0-0 doesn't work."

The first key idea is that, after trying 0-0-0-0, we learn something - we acquire new knowledge, which should then affect how we plan to continue from there. This is knowledge, quite a different thing from randomness...

What exactly have we learned? We've learned that 0-0-0-0 doesn't work; or to put it another way, given that 0-0-0-0 failed on the first try, the conditional probability of it working on the second try, is negligible.

Consider your probability distribution over all the possible combinations: Your probability distribution starts out in a state of maximum entropy, with all 160,000 combinations having a 1/160,000 probability of working. After you try 0-0-0-0, you have a new probability distribution, which has slightly less entropy; 0-0-0-0 has an infinitesimal probability of working, and the remaining 159,999 possibilities each have a 1/159,999 probability of working. To try 0-0-0-0 again would now be stupid - the expected utility of trying 0-0-0-0 is less than average; the vast majority of potential actions now have higher expected utility than does 0-0-0-0. An algorithm that tries 0-0-0-0 again would do worse than random, and we can improve the algorithm by randomizing it.

One may also consider an algorithm as a sequence of tries: The "unrandomized algorithm" describes the sequence of tries 0-0-0-0, 0-0-0-0, 0-0-0-0... and this sequence of tries is a special sequence that has far-below-average expected utility in the space of all possible sequences. Thus we can improve on this sequence by selecting a random sequence instead.

Or imagine that the combination changes every second. In this case, 0-0-0-0, 0-0-0-0 is just as good as the randomized algorithm - no better and no worse. What this shows you is that the supposedly "random" algorithm is "better" relative to a known regularity of the lock - that the combination is constant on each try. Or to be precise, the reason the random algorithm does predictably better than the stupid one is that the stupid algorithm is "stupid" relative to a known regularity of the lock.

In other words, in order to say that the random algorithm is an "improvement", we must have used specific knowledge about the lock to realize that the unrandomized algorithm is worse-than-average. Having realized this, can we reflect further on our information, and take full advantage of our knowledge to do better-than-average?

The random lockpicker is still not optimal - it does not take full advantage of the knowledge we have acquired. A random algorithm might randomly try 0-0-0-0 again; it's not impossible, but it could happen. The longer the random algorithm runs, the more likely it is to try the same combination twice; and if the random algorithm is sufficiently unlucky, it might still fail to solve the lock after millions of tries. We can take full advantage of all our knowledge by using an algorithm that systematically tries 0-0-0-0, 0-0-0-1, 0-0-0-2... This algorithm is guaranteed not to repeat itself, and will find the solution in bounded time. Considering the algorithm as a sequence of tries, no other sequence in sequence-space is expected to do better, given our initial knowledge. (Any other nonrepeating sequence is equally good; but nonrepeating sequences are rare in the space of all possible sequences.)

A combination dial often has a tolerance of 2 in either direction. 20-45-35 will open a lock set to 22-44-33. In this case, the algorithm that tries 0-1-0, 0-2-0, et cetera, ends up being stupid again; a randomized algorithm will (usually) work better. But an algorithm that tries 0-5-0, 0-10-0, 0-10-5, will work better still.

Sometimes it is too expensive to take advantage of all the knowledge that we could, in theory, acquire from previous tests. Moreover, a complete enumeration or interval-skipping algorithm would still end up being stupid. In this case, computer scientists often use a cheap pseudo-random algorithm, because the computational cost of using our knowledge exceeds the benefit to be gained from using it. This does not show the power of randomness, but, rather, the predictable stupidity of certain specific deterministic algorithms on that particular problem. Remember, the pseudo-random algorithm is also deterministic! But the deterministic pseudo-random algorithm doesn't belong to the class of algorithms that are predictably stupid (do much worse than average).

There are also subtler ways for adding noise to improve algorithms. For example, there are neural network training algorithms that work better if you simulate noise in the neurons. On this occasion it is especially tempting to say something like:

"Lo! When we make our artificial neurons noisy, just like biological neurons, they work better! Behold the healing life-force of entropy!"

What might actually be happening - for example - is that the network training algorithm, operating on noiseless neurons, would vastly overfit the data.

If you expose the noiseless network to the series of coinflips "HTTTHHTTH"... the training algorithm will say the equivalent of, "I bet this coin was specially designed to produce HTTTHHTTH every time it's flipped!" instead of "This coin probably alternates randomly between heads and tails." A hypothesis overfitted to the data does not generalize. On the other hand, when we add noise to the neurons and then try training them again, they can no longer fit the data precisely, so instead they settle into a simpler hypothesis like "This coin alternates randomly between heads and tails." Note that this requires us - the programmers - to know in advance that probabilistic hypotheses are more likely to be true.

Or here's another way of looking at it: A neural network algorithm typically looks at a set of training instances, tweaks the units and their connections based on the training instances, and in this way tries to "stamp" the experience onto itself. But the algorithms which do the stamping are often poorly understood, and it is possible to stamp too hard. If this mistake has already been made, then blurring the sensory information, or blurring the training algorithm, or blurring the units, can partially cancel out the "overlearning".

Here's a simplified example of a similar (but not identical) case: Imagine that the environment deals us a random mix of cards, 70% blue and 30% red. But instead of just predicting "blue" or "red", we have to assign a quantitative probability to seeing blue - and the scoring rule for our performance is one that elicits an honest estimate; if the actual frequency is 70% blue cards, we do best by replying "70%", not 60% or 80%. ("Proper scoring rule.")

If you don't know in advance the frequency of blue and red cards, one way to handle the problem would be to have a blue unit and a red unit, both wired to an output unit. The blue unit sends signals with a positive effect that make the target unit more likely to fire; the red unit inhibits its targets - just like the excitatory and inhibitory synapses in the human brain! (Or an earthworm's brain, for that matter...)

Each time we see a blue card in the training data, the training algorithm increases the strength of the (excitatory) synapse from the blue unit to the output unit; and each time we see a red card, the training algorithm strengthens the (inhibitory) synapse from the red unit to the output unit.

But wait! We haven't said why the blue or red units might fire in the first place. So we'll add two more excitatory units that spike randomly, one connected to the blue unit and one connected to red unit. This simulates the natural background noise present in the human brain (or an earthworm's brain).

Finally, the spiking frequency of the output unit becomes the predicted probability that the next card is blue.

As you can see - assuming you haven't lost track of all the complexity - this neural network learns to predict whether blue cards or red cards are more common in the mix. Just like a human brain!

At least that's the theory. However, when we boot up the neural network and give it a hundred training examples with 70 blue and 30 red cards, it ends up predicting a 90% probability that each card will be blue. Now, there are several things that could go wrong with a system this complex; but my own first impulse would be to guess that the training algorithm is too strongly adjusting the synaptic weight from the blue or red unit to the output unit on each training instance. The training algorithm needs to shift a little less far - alter the synaptic weights a little less.

But the programmer of the neural network comes up with a different hypothesis: the problem is that there's no noise in the input. This is biologically unrealistic; real organisms do not have perfect vision or perfect information about the environment. So the programmer shuffles a few randomly generated blue and red cards (50% probability of each) into the training sequences. Then the programmer adjusts the noise level until the network finally ends up predicting blue with 70% probability. And it turns out that using almost the same noise level (with just a bit of further tweaking), the improved training algorithm can learn to assign the right probability to sequences of 80% blue or 60% blue cards.

Success! We have found the Right Amount of Noise.

Of course this success comes with certain disadvantages. For example, maybe the blue and red cards are predictable, in the sense of coming in a learnable sequence. Maybe the sequence is 7 blue cards followed by 3 red cards. If we mix noise into the sensory data, we may never notice this important regularity, or learn it imperfectly... but that's the price you pay for biological realism.

What's really happening is that the training algorithm moves too far given its data, and adulterating noise with the data diminishes the impact of the data. The two errors partially cancel out, but at the cost of a nontrivial loss in what we could, in principle, have learned from the data. It would be better to adjust the training algorithm and keep the data clean.

This is an extremely oversimplified example, but it is not a total strawman. The scenario seems silly only because it is simplified to the point where you can clearly see what is going wrong. Make the neural network a lot more complicated, and the solution of adding noise to the inputs might sound a lot more plausible. While some neural network algorithms are well-understood mathematically, others are not. In particular, systems crafted with the aim of biological realism are often not well-understood.

But it is an inherently odd proposition that you can get a better picture of the environment by adding noise to your sensory information - by deliberately throwing away your sensory acuity. This can only degrade the mutual information between yourself and the environment. It can only diminish what in principle can be extracted from the data. And this is as true for every step of the computation, as it is for the eyes themselves. The only way that adding random noise will help is if some step of the sensory processing is doing worse than random.

Now it is certainly possible to design an imperfect reasoner that only works in the presence of an accustomed noise level. Biological systems are unable to avoid noise, and therefore adapt to overcome noise. Subtract the noise, and mechanisms adapted to the presence of noise may do strange things.

Biological systems are often fragile under conditions that have no evolutionary precedent in the ancestral environment. If somehow the Earth's gravity decreased, then birds might become unstable, lurching up in the air as their wings overcompensated for the now-decreased gravity. But this doesn't mean that stronger gravity helps birds fly better. Gravity is still the difficult challenge that a bird's wings work to overcome - even though birds are now adapted to gravity as an invariant.

What about hill-climbing, simulated annealing, or genetic algorithms? These AI algorithms are local search techniques that randomly investigate some of their nearest neighbors. If an investigated neighbor is superior to the current position, the algorithm jumps there. (Or sometimes probabilistically jumps to a neighbor with probability determined by the difference between neighbor goodness and current goodness.) Are these techniques drawing on the power of noise?

Local search algorithms take advantage of the regularity of the search space - that if you find a good point in the search space, its neighborhood of closely similar points is a likely place to search for a slightly better neighbor. And then this neighbor, in turn, is a likely place to search for a still better neighbor; and so on. To the extent this regularity of the search space breaks down, hill-climbing algorithms will perform poorly. If the neighbors of a good point are no more likely to be good than randomly selected points, then a hill-climbing algorithm simply won't work.

But still, doesn't a local search algorithm need to make random changes to the current point in order to generate neighbors for evaluation? Not necessarily; some local search algorithms systematically generate all possible neighbors, and select the best one. These greedy algorithms work fine for some problems, but on other problems it has been found that greedy local algorithms get stuck in local minima. The next step up from greedy local algorithms, in terms of added randomness, is random-restart hill-climbing - as soon as we find a local maximum, we restart someplace random, and repeat this process a number of times. For our final solution, we return the best local maximum found when time runs out. Random-restart hill-climbing is surprisingly useful; it can easily solve some problem classes where any individual starting point is unlikely to lead to a global maximum or acceptable solution, but it is likely that at least one of a thousand individual starting points will lead to the global maximum or acceptable solution.

The non-randomly-restarting, greedy, local-maximum-grabbing algorithm, is "stupid" at the stage where it gets stuck in a local maximum. Once you find a local maximum, you know you're not going to do better by greedy local search - so you may as well try something else with your time. Picking a random point and starting again is drastic, but it's not as stupid as searching the neighbors of a particular local maximum over and over again. (Biological species often do get stuck in local optima. Evolution, being unintelligent, has no mind with which to "notice" when it is testing the same gene complexes over and over.)

Even more stupid is picking a particular starting point, and then evaluating its fitness over and over again, without even searching its neighbors. This is the lockpicker who goes on trying 0-0-0-0 forever.

Hill-climbing search is not so much a little bit randomized compared to the completely stupid lockpicker, as almost entirely nonrandomized compared to a completely ignorant searcher. We search only the local neighborhood, rather than selecting a random point from the entire state space. That probability distribution has been narrowed enormously, relative to the overall state space. This exploits the belief - the knowledge, if the belief is correct - that a good point probably has good neighbors.

You can imagine splitting a hill-climbing algorithm into components that are "deterministic" (or rather, knowledge-exploiting) and "randomized" (the leftover ignorance).

A programmer writing a probabilistic hill-climber will use some formula to assign probabilities to each neighbor, as a function of the neighbor's fitness. For example, a neighbor with a fitness of 60 might have probability 80% of being selected, while other neighbors with fitnesses of 55, 52, and 40 might have selection probabilities of 10%, 9%, and 1%. The programmer writes a deterministic algorithm, a fixed formula, that produces these numbers - 80, 10, 9, and 1.

What about the actual job of making a random selection at these probabilities? Usually the programmer will hand that job off to someone else's pseudo-random algorithm - most language's standard libraries contain a standard pseudo-random algorithm; there's no need to write your own.

If the hill-climber doesn't seem to work well, the programmer tweaks the deterministic part of the algorithm, the part that assigns these fixed numbers 80, 10, 9, and 1. The programmer does not say - "I bet these probabilities are right, but I need a source that's even more random, like a thermal noise generator, instead of this merely pseudo-random algorithm that is ultimately deterministic!" The programmer does not go in search of better noise.

It is theoretically possible for a poorly designed "pseudo-random algorithm" to be stupid relative to the search space; for example, it might always jump in the same direction. But the "pseudo-random algorithm" has to be really shoddy for that to happen. You're only likely to get stuck with that problem if you reinvent the wheel instead of using a standard, off-the-shelf solution. A decent pseudo-random algorithm works just as well as a thermal noise source on optimization problems. It is possible (though difficult) for an exceptionally poor noise source to be exceptionally stupid on the problem, but you cannot do exceptionally well by finding a noise source that is exceptionally random. The power comes from the knowledge - the deterministic formula that assigns a fixed probability distribution. It does not reside in the remaining ignorance.

And that's why I always say that the power of natural selection comes from the selection part, not the mutation part.

As a general principle, on any problem for which you know that a particular unrandomized algorithm is unusually stupid - so that a randomized algorithm seems wiser - you should be able to use the same knowledge to produce a superior derandomized algorithm. If nothing else seems obvious, just avoid outputs that look "unrandomized"! If you know something is stupid, deliberately avoid it! (There are exceptions to this rule, but they have to do with defeating cryptographic adversaries - that is, preventing someone else's intelligence from working on you. Certainly entropy can act as an antidote to intelligence!) And of course there are very common practical exceptions whenever the computational cost of using all our knowledge exceeds the marginal benefit...

Still you should find, as a general principle, that randomness hath no power: there is no beauty in entropy, nor strength from noise.

Referenced by: The Weighted Majority Algorithm • Selling Nonapples • The First World Takeover

Original with comments: Worse Than Random

The Weighted Majority Algorithm

Eliezer Yudkowsky, 12 November 2008 11:19PM

Followup to: Worse Than Random, Trust In Bayes

In the wider field of Artificial Intelligence, it is not universally agreed and acknowledged that noise hath no power. Indeed, the conventional view in machine learning is that randomized algorithms sometimes perform better than unrandomized counterparts and there is nothing peculiar about this. Thus, reading an ordinary paper in the AI literature, you may suddenly run across a remark: "There is also an improved version of this algorithm, which takes advantage of randomization..."

Now for myself I will be instantly suspicious; I shall inspect the math for reasons why the unrandomized algorithm is being somewhere stupid, or why the randomized algorithm has a hidden disadvantage. I will look for something peculiar enough to explain the peculiar circumstance of a randomized algorithm somehow doing better.

I am not completely alone in this view. E. T. Jaynes, I found, was of the same mind: "It appears to be a quite general principle that, whenever there is a randomized way of doing something, then there is a nonrandomized way that delivers better performance but requires more thought." Apparently there's now a small cottage industry in derandomizing algorithms. But so far as I know, it is not yet the majority, mainstream view that "we can improve this algorithm by randomizing it" is an extremely suspicious thing to say.

Let us now consider a specific example - a mainstream AI algorithm where there is, apparently, a mathematical proof that the randomized version performs better. By showing how subtle the gotcha can be, I hope to convince you that, even if you run across a case where the randomized algorithm is widely believed to perform better, and you can't find the gotcha yourself, you should nonetheless trust that there's a gotcha to be found.

For our particular example we shall examine the weighted majority algorithm, first introduced by Littlestone and Warmuth, but as a substantially more readable online reference I will use section 2 of Blum's On-Line Algorithms in Machine Learning. (Blum has an easier-to-read, more concrete version of the proofs; and the unrandomized and randomized algorithms are presented side-by-side rather than far part.)

The weighted majority algorithm is an ensemble method - a way to combine the advice from several other algorithms or hypotheses, called "experts". Experts strive to classify a set of environmental instances into "positive" and "negative" categories; each expert produces a prediction for each instance. For example, each expert might look at a photo of a forest, and try to determine whether or not there is a camouflaged tank in the forest. Expert predictions are binary, either "positive" or "negative", with no probability attached. We do not assume that any of our experts is perfect, and we do not know which of our experts is the best. We also do not assume anything about the samples (they may not be independent and identically distributed).

The weighted majority algorithm initially assigns an equal weight of 1 to all experts. On each round, we ask all the experts for their predictions, and sum up the weights for each of the two possible predictions, "positive" or "negative". We output the prediction that has the higher weight. Then, when we see the actual result, we multiply by 1/2 the weight of every expert that got the prediction wrong.

Suppose the total number of experts is n, and the best expert makes no more than m mistakes over some given sampling sequence. Then we can prove that the weighted majority algorithm makes a total number of mistakes M that is bounded by 2.41*(m + log2(n)). In other words, the weighted majority algorithm makes no more mistakes than the best expert, plus a factor of log n, times a constant factor of 2.41.

Proof (taken from Blum 1996; a similar proof appears in Littlestone and Warmuth 1989): The combined weight of all the experts at the start of the problem is W = n. If the weighted majority algorithm makes a mistake, at least half the total weight of experts predicted incorrectly, so the total weight is reduced by a factor of at least 1/4. Thus, after the weighted majority algorithm makes M mistakes, its total weight W has been reduced by at least (3/4)^M. In other words:

W <= n*( (3/4)^M )

But if the best expert has made at most m mistakes, its weight is at least (1/2)^m. And since W includes the weight of all experts,

W >= (1/2)^m

Therefore:

(1/2)^m <= W <= n*( (3/4)^M )
(1/2)^m <= n*( (3/4)^M )
-m    <= log2(n) + M*log2(3/4)
-m    <= log2(n) + M*-log2(4/3)
M*log2(4/3) <= m + log2(n)
M    <= (1/log2(4/3))*(m + log2(n))
M    <= 2.41*(m + log2(n))

Blum then says that "we can achieve a better bound than that described above", by randomizing the algorithm to predict "positive" or "negative" with probability proportional to the weight assigned each prediction. Thus, if 3/4 of the expert weight went to "positive", we would predict "positive" with probability 75%, and "negative" with probability 25%.

An essentially similar proof, summing over the expected probability of a mistake on each round, will show that in this case:

M <= 1.39m + 2 ln(n)    (note: M is now an expectation)

Since the penalty applied to particular experts does not depend on the global prediction but only on the actual outcome, most of the proof proceeds as before. We have

W >= (1/2)^m

where again m is the best expert and W is the total weight of all experts.

We also have that W is the starting weight n times the product of (1 - 1/2 F_i), where F_i is the fraction of mistaken expert weight on round i:

W = n * Prod_i (1 - 1/2 F_i)

And since we predict with probability proportional to the expert weight, the expected number of mistakes is just the sum over F_i:

M = Sum_i F_i

So:

(1/2)^m             <= n * Prod_i (1 - 1/2 F_i)
-m*ln(2)             <= ln(n) + Sum_i ln(1 - 1/2 F_i)
Sum_i -ln(1 - 1/2 F_i) <= ln(n) + m*ln(2)
Sum_i (1/2 F_i)       <= ln(n) + m*ln(2)    (because x < -ln(1 - x) )
Sum_i F_i             <= 2*(ln(n) + m*ln(2))
M                   <= 1.39m + 2 ln(n)

Behold, we have done better by randomizing!

We should be especially suspicious that the randomized algorithm guesses with probability proportional to the expert weight assigned. This seems strongly reminiscent of betting with 70% probability on blue, when the environment is a random mix of 70% blue and 30% red cards. We know the best bet - and yet we only sometimes make this best bet, at other times betting on a condition we believe to be less probable.

Yet we thereby prove a smaller upper bound on the expected error. Is there an algebraic error in the second proof? Are we extracting useful work from a noise source? Is our knowledge harming us so much that we can do better through ignorance?

Maybe the unrandomized algorithm fails to take into account the Bayesian value of information, and hence fails to exhibit exploratory behavior? Maybe it doesn't test unusual cases to see if they might have surprising results?

But, examining the assumptions, we see that the feedback we receive is fixed, regardless of the prediction's output. Nor does the updating of the expert weights depend on the predictions we output. It doesn't matter whether we substitute a completely random string of 1s and 0s as our actual output. We get back exactly the same data from the environment, and the weighted majority algorithm updates the expert weights in exactly the same way. So that can't be the explanation.

Are the component experts doing worse than random, so that by randomizing our predictions, we can creep back up toward maximum entropy? But we didn't assume anything about how often the component experts were right, or use the fact in the proofs. Therefore the proofs would carry even if we specified that all experts were right at least 60% of the time. It's hard to see how randomizing our predictions could help, in that case - but the proofs still go through, unchanged.

So where's the gotcha?

Maybe I'm about to tell you that I looked, and I couldn't find the gotcha either. What would you believe, in that case? Would you think that the whole thesis on the futility of randomness was probably just wrong - a reasonable-sounding philosophical argument that simply wasn't correct in practice? Would you despair of being able to follow the math, and give up and shrug, unable to decide who might be right?

We don't always see the flaws right away, and that's something to always remember.

In any case, I'm about to start explaining the gotcha. If you want to go back, read the paper, and look yourself, you should stop reading now...

There does exist a rare class of occasions where we want a source of "true" randomness, such as a quantum measurement device. For example, you are playing rock-paper-scissors against an opponent who is smarter than you are, and who knows exactly how you will be making your choices. In this condition it is wise to choose randomly, because any method your opponent can predict will do worse-than-average. Assuming that the enemy knows your source code has strange consequences: the action sequence 1, 2, 1, 3 is good when derived from a quantum noise generator, but bad when derived from any deterministic algorithm, even though it is the same action sequence. Still it is not a totally unrealistic situation. In real life, it is, in fact, a bad idea to play rock-paper-scissors using an algorithm your opponent can predict. So are we, in this situation, deriving optimization from noise?

The random rock-paper-scissors player does not play cleverly, racking up lots of points. It does not win more than 1/3 of the time, or lose less than 1/3 of the time (on average). The randomized player does better because its alternatives perform poorly, not from being smart itself. Moreover, by assumption, the opponent is an intelligence whom we cannot outsmart and who always knows everything about any method we use. There is no move we can make that does not have a possible countermove. By assumption, our own intelligence is entirely useless. The best we can do is to avoid all regularity that the enemy can exploit. In other words, we do best by minimizing the effectiveness of intelligence within the system-as-a-whole, because only the enemy's intelligence is allowed to be effective. If we can't be clever ourselves, we might as well think of ourselves as the environment and the enemy as the sole intelligence within that environment. By becoming the maximum-entropy environment for rock-paper-scissors, we render all intelligence useless, and do (by assumption) the best we can do.

When the environment is adversarial, smarter than you are, and informed about your methods, then in a theoretical sense it may be wise to have a quantum noise source handy. (In a practical sense you're just screwed.) Again, this is not because we're extracting useful work from a noise source; it's because the most powerful intelligence in the system is adversarial, and we're minimizing the power that intelligence can exert in the system-as-a-whole. We don't do better-than-average, we merely minimize the extent to which the adversarial intelligence produces an outcome that is worse-than-average (from our perspective).

Similarly, cryptographers have a legitimate interest in strong randomness generators because cryptographers are trying to minimize the effectiveness of an intelligent adversary. Certainly entropy can act as an antidote to intelligence.

Now back to the weighted majority algorithm. Blum (1996) remarks:

"Intuitively, the advantage of the randomized approach is that it dilutes the worst case. Previously, the worst case was that slightly more than half of the total weight predicted incorrectly, causing the algorithm to make a mistake and yet only reduce the total weight by 1/4. Now there is roughly a 50/50 chance that the [randomized] algorithm will predict correctly in this case, and more generally, the probability that the algorithm makes a mistake is tied to the amount that the weight is reduced."

From the start, we did our analysis for an upper bound on the number of mistakes made. A global upper bound is no better than the worst individual case; thus, to set a global upper bound we must bound the worst individual case. In the worst case, our environment behaves like an adversarial superintelligence.

Indeed randomness can improve the worst-case scenario, if the worst-case environment is allowed to exploit "deterministic" moves but not "random" ones. It is like an environment that can decide to produce a red card whenever you bet on blue - unless you make the same bet using a "random" number generator instead of your creative intelligence.

Suppose we use a quantum measurement device to produce a string of random ones and zeroes; make two copies of this string; use one copy for the weighted majority algorithm's random number generator; and give another copy to an intelligent adversary who picks our samples. In other words, we let the weighted majority algorithm make exactly the same randomized predictions, produced by exactly the same generator, but we also let the "worst case" environment know what these randomized predictions will be. Then the improved upper bound of the randomized version is mathematically invalidated.

This shows that the improved upper bound proven for the randomized algorithm did not come from the randomized algorithm making systematically better predictions - doing superior cognitive work, being more intelligent - but because we arbitrarily declared that an intelligent adversary could read our mind in one case but not the other.

This is not just a minor quibble. It leads to the testable prediction that on real-world problems, where the environment is usually not an adversarial telepath, the unrandomized weighted-majority algorithm should do better than the randomized version. (Provided that some component experts outperform maximum entropy - are right more than 50% of the time on binary problems.)

Analyzing the worst-case scenario is standard practice in computational learning theory, but it makes the math do strange things. Moreover, the assumption of the worst case can be subtle; it is not always explicitly labeled. Consider the upper bound for the unrandomized weighted-majority algorithm. I did not use the term "worst case" - I merely wrote down some inequalities. In fact, Blum (1996), when initially introducing this bound, does not at first use the phrase "worst case". The proof's conclusion says only:

"Theorem 1. The number of mistakes M made by the Weighted Majority algorithm described above is never more than 2.41(m+lg n), where m is the number of mistakes made by the best expert so far."

Key word: Never.

The worst-case assumption for the unrandomized algorithm was implicit in calculating the right-hand-side of the inequality by assuming that, on each mistake, the total weight went down by a factor of 1/4, and the total weight never decreased after any successful prediction. This is the absolute worst possible performance the weighted-majority algorithm can give.

The assumption implicit in those innocent-looking equations is that the environment carefully maximizes the anguish of the predictor: The environment so cleverly chooses its data samples, that on each case where the weighted-majority algorithm is allowed to be successful, it shall receive not a single iota of useful feedback - not a single expert shall be wrong. And the weighted-majority algorithm will be made to err on sensory cases that produce the minimum possible useful feedback, maliciously fine-tuned to the exact current weighting of experts. We assume that the environment can predict every aspect of the predictor and exploit it - unless the predictor uses a "random" number generator which we arbitrarily declare to be unknowable to the adversary.

What strange assumptions are buried in that innocent little inequality,

<=

Moreover, the entire argument in favor of the randomized algorithm was theoretically suspect from the beginning, because it rested on non-transitive inequalities. If I prove an upper bound on the errors of algorithm X, and then prove a smaller upper bound on the errors of algorithm Y, it does not prove that in the real world Y will perform better than X. For example, I prove that X cannot do worse than 1000 errors, and that Y cannot do worse than 100 errors. Is Y a better algorithm? Not necessarily. Tomorrow I could find an improved proof which shows that X cannot do worse than 90 errors. And then in real life, both X and Y could make exactly 8 errors.

4 <= 1,000,000,000
9 <= 10

But that doesn't mean that 4 > 9.

So the next time you see an author remark, "We can further improve this algorithm using randomization..." you may not know exactly where to find the oddity. If you'd run across the above-referenced example (or any number of others in the machine-learning literature), you might not have known how to deconstruct the randomized weighted-majority algorithm. If such a thing should happen to you, then I hope that I have given you grounds to suspect that an oddity exists somewhere, even if you cannot find it - just as you would suspect a machine that proposed to extract work from a heat bath, even if you couldn't keep track of all the gears and pipes.

Nominull put it very compactly when he said that, barring an environment which changes based on the form of your algorithm apart from its output, "By adding randomness to your algorithm, you spread its behaviors out over a particular distribution, and there must be at least one point in that distribution whose expected value is at least as high as the average expected value of the distribution."

As I remarked in Perpetual Motion Beliefs:

I once knew a fellow who was convinced that his system of wheels and gears would produce reactionless thrust, and he had an Excel spreadsheet that would prove this - which of course he couldn't show us because he was still developing the system. In classical mechanics, violating Conservation of Momentum is provably impossible. So any Excel spreadsheet calculated according to the rules of classical mechanics must necessarily show that no reactionless thrust exists - unless your machine is complicated enough that you have made a mistake in the calculations.

If you ever find yourself mathematically proving that you can do better by randomizing, I suggest that you suspect your calculations, or suspect your interpretation of your assumptions, before you celebrate your extraction of work from a heat bath.

Original with comments: The Weighted Majority Algorithm

Bay Area Meetup: 11/17 8PM Menlo Park

Eliezer Yudkowsky, 13 November 2008 05:32AM

Robin Gane-McCalla plans to organize regular OB meetups in the Bay Area. The next one is 8PM, November 17th, 2008 (Monday night) in Menlo Park at TechShop. (Note that this is a room with seating, not a restaurant, so we hopefully get a chance to actually talk to each other - though I'll try to stay in the background myself.)

RSVP at Meetup.com.

Original with comments: Bay Area Meetup: 11/17 8PM Menlo Park

Selling Nonapples

Eliezer Yudkowsky, 13 November 2008 08:10PM

Previously in series: Worse Than Random

A tale of two architectures...

Once upon a time there was a man named Rodney Brooks, who could justly be called the King of Scruffy Robotics. (Sample paper titles: "Fast, Cheap, and Out of Control", "Intelligence Without Reason"). Brooks invented the "subsumption architecture" - robotics based on many small modules, communicating asynchronously and without a central world-model or central planning, acting by reflex, responding to interrupts. The archetypal example is the insect-inspired robot that lifts its leg higher when the leg encounters an obstacle - it doesn't model the obstacle, or plan how to go around it; it just lifts its leg higher.

In Brooks's paradigm - which he labeled nouvelle AI - intelligence emerges from "situatedness". One speaks not of an intelligent system, but rather the intelligence that emerges from the interaction of the system and the environment.

And Brooks wrote a programming language, the behavior language, to help roboticists build systems in his paradigmatic subsumption architecture - a language that includes facilities for asynchronous communication in networks of reflexive components, and programming finite state machines.

My understanding is that, while there are still people in the world who speak with reverence of Brooks's subsumption architecture, it's not used much in commercial systems on account of being nearly impossible to program.

Once you start stacking all these modules together, it becomes more and more difficult for the programmer to decide that, yes, an asynchronous local module which raises the robotic leg higher when it detects a block, and meanwhile sends asynchronous signal X to module Y, will indeed produce effective behavior as the outcome of the whole intertwined system whereby intelligence emerges from interaction with the environment...

Asynchronous parallel decentralized programs are harder to write. And it's not that they're a better, higher form of sorcery that only a few exceptional magi can use. It's more like the difference between the two business plans, "sell apples" and "sell nonapples".

One noteworthy critic of Brooks's paradigm in general, and subsumption architecture in particular, is a fellow by the name of Sebastian Thrun.

You may recall the 2005 DARPA Grand Challenge for the driverless cars. How many ways was this a fair challenge according to the tenets of Scruffydom? Let us count the ways:

- The challenge took place in the real world, where sensors are imperfect, random factors intervene, and macroscopic physics is only approximately lawful.

- The challenge took place outside the laboratory - not even on paved roads, but 212km of desert.

- The challenge took place in real time - continuous perception, continuous action, using only computing power that would fit on a car.

- The teams weren't told the specific race course until 2 hours before the race.

- You could write the code any way you pleased, so long as it worked.

- The challenge was competitive: The prize went to the fastest team that completed the race. Any team which, for ideological reasons, preferred elegance to speed - any team which refused to milk every bit of performance out of their systems - would surely lose to a less principled competitor.

And the winning team was Stanley, the Stanford robot, built by a team led by Sebastian Thrun.

How did he do it? If I recall correctly, Thrun said that the key was being able to integrate probabilistic information from many different sensors, using a common representation of uncertainty. This is likely code for "we used Bayesian methods", at least if "Bayesian methods" is taken to include algorithms like particle filtering.

And to heavily paraphrase and summarize some of Thrun's criticisms of Brooks's subsumption architecture:

Robotics becomes pointlessly difficult if, for some odd reason, you insist that there be no central model and no central planning.

Integrating data from multiple uncertain sensors is a lot easier if you have a common probabilistic representation. Likewise, there are many potential tasks in robotics - in situations as simple as navigating a hallway - when you can end up in two possible situations that look highly similar and have to be distinguished by reasoning about the history of the trajectory.

To be fair, it's not as if the subsumption architecture has never made money. Rodney Brooks is the founder of iRobot, and I understand that the Roomba uses the subsumption architecture. The Roomba has no doubt made more money than was won in the DARPA Grand Challenge... though the Roomba might not seem quite as impressive...

But that's not quite today's point.

Earlier in his career, Sebastian Thrun also wrote a programming language for roboticists. Thrun's language was named CES, which stands for C++ for Embedded Systems.

CES is a language extension for C++. Its types include probability distributions, which makes it easy for programmers to manipulate and combine multiple sources of uncertain information. And for differentiable variables - including probabilities - the language enables automatic optimization using techniques like gradient descent. Programmers can declare 'gaps' in the code to be filled in by training cases: "Write me this function."

As a result, Thrun was able to write a small, corridor-navigating mail-delivery robot using 137 lines of code, and this robot required less than 2 hours of training. As Thrun notes, "Comparable systems usually require at least two orders of magnitude more code and are considerably more difficult to implement." Similarly, a 5,000-line robot localization algorithm was reimplemented in 52 lines.

Why can't you get that kind of productivity with the subsumption architecture? Scruffies, ideologically speaking, are supposed to believe in learning - it's only those evil logical Neats who try to program everything into their AIs in advance. Then why does the subsumption architecture require so much sweat and tears from its programmers?

Suppose that you're trying to build a wagon out of wood, and unfortunately, the wagon has a problem, which is that it keeps catching on fire. Suddenly, one of the wagon-workers drops his wooden beam. His face lights up. "I have it!" he says. "We need to build this wagon from nonwood materials!"

You stare at him for a bit, trying to get over the shock of the new idea; finally you ask, "What kind of nonwood materials?"

The wagoneer hardly hears you. "Of course!" he shouts. "It's all so obvious in retrospect! Wood is simply the wrong material for building wagons! This is the dawn of a new era - the nonwood era - of wheels, axles, carts all made from nonwood! Not only that, instead of taking apples to market, we'll take nonapples! There's a huge market for nonapples - people buy far more nonapples than apples - we should have no trouble selling them! It will be the era of the nouvelle wagon!"

The set "apples" is much narrower than the set "not apples". Apples form a compact cluster in thingspace, but nonapples vary much more widely in price, and size, and use. When you say to build a wagon using "wood", you're giving much more concrete advice than when you say "not wood". There are different kinds of wood, of course - but even so, when you say "wood", you've narrowed down the range of possible building materials a whole lot more than when you say "not wood".

In the same fashion, "asynchronous" - literally "not synchronous" - is a much larger design space than "synchronous". If one considers the space of all communicating processes, then synchrony is a very strong constraint on those processes. If you toss out synchrony, then you have to pick some other method for preventing communicating processes from stepping on each other - synchrony is one way of doing that, a specific answer to the question.

Likewise "parallel processing" is a much huger design space than "serial processing", because serial processing is just a special case of parallel processing where the number of processors happens to be equal to 1. "Parallel processing" reopens all sorts of design choices that are premade in serial processing. When you say "parallel", it's like stepping out of a small cottage, into a vast and echoing country. You have to stand someplace specific, in that country - you can't stand in the whole place, in the noncottage.

So when you stand up and shout: "Aha! I've got it! We've got to solve this problem using asynchronous processes!", it's like shouting, "Aha! I've got it! We need to build this wagon out of nonwood! Let's go down to the market and buy a ton of nonwood from the nonwood shop!" You've got to choose some specific alternative to synchrony.

Now it may well be that there are other building materials in the universe than wood. It may well be that wood is not the best building material. But you still have to come up with some specific thing to use in its place, like iron. "Nonwood" is not a building material, "sell nonapples" is not a business strategy, and "asynchronous" is not a programming architecture.

And this is strongly reminiscent of - arguably a special case of - the dilemma of inductive bias. There's a tradeoff between the strength of the assumptions you make, and how fast you learn. If you make stronger assumptions, you can learn faster when the environment matches those assumptions well, but you'll learn correspondingly more slowly if the environment matches those assumptions poorly. If you make an assumption that lets you learn faster in one environment, it must always perform more poorly in some other environment. Such laws are known as the "no-free-lunch" theorems, and the reason they don't prohibit intelligence entirely is that the real universe is a low-entropy special case.

Programmers have a phrase called the "Turing Tarpit"; it describes a situation where everything is possible, but nothing is easy. A Universal Turing Machine can simulate any possible computer, but only at an immense expense in time and memory. If you program in a high-level language like Python, then - while most programming tasks become much simpler - you may occasionally find yourself banging up against the walls imposed by the programming language; sometimes Python won't let you do certain things. If you program directly in machine language, raw 1s and 0s, there are no constraints; you can do anything that can possibly be done by the computer chip; and it will probably take you around a thousand times as much time to get anything done. You have to do, all by yourself, everything that a compiler would normally do on your behalf.

Usually, when you adopt a program architecture, that choice takes work off your hands. If I use a standard container library - lists and arrays and hashtables - then I don't need to decide how to implement a hashtable, because that choice has already been made for me.

Adopting the subsumption paradigm means losing order, instead of gaining it. The subsumption architecture is not-synchronous, not-serial, and not-centralized. It's also not-knowledge-modelling and not-planning.

This absence of solution implies an immense design space, and it requires a correspondingly immense amount of work by the programmers to reimpose order. Under the subsumption architecture, it's the programmer who decides to add an asynchronous local module which detects whether a robotic leg is blocked, and raises it higher. It's the programmer who has to make sure that this behavior plus other module behaviors all add up to an (ideologically correct) emergent intelligence. The lost structure is not replaced. You just get tossed into the Turing Tarpit, the space of all other possible programs.

On the other hand, CES creates order; it adds the structure of probability distributions and gradient optimization. This narrowing of the design space takes so much work off your hands that you can write a learning robot in 137 lines (at least if you happen to be Sebastian Thrun).

The moral:

Quite a few AI architectures aren't.

If you want to generalize, quite a lot of policies aren't.

They aren't choices. They're just protests.

Added: Robin Hanson says, "Economists have to face this in spades. So many people say standard econ has failed and the solution is to do the opposite - non-equilibrium instead of equilibrium, non-selfish instead of selfish, non-individual instead of individual, etc." It seems that selling nonapples is a full-blown Standard Iconoclast Failure Mode.

Referenced by: Reversed Stupidity Is Not Intelligence • The Nature of Logic

Original with comments: Selling Nonapples

The Nature of Logic

Eliezer Yudkowsky, 15 November 2008 06:20AM

Previously in series: Selling Nonapples
Followup to: The Parable of Hemlock

Decades ago, there was a tremendous amount of effort invested in certain kinds of systems that centered around first-order logic - systems where you entered "Socrates is a human" and "all humans are mortal" into a database, and it would spit out "Socrates is mortal".

The fact that these systems failed to produce "general intelligence" is sometimes taken as evidence of the inadequacy, not only of logic, but of Reason Itself. You meet people who (presumably springboarding off the Spock meme) think that what we really need are emotional AIs, since logic hasn't worked.

What's really going on here is so completely different from the popular narrative that I'm not even sure where to start. So I'm going to try to explain what I see when I look at a "logical AI". It's not a grand manifestation of the ultimate power of Reason (which then fails).

Let's start with the logical database containing the statements:

|- Human(Socrates)
|- \x.(Human(x) -> Mortal(x))

which then produces the output

|- Mortal(Socrates)

where |- is how we say that our logic asserts something, and \x. means "all x" or "in what follows, we can substitute anything we want for x".

Thus the above is how we'd write the classic syllogism, "Socrates is human, all humans are mortal, therefore Socrates is mortal".

Now a few months back, I went through the sequence on words, which included The Parable of Hemlock, which runs thusly:

Socrates raised the glass of hemlock to his lips...
"Do you suppose," asked one of the onlookers, "that even hemlock will not be enough to kill so wise and good a man?"
"No," replied another bystander, a student of philosophy; "all men are mortal, and Socrates is a man; and if a mortal drink hemlock, surely he dies."
"Well," said the onlooker, "what if it happens that Socrates isn't mortal?"
"Nonsense," replied the student, a little sharply; "all men are mortal by definition; it is part of what we mean by the word 'man'. All men are mortal, Socrates is a man, therefore Socrates is mortal. It is not merely a guess, but a logical certainty."
"I suppose that's right..." said the onlooker. "Oh, look, Socrates already drank the hemlock while we were talking."
"Yes, he should be keeling over any minute now," said the student.
And they waited, and they waited, and they waited...
"Socrates appears not to be mortal," said the onlooker.
"Then Socrates must not be a man," replied the student. "All men are mortal, Socrates is not mortal, therefore Socrates is not a man. And that is not merely a guess, but a logical certainty."

If you're going to take "all men are mortal" as something true by definition, then you can never conclude that Socrates is "human" until after you've observed him to be mortal. Since logical truths are true in all possible worlds, they never tell you which possible world you live in - no logical truth can predict the result of an empirical event which could possibly go either way.

Could a skeptic say that this logical database is not doing any cognitive work, since it can only tell us what we already know? Or that, since the database is only using logic, it will be unable to do anything empirical, ever?

Even I think that's too severe. The "logical reasoner" is doing a quantum of cognitive labor - it's just a small quantum.

Consider the following sequence of events:

- We notice that there's an empirical cluster of similar things that have ten fingers, wear clothes, use tools, and speak a language, bleed red blood when cut. We think this empirical cluster is significant.

- We decide to add the sentence |- human(x) to our database, whenever we observe that X has ten fingers, wears clothes, uses tools, and speaks a language. These particular characteristics are easier to observe (we think) than e.g. cutting X to see if red blood comes out.

- In our experience so far, all entities with ten fingers etc. have died whenever administered hemlock. We think this is a significant fact.

- Therefore, we add the sentence |- \x.(human(x)->mortal(x)) to our database.

- Now we see Socrates, and he has ten fingers, wears clothes, uses tools, and fluently speaks Ancient Greek. We decide that Socrates is human. We add the sentence |- human(Socrates) to the database.

- After a bit of grinding, the database spits out |- mortal(Socrates) on its screen.

- We interpret this to mean that Socrates will die if he ingests hemlock.

- It so happens that, in Socrates's hand, is a drink that we recognize as hemlock.

- We act accordingly by crying "Watch out!" and knocking the cup out of his hand.

This process, taken as a whole, is hardly absolutely certain, as in the Spock stereotype of rationalists who cannot conceive that they are wrong. The process did briefly involve a computer program which mimicked a system, first-order classical logic, which also happens to be used by some mathematicians in verifying their proofs. That doesn't lend the entire process the character of mathematical proof. And if the process fails, somewhere along the line, that's no call to go casting aspersions on Reason itself.

In this admittedly contrived example, only an infinitesimal fraction of the cognitive work is being performed by the computer program. It's such a small fraction that anything you could say about "logical AI", wouldn't say much about the process as a whole.

So what's an example of harder cognitive labor?

How about deciding that "human" is an important category to put things in? It's not like we're born seeing little "human" tags hovering over objects, with high priority attached. You have to discriminate stable things in the environment, like Socrates, from your raw sensory information. You have to notice that various stable things are all similar to one another, wearing clothes and talking. Then you have to draw a category boundary around the cluster, and harvest characteristics like vulnerability to hemlock.

A human operator, not the computer, decides whether or not to classify Socrates as a "human", based on his shape and clothes. The human operator types |- human(Socrates) into the database (itself an error-prone sort of process). Then the database spits out |- mortal(Socrates) - in the scenario, this is the only fact we've ever told it about humans, so we don't ask why it makes this deduction instead of another one. A human looks at the screen, interprets |- mortal(Socrates) to refer to a particular thing in the environment and to imply that thing's vulnerability to hemlock. Then the human decides, based on their values, that they'd rather not see Socrates die; works out a plan to stop Socrates from dying; and executes motor actions to dash the chalice from Socrates's fingers.

Are the off-computer steps "logical"? Are they "illogical"? Are they unreasonable? Are they unlawful? Are they unmathy?

Let me interrupt this tale, to describe a case where you very much do want a computer program that processes logical statements:

Suppose you've got to build a computer chip with a hundred million transistors, and you don't want to recall your product when a bug is discovered in multiplication. You might find it very wise to describe the transistors in first-order logic, and try to prove statements about how the chip performs multiplication.

But then why is logic suited to this particular purpose?

Logic relates abstract statements to specific models. Let's say that I have an abstract statement like "all green objects are round" or "all round objects are soft". Operating syntactically, working with just the sentences, I can derive "all green objects are soft".

Now if you show me a particular collection of shapes, and if it so happens to be true that every green object in that collection is round, and it also happens to be true that every round object is soft, then it will likewise be true that all green objects are soft.

We are not admitting of the possibility that a green-detector on a borderline green object will fire "green" on one occasion and "not-green" on the next. The form of logic in which every proof step preserves validity, relates crisp models to crisp statements. So if you want to draw a direct correspondence between elements of a logical model, and high-level objects in the real world, you had better be dealing with objects that have crisp identities, and categories that have crisp boundaries.

Transistors in a computer chip generally do have crisp identities. So it may indeed be suitable to make a mapping between elements in a logical model, and real transistors in the real world.

So let's say you can perform the mapping and get away with it - then what?

The power of logic is that it relates models and statements. So you've got to make that mental distinction between models on the one hand, and statements on the other. You've got to draw a sharp line between the elements of a model that are green or round, and statements like \x.(green(x)->round(x)). The statement itself isn't green or round, but it can be true or false about a collection of objects that are green or round.

And here is the power of logic: For each syntactic step we do on our statements, we preserve the match to any model. In any model where our old collection of statements was true, the new statement will also be true. We don't have to check all possible conforming models to see if the new statement is true in all of them. We can trust certain syntactic steps in general - not to produce truth, but to preserve truth.

Then you do a million syntactic steps in a row, and because each step preserves truth, you can trust that the whole sequence will preserve truth.

We start with a chip. We do some physics and decide that whenever transistor X is 0 at time T, transistor Y will be 1 at T+1, or some such - we credit that real events in the chip will correspond quite directly to a model of this statement. We do a whole lot of syntactic manipulation on the abstract laws. We prove a statement that describes binary multiplication. And then we jump back to the model, and then back to the chip, and say, "Whatever the exact actual events on this chip, if they have the physics we described, then multiplication will work the way we want."

It would be considerably harder (i.e. impossible) to work directly with logical models of every possible computation the chip could carry out. To verify multiplication on two 64-bit inputs, you'd need to check 340 trillion trillion trillion models.

But this trick of doing a million derivations one after the other, and preserving truth throughout, won't work if the premises are only true 999 out of 1000 times. You could get away with ten steps in the derivation and not lose too much, but a million would be out of the question.

So the truth-preserving syntactic manipulations we call "logic" can be very useful indeed, when we draw a correspondence to a digital computer chip where the transistor error rate is very low.

But if you're trying to draw a direct correspondence between the primitive elements of a logical model and, say, entire biological humans, that may not work out as well.

First-order logic has a number of wonderful properties, like detachment. We don't care how you proved something - once you arrive at a statement, we can forget how we got it. The syntactic rules are local, and use statements as fodder without worrying about their provenance. So once we prove a theorem, there's no need to keep track of how, in particular, it was proved.

But what if one of your premises turns out to be wrong, and you have to retract something you already concluded? Wouldn't you want to keep track of which premises you'd used?

If the burglar alarm goes off, that means that a burglar is burgling your house. But if there was an earthquake that day, it's probably the earthquake that set off the alarm instead. But if you learned that there was a burglar from the police, rather than the alarm, then you don't want to retract the "burglar" conclusion on finding that there was an earthquake...

It says a lot about the problematic course of early AI, that people first tried to handle this problem with nonmonotonic logics. They would try to handle statements like "A burglar alarm indicates there's a burglar - unless there's an earthquake" using a slightly modified logical database that would draw conclusions and then retract them.

And this gave rise to huge problems for many years, because they were trying to do, in the style of logic, something that was not at all like the actual nature of logic as math. Trying to retract a particular conclusion goes completely against the nature of first-order logic as a mathematical structure.

If you were given to jumping to conclusions, you might say "Well, math can't handle that kind of problem because there are no absolute laws as to what you conclude when you hear the burglar alarm - you've got to use your common-sense judgment, not math."

But it's not an unlawful or even unmathy question.

It turns out that for at least the kind of case I've described above - where you've got effects that have more than one possible cause - we can excellently handle a wide range of scenarios using a crystallization of probability theory known as "Bayesian networks". And lo, we can prove all sorts of wonderful theorems that I'm not going to go into. (See Pearl's "Probabilistic Reasoning in Intelligent Systems".)

And the real solution turned out to be much more elegant than all the messy ad-hoc attempts at "nonmonotonic logic". On non-loopy networks, you can do all sorts of wonderful things like propagate updates in parallel using asynchronous messages, where each node only tracks the messages coming from its immediate neighbors in the graph, etcetera. And this parallel, asynchronous, decentralized algorithm is provably correct as probability theory, etcetera.

So... are Bayesian networks illogical?

Certainly not in the colloquial sense of the word.

You could write a logic that implemented a Bayesian network. You could represent the probabilities and graphs in a logical database. The elements of your model would no longer correspond to things like Socrates, but rather correspond to conditional probabilities or graph edges... But why bother? Non-loopy Bayesian networks propagate their inferences in nicely local ways. There's no need to stick a bunch of statements in a centralized logical database and then waste computing power to pluck out global inferences.

What am I trying to convey here? I'm trying to convey that thinking mathematically about uncertain reasoning is a completely different concept from AI programs that assume direct correspondences between the elements of a logical model and the high-level regularities of reality.

"The failure of logical AI" is not "the failure of mathematical thinking about AI" and certainly not "the limits of lawful reasoning". The "failure of logical AI" is more like, "That thing with the database containing statements about Socrates and hemlock - not only were you using the wrong math, but you weren't even looking at the interesting parts of the problem."

Now I did concede that the logical reasoner talking about Socrates and hemlock, was performing a quantum of cognitive labor. We can now describe that quantum:

"After you've arrived at such-and-such hypothesis about what goes on behind the scenes of your sensory information, and distinguished the pretty-crisp identity of 'Socrates' and categorized it into the pretty-crisp cluster of 'human', then, if the other things you've observed to usually hold true of 'humans' are accurate in this case, 'Socrates' will have the pretty-crisp property of 'mortality'."

This quantum of labor tells you a single implication of what you already believe... but actually it's an even smaller quantum than this. The step carried out by the logical database corresponds to verifying this step of inference, not deciding to carry it out. Logic makes no mention of which inferences we should perform first - the syntactic derivations are timeless and unprioritized. It is nowhere represented in the nature of logic as math, that if the 'Socrates' thingy is drinking hemlock, right now is a good time to ask if he's mortal.

And indeed, modern AI programs still aren't very good at guiding inference. If you want to prove a computer chip correct, you've got to have a human alongside to suggest the lemmas to be proved next. The nature of logic is better suited to verification than construction - it preserves truth through a million syntactic manipulations, but it doesn't prioritize those manipulations in any particular order. So saying "Use logic!" isn't going to solve the problem of searching for proofs.

This doesn't mean that "What inference should I perform next?" is an unlawful question to which no math applies. Just that the math of logic that relates models and statements, relates timeless models to timeless statements in a world of unprioritized syntactic manipulations. You might be able to use logic to reason about time or about expected utility, the same way you could use it to represent a Bayesian network. But that wouldn't introduce time, or wanting, or nonmonotonicity, into the nature of logic as a mathematical structure.

Now, math itself tends to be timeless and detachable and proceed from premise to conclusion, at least when it happens to be right. So logic is well-suited to verifying mathematical thoughts - though producing those thoughts in the first place, choosing lemmas and deciding which theorems are important, is a whole different problem.

Logic might be well-suited to verifying your derivation of the Bayesian network rules from the axioms of probability theory. But this doesn't mean that, as a programmer, you should try implementing a Bayesian network on top of a logical database. Nor, for that matter, that you should rely on a first-order theorem prover to invent the idea of a "Bayesian network" from scratch.

Thinking mathematically about uncertain reasoning, doesn't mean that you try to turn everything into a logical model. It means that you comprehend the nature of logic itself within your mathematical vision of cognition, so that you can see which environments and problems are nicely matched to the structure of logic.

Referenced by: Logical or Connectionist AI? • Engelbart: Insufficiently Recursive • Artificial Mysterious Intelligence • What I Think, If Not Why

Original with comments: The Nature of Logic

Boston-area Meetup: 11/18/08 9pm MIT/Cambridge

Eliezer Yudkowsky, 16 November 2008 03:19AM

There will be an OB meetup this Tuesday in Cambridge MA, hosted by Michael Vassar, Owain Evans (grad student at MIT), and Dario Amodei (grad student at Princeton). The event will take place on the MIT campus, in a spacious seminar room in MIT's Stata Center. Refreshments will be provided. Details and directions below the fold.

Please let us know in the comments if you plan to attend.

(Posted on behalf of Owain Evans.)

Time/date: 9pm, Tuesday 18 November.

Place: Room d461 in MIT's Stata Center.

Address: The Stata Center: 32 Vassar Street, Cambridge.

Directions: The nearest T stop is Kendall/MIT on the Red Line.

Enter Stata via the entrance facing Main Street (with a big metal "MIT" sign outside it) from 8.45pm and one of the hosts will guide you to d461. Alternatively, here are directions to d461 once you reach Stata.

Email: owain (at) mit edu

Phone: If you can't find the room or if you arrive late and are unable to enter the building, call 610-608-3345.

Original with comments: Boston-area Meetup: 11/18/08 9pm MIT/Cambridge

Logical or Connectionist AI?

Eliezer Yudkowsky, 17 November 2008 08:03AM

Previously in series: The Nature of Logic

People who don't work in AI, who hear that I work in AI, often ask me: "Do you build neural networks or expert systems?" This is said in much the same tones as "Are you a good witch or a bad witch?"

Now that's what I call successful marketing.

Yesterday I covered what I see when I look at "logic" as an AI technique. I see something with a particular shape, a particular power, and a well-defined domain of useful application where cognition is concerned. Logic is good for leaping from crisp real-world events to compact general laws, and then verifying that a given manipulation of the laws preserves truth. It isn't even remotely close to the whole, or the center, of a mathematical outlook on cognition.

But for a long time, years and years, there was a tremendous focus in Artificial Intelligence on what I call "suggestively named LISP tokens" - a misuse of logic to try to handle cases like "Socrates is human, all humans are mortal, therefore Socrates is mortal". For many researchers, this one small element of math was indeed their universe.

And then along came the amazing revolution, the new AI, namely connectionism.

In the beginning (1957) was Rosenblatt's Perceptron. It was, I believe, billed as being inspired by the brain's biological neurons. The Perceptron had exactly two layers, a set of input units, and a single binary output unit. You multiplied the inputs by the weightings on those units, added up the results, and took the sign: that was the classification. To learn from the training data, you checked the current classification on an input, and if it was wrong, you dropped a delta on all the weights to nudge the classification in the right direction.

The Perceptron could only learn to deal with training data that was linearly separable - points in a hyperspace that could be cleanly separated by a hyperplane.

And that was all that this amazing algorithm, "inspired by the brain", could do.

In 1969, Marvin Minsky and Seymour Papert pointed out that Perceptrons couldn't learn the XOR function because it wasn't linearly separable. This killed off research in neural networks for the next ten years.

Now, you might think to yourself: "Hey, what if you had more than two layers in a neural network? Maybe then it could learn the XOR function?"

Well, but if you know a bit of linear algebra, you'll realize that if the units in your neural network have outputs that are linear functions of input, then any number of hidden layers is going to behave the same way as a single layer - you'll only be able to learn functions that are linearly separable.

Okay, so what if you had hidden layers and the outputs weren't linear functions of the input?

But you see - no one had any idea how to train a neural network like that. Cuz, like, then this weight would affect that output and that other output too, nonlinearly, so how were you supposed to figure out how to nudge the weights in the right direction?

Just make random changes to the network and see if it did any better? You may be underestimating how much computing power it takes to do things the truly stupid way. It wasn't a popular line of research.

Then along came this brilliant idea, called "backpropagation":

You handed the network a training input. The network classified it incorrectly. So you took the partial derivative of the output error (in layer N) with respect to each of the individual nodes in the preceding layer (N - 1). Then you could calculate the partial derivative of the output error with respect to any single weight or bias in the layer N - 1. And you could also go ahead and calculate the partial derivative of the output error with respect to each node in the layer N - 2. So you did layer N - 2, and then N - 3, and so on back to the input layer. (Though backprop nets usually had a grand total of 3 layers.) Then you just nudged the whole network a delta - that is, nudged each weight or bias by delta times its partial derivative with respect to the output error.

It says a lot about the nonobvious difficulty of doing math that it took years to come up with this algorithm.

I find it difficult to put into words just how obvious this is in retrospect. You're just taking a system whose behavior is a differentiable function of continuous paramaters, and sliding the whole thing down the slope of the error function. There are much more clever ways to train neural nets, taking into account more than the first derivative, e.g. conjugate gradient optimization, and these take some effort to understand even if you know calculus. But backpropagation is ridiculously simple. Take the network, take the partial derivative of the error function with respect to each weight in the network, slide it down the slope.

If I didn't know the history of connectionism, and I didn't know scientific history in general - if I had needed to guess without benefit of hindsight how long it ought to take to go from Perceptrons to backpropagation - then I would probably say something like: "Maybe a couple of hours? Lower bound, five minutes - upper bound, three days."

"Seventeen years" would have floored me.

And I know that backpropagation may be slightly less obvious if you don't have the idea of "gradient descent" as a standard optimization technique bopping around in your head. I know that these were smart people, and I'm doing the equivalent of complaining that Newton only invented first-year undergraduate stuff, etc.

So I'm just mentioning this little historical note about the timescale of mathematical progress, to emphasize that all the people who say "AI is 30 years away so we don't need to worry about Friendliness theory yet" have moldy jello in their skulls.

(Which I suspect is part of a general syndrome where people's picture of Science comes from reading press releases that announce important discoveries, so that they're like, "Really? You do science? What kind of important discoveries do you announce?" Apparently, in their world, when AI finally is "visibly imminent", someone just needs to issue a press release to announce the completion of Friendly AI theory.)

Backpropagation is not just clever; much more importantly, it turns out to work well in real life on a wide class of tractable problems. Not all "neural network" algorithms use backprop, but if you said, "networks of connected units with continuous parameters and differentiable behavior which learn by traveling up a performance gradient", you would cover a pretty large swathe.

But the real cleverness is in how neural networks were marketed.

They left out the math.

To me, at least, it seems that a backprop neural network involves substantially deeper mathematical ideas than "Socrates is human, all humans are mortal, Socrates is mortal". Newton versus Aristotle. I would even say that a neural network is more analyzable - since it does more real cognitive labor on board a computer chip where I can actually look at it, rather than relying on inscrutable human operators who type "|- Human(Socrates)" into the keyboard under God knows what circumstances.

But neural networks were not marketed as cleverer math. Instead they were marketed as a revolt against Spock.

No, more than that - the neural network was the new champion of the Other Side of the Force - the antihero of a Manichaean conflict between Law and Chaos. And all good researchers and true were called to fight on the side of Chaos, to overthrow the corrupt Authority and its Order. To champion Freedom and Individuality against Control and Uniformity. To Decentralize instead of Centralize, substitute Empirical Testing for mere Proof, and replace Rigidity with Flexibility.

I suppose a grand conflict between Law and Chaos, beats trying to explain calculus in a press release.

But the thing is, a neural network isn't an avatar of Chaos any more than an expert system is an avatar of Law.

It's just... you know... a system with continuous parameters and differentiable behavior traveling up a performance gradient.

And logic is a great way of verifying truth preservation by syntactic manipulation of compact generalizations that are true in crisp models. That's it. That's all. This kind of logical AI is not the avatar of Math, Reason, or Law.

Both algorithms do what they do, and are what they are; nothing more.

But the successful marketing campaign said,

"The failure of logical systems to produce real AI has shown that intelligence isn't logical. Top-down design doesn't work; we need bottom-up techniques, like neural networks."

And this is what I call the Lemon Glazing Fallacy, which generates an argument for a fully arbitrary New Idea in AI using the following template:

- Major premise: All previous AI efforts failed to yield true intelligence.

- Minor premise: All previous AIs were built without delicious lemon glazing.

- Conclusion: If we build AIs with delicious lemon glazing, they will work.

This only has the appearance of plausibility if you present a Grand Dichotomy. It doesn't do to say "AI Technique #283 has failed for years to produce general intelligence - that's why you need to adopt my new AI Technique #420." Someone might ask, "Well, that's very nice, but what about AI technique #59,832?"

No, you've got to make 420 and ¬420 into the whole universe - allow only these two possibilities - put them on opposite sides of the Force - so that ten thousand failed attempts to build AI are actually arguing for your own success. All those failures are weighing down the other side of the scales, pushing up your own side... right? (In Star Wars, the Force has at least one Side that does seem pretty Dark. But who says the Jedi are the Light Side just because they're not Sith?)

Ten thousand failures don't tell you what will work. They don't even say what should not be part of a successful AI system. Reversed stupidity is not intelligence.

If you remove the power cord from your computer, it will stop working. You can't thereby conclude that everything about the current system is wrong, and an optimal computer should not have an Intel processor or Nvidia video card or case fans or run on electricity. Even though your current system has these properties, and it doesn't work.

As it so happens, I do believe that the type of systems usually termed GOFAI will not yield general intelligence, even if you run them on a computer the size of the moon. But this opinion follows from my own view of intelligence. It does not follow, even as suggestive evidence, from the historical fact that a thousand systems built using Prolog did not yield general intelligence. So far as the logical sequitur goes, one might as well say that Silicon-Based AI has shown itself deficient, and we must try to build transistors out of carbon atoms instead.

Not to mention that neural networks have also been "failing" (i.e., not yet succeeding) to produce real AI for 30 years now. I don't think this particular raw fact licenses any conclusions in particular. But at least don't tell me it's still the new revolutionary idea in AI.

This is the original example I used when I talked about the "Outside the Box" box - people think of "amazing new AI idea" and return their first cache hit, which is "neural networks" due to a successful marketing campaign thirty goddamned years ago. I mean, not every old idea is bad - but to still be marketing it as the new defiant revolution? Give me a break.

And pity the poor souls who try to think outside the "outside the box" box - outside the ordinary bounds of logical AI vs. connectionist AI - and, after mighty strains, propose a hybrid system that includes both logical and neural-net components.

It goes to show that compromise is not always the path to optimality - though it may sound Deeply Wise to say that the universe must balance between Law and Chaos.

Where do Bayesian networks fit into this dichotomy? They're parallel, asynchronous, decentralized, distributed, probabilistic. And they can be proven correct from the axioms of probability theory. You can preprogram them, or learn them from a corpus of unsupervised data - using, in some cases, formally correct Bayesian updating. They can reason based on incomplete evidence. Loopy Bayes nets, rather than computing the correct probability estimate, might compute an approximation using Monte Carlo - but the approximation provably converges - but we don't run long enough to converge...

Where does that fit on the axis that runs from logical AI to neural networks? And the answer is that it doesn't. It doesn't fit.

It's not that Bayesian networks "combine the advantages of logic and neural nets". They're simply a different point in the space of algorithms, with different properties.

At the inaugural seminar of Redwood Neuroscience, I once saw a presentation describing a robot that started out walking on legs, and learned to run... in real time, over the course of around a minute. The robot was stabilized in the Z axis, but it was still pretty darned impressive. (When first exhibited, someone apparently stood up and said "You sped up that video, didn't you?" because they couldn't believe it.)

This robot ran on a "neural network" built by detailed study of biology. The network had twenty neurons or so. Each neuron had a separate name and its own equation. And believe me, the robot's builders knew how that network worked.

Where does that fit into the grand dichotomy? Is it top-down? Is it bottom-up? Calling it "parallel" or "distributed" seems like kind of a silly waste when you've only got 20 neurons - who's going to bother multithreading that?

This is what a real biologically inspired system looks like. And let me say again, that video of the running robot would have been damned impressive even if it hadn't been done using only twenty neurons. But that biological network didn't much resemble - at all, really - the artificial neural nets that are built using abstract understanding of gradient optimization, like backprop.

That network of 20 neurons, each with its own equation, built and understood from careful study of biology - where does it fit into the Manichaean conflict? It doesn't. It's just a different point in AIspace.

At a conference ysterday, I spoke to someone who thought that Google's translation algorithm was a triumph of Chaotic-aligned AI, because none of the people on the translation team spoke Arabic and yet they built an Arabic translator using a massive corpus of data. And I said that, while I wasn't familiar in detail with Google's translator, the little I knew about it led me to believe that they were using well-understood algorithms - Bayesian ones, in fact - and that if no one on the translation team knew any Arabic, this was no more significant than Deep Blue's programmers playing poor chess.

Since Peter Norvig also happened to be at the conference, I asked him about it, and Norvig said that they started out doing an actual Bayesian calculation, but then took a couple of steps away. I remarked, "Well, you probably weren't doing the real Bayesian calculation anyway - assuming conditional independence where it doesn't exist, and stuff", and Norvig said, "Yes, so we've already established what kind of algorithm it is, and now we're just haggling over the price."

Where does that fit into the axis of logical AI and neural nets? It doesn't even talk to that axis. It's just a different point in the design space.

The grand dichotomy is a lie - which is to say, a highly successful marketing campaign which managed to position two particular fragments of optimization as the Dark Side and Light Side of the Force.

Referenced by: Failure By Analogy • Artificial Mysterious Intelligence

Original with comments: Logical or Connectionist AI?

Whither OB?

Eliezer Yudkowsky, 17 November 2008 07:38PM

Robin plans to cut back posting shortly, after he and I have our long-awaited Disagreement about AI self-improvement. As for myself - I'm not finished, but I'm way over schedule and need to move on soon. I'm not going to stop posting entirely (I doubt I could if I tried) but I'm not going to be posting daily.

There are three directions that Overcoming Bias could go from here:

First, we could find enough good authors to keep going at a post per day. Say, seven people who can and will write one post per week. We can't compromise on quality, though.

Second, we could try to shift to a more community-based format. Our most popular post ever, still getting hits to this day, was not written by Robin or myself or any of the recurring editors. It's "My Favorite Liar" by Kai Chang, about the professor who inserted one false statement into each lecture. If one-tenth of our readers contributed a single story as good as this... but neither Robin nor myself have time to vet them all. So one approach would be to have a community forum where anyone could post, readers voted the posts up and down, and a front page to which the editors promoted posts deemed worthy. I understand that Scoop has software like this, but I would like to know if our readers can recommend better community software (see below).

Third, we could close OB to new submissions and keep the archives online eternally, saying, "It had a good run." As Nick put it, we shouldn't keep going if it means a slow degeneration.

My own perspective: Overcoming Bias presently gets over a quarter-million monthly pageviews. We've built something that seems like it should be important. It feels premature, but I would like to try to launch an online rationalist community.

At this point, I'm advocating a hybrid approach: Keep OB open with fewer posts of the same gravitas, hang out a sign for new authors, and also try starting up a community-based site with user submissions and more frequent shorter posts. I've got plenty of light stuff to post, links and the like.

But: What software should we use to support a rationalist community?

The Oxford Future of Humanity Institute and the Singularity Institute have volunteered to provide funding if necessary, so we aren't limited to free software.

And obviously we're not looking for software that lets our users throw sheep at one another. The Internet already offers enough ways to waste time, thank you. More like - how people can find each other geographically and meet up; something Reddit-like for upvoting and downvoting posts, links, and comments; better comment threading; ways to see only new comments on posts you've flagged - that sort of thing. You know, actually useful stuff. A lot of Web 2.0 seems to be designed for people with lots of time to waste, but I don't think we can assume that fact about our readership.

Even if you don't know the name of the software, if there's a community site you visit that does an exceptional job - letting users upvote and downvote to keep the quality high, threading discussions while still giving busy people a fast way to see what new comments have been posted, making it easy for both newcomers and oldtimers - go ahead and say what we should be looking at.

Referenced by: OB Status Update

Original with comments: Whither OB?

Failure By Analogy

Eliezer Yudkowsky, 18 November 2008 02:54AM

Previously in series: Logical or Connectionist AI?
Followup to: Surface Analogies and Deep Causes

"One of [the Middle Ages'] characteristics was that 'reasoning by analogy' was rampant; another characteristic was almost total intellectual stagnation, and we now see why the two go together. A reason for mentioning this is to point out that, by developing a keen ear for unwarranted analogies, one can detect a lot of medieval thinking today."
-- Edsger W. Dijkstra

<geoff> neural nets are over-rated
<starglider> Their potential is overrated.
<geoff> their potential is us
-- #sl4

Wasn't it in some sense reasonable to have high hopes of neural networks? After all, they're just like the human brain, which is also massively parallel, distributed, asynchronous, and -

Hold on. Why not analogize to an earthworm's brain, instead of a human's?

A backprop network with sigmoid units... actually doesn't much resemble biology at all. Around as much as a voodoo doll resembles its victim. The surface shape may look vaguely similar in extremely superficial aspects at a first glance. But the interiors and behaviors, and basically the whole thing apart from the surface, are nothing at all alike. All that biological neurons have in common with gradient-optimization ANNs is... the spiderwebby look.

And who says that the spiderwebby look is the important fact about biology? Maybe the performance of biological brains has nothing to do with being made out of neurons, and everything to do with the cumulative selection pressure put into the design. Just like how the performance of biological brains has little to do with proteins being held together by van der Waals forces, instead of the much stronger covalent bonds that hold together silicon. Sometimes evolution gets stuck with poor building material, and it can't refactor because it can't execute simultaneous changes to migrate the design. If biology does some neat tricks with chemistry, it's because of the greater design pressures exerted by natural selection, not because the building materials are so wonderful.

Maybe neurons are just what brains happen to be made out of, because the blind idiot god is too stupid to sit down and invent transistors. All the modules get made out of neurons because that's all there is, even if the cognitive work would be much better-suited to a 2GHz CPU.

"Early attempts to make flying machines often did things like attaching beak onto the front, or trying to make a wing which would flap like a bird's wing. (This extraordinary persistent idea is found in Leonardo's notebooks and in a textbook on airplane design published in 1911.) It is easy for us to smile at such naivete, but one should realize that it made good sense at the time. What birds did was incredible, and nobody really knew how they did it. It always seemed to involve feathers and flapping. Maybe the beak was critical for stability..."
-- Hayes and Ford, "Turing Test Considered Harmful"

So... why didn't the flapping-wing designs work? Birds flap wings and they fly. The flying machine flaps its wings. Why, oh why, doesn't it fly?

Because... well... it just doesn't. This kind of analogic reasoning is not binding on Reality.

One of the basic tests to apply to reasoning that sounds kinda-good is "How shocked can I justifiably be if Reality comes back and says 'So what'?"

For example: Suppose that, after keeping track of the motions of the planets for centuries, and after confirming the underlying theory (General Relativity) to 14 decimal places, we predict where Mercury will be on July 1st, 2009. So we have a prediction, but that's not the same thing as a fact, right? Anyway, we look up in the sky on July 1st, 2009, and Reality says "So what!" - the planet Mercury has shifted outward to the same orbit as Mars.

In a case like this, I would be highly indignant and would probably sue Reality for breach of contract.

But suppose alternatively that, in the last twenty years, real estate prices have never gone down. You say, "Real estate prices have never gone down - therefore, they won't go down next year!" And next year, Reality says "So what?" It seems to me that you have no right to be shocked. You have used an argument to which Reality can easily say "So what?"

"Nature is the ultimate bigot, because it is obstinately and intolerantly devoted to its own prejudices and absolutely refuses to yield to the most persuasive rationalizations of humans."
-- J. R. Molloy

It's actually pretty hard to find arguments so persuasive that even Reality finds them binding. This is why Science is more difficult - why it's harder to successfully predict reality - than medieval scholars once thought.

One class of persuasive arguments that Reality quite often ignores is the Law of Similarity - that is, the argument that things which look similar ought to behave similarly.

A medieval alchemist puts lemon glazing onto a lump of lead. The lemon glazing is yellow, and gold is yellow. It seems like it ought to work... but the lead obstinately refuses to turn into gold. Reality just comes back and says, "So what? Things can be similar in some aspects without being similar in other aspects."

You should be especially suspicious when someone says, "I am building X, which will do P, because it is similar to Y, which also does P."

An abacus performs addition; and the beads of solder on a circuit board bear a certain surface resemblance to the beads on an abacus. Nonetheless, the circuit board does not perform addition because we can find a surface similarity to the abacus. The Law of Similarity and Contagion is not relevant. The circuit board would work in just the same fashion if every abacus upon Earth vanished in a puff of smoke, or if the beads of an abacus looked nothing like solder. A computer chip is not powered by its similarity to anything else, it just is. It exists in its own right, for its own reasons.

The Wright Brothers calculated that their plane would fly - before it ever flew - using reasoning that took no account whatsoever of their aircraft's similarity to a bird. They did look at birds (and I have looked at neuroscience) but the final calculations did not mention birds (I am fairly confident in asserting). A working airplane does not fly because it has wings "just like a bird". An airplane flies because it is an airplane, a thing that exists in its own right; and it would fly just as high, no more and no less, if no bird had ever existed.

The general form of failing-by-analogy runs something like this:

- You want property P.

- X has property P.

- You build Y, which has one or two surface similarities S to X.

- You argue that Y resembles X and should also P.

- Yet there is no reasoning which you can do on Y as a thing-in-itself to show that it will have property P, regardless of whether or not X had ever existed.

Analogical reasoning of this type is a very weak form of understanding. Reality often says "So what?" and ignores the argument.

The one comes to us and says: "Calculate how many synaptic transmissions per second take place in the human brain. This is the computing power required for human-equivalent intelligence. Raise enough venture capital to buy a supercomputer which performs the same number of floating-point operations per second. Intelligence is bound to emerge from a machine that powerful."

So you reply: "I'm sorry, I've never seen a human brain and I don't know anything about them. So, without talking about a human brain, can you explain how you calculated that 10^17 floating-point operations per second is the exact amount necessary and sufficient to yield human-equivalent intelligence?"

And the one says: "..."

You ask: "Say, what is this property of 'human-equivalent intelligence' which you expect to get? Can you explain it to me without pointing to a human?"

And the one says: "..."

You ask: "What makes you think that large amounts of computing power have something to do with 'intelligence', anyway? Can you answer without pointing to the example of the human brain? Pretend that I've never seen an 'intelligence' and that I have no reason as yet to believe any such thing can exist."

But you get the idea.

Now imagine that you go to the Wright Brothers and say: "I've never seen a bird. Why does your aircraft have 'wings'? And what is it you mean by 'flying'?"

And the Wright Brothers respond: "Well, by flying, we mean that this big heavy object is going to rise off the ground and move through the air without being supported. Once the plane is moving forward, the wings accelerate air downward, which generates lift that keeps the plane aloft."

If two processes have forms that are nearly identical, including internal structure that is similar to as many decimal places as you care to reason about, then you may be able to almost-prove results from one to the other. But if there is even one difference in the internal structure, then any number of other similarities may be rendered void. Two deterministic computations with identical data and identical rules will yield identical outputs. But if a single input bit is flipped from zero to one, the outputs are no longer required to have anything in common. The strength of analogical reasoning can be destroyed by a single perturbation.

Yes, sometimes analogy works. But the more complex and dissimilar the objects are, the less likely it is to work. The narrower the conditions required for success, the less likely it is to work. The more complex the machinery doing the job, the less likely it is to work. The more shallow your understanding of the object of the analogy, the more you are looking at its surface characteristics rather than its deep mechanisms, the less likely analogy is to work.

Analogy might work for something on the order of: "I crossed this river using a fallen log last time, so if I push another log across it, I might be able to cross it again." It doesn't work for creating objects of the order of complexity of, say, a toaster oven. (And hunter-gatherer bands face many rivers to cross, but not many toaster ovens to rewire.)

Admittedly, analogy often works in mathematics - much better than it does in science, in fact. In mathematics you can go back and prove the idea which analogy originally suggested. In mathematics, you get quick feedback about which analogies worked and which analogies didn't, and soon you pick up the pattern. And in mathematics you can always see the entire insides of things; you are not stuck examining the surface of an opaque mystery. Mathematical proposition A may be analogous to mathematical proposition B, which suggests the method; but afterward you can go back and prove A in its own right, regardless of whether or not B is true. In some cases you may need proposition B as a lemma, but certainly not all cases.

Which is to say: despite the misleading surface similarity, the "analogies" which mathematicians use are not analogous to the "analogies" of alchemists, and you cannot reason from the success of one to the success of the other.

Referenced by: Failure By Affective Analogy • Artificial Mysterious Intelligence • Disjunctions, Antipredictions, Etc. • Getting Nearer

Original with comments: Failure By Analogy

Failure By Affective Analogy

Eliezer Yudkowsky, 18 November 2008 07:14AM

Previously in series: Failure By Analogy

Alchemy is a way of thinking that humans do not instinctively spot as stupid. Otherwise alchemy would never have been popular, even in medieval days. Turning lead into gold by mixing it with things that seemed similar to gold, sounded every bit as reasonable, back in the day, as trying to build a flying machine with flapping wings. (And yes, it was worth trying once, but you should notice if Reality keeps saying "So what?")

And the final and most dangerous form of failure by analogy is to say a lot of nice things about X, which is similar to Y, so we should expect nice things of Y. You may also say horrible things about Z, which is the polar opposite of Y, so if Z is bad, Y should be good.

Call this "failure by affective analogy".

Failure by affective analogy is when you don't just say, "This lemon glazing is yellow, gold is yellow, QED." But rather say:

"And now we shall add delicious lemon glazing to the formula for the Philosopher's Stone the root of all wisdom, since lemon glazing is beautifully yellow, like gold is beautifully yellow, and also lemon glazing is delightful on the tongue, indicating that it is possessed of a superior potency that delights the senses, just as the beauty of gold delights the senses..."

That's why you find people saying things like, "Neural networks are decentralized, just like democracies" or "Neural networks are emergent, just like capitalism".

A summary of the Standard Prepackaged Revolutionary New AI Paradigm might look like the following - and when reading, ask yourself how many of these ideas are affectively laden:

- The Dark Side is Top-Down. The Light Side is Bottom-Up.

- The Dark Side is Centralized. The Light Side is Distributed.

- The Dark Side is Logical. The Light Side is Fuzzy.

- The Dark Side is Serial. The Light Side is Parallel.

- The Dark Side is Rational. The Light Side is Intuitive.

- The Dark Side is Deterministic. The Light Side is Stochastic.

- The Dark Side tries to Prove things. The Light Side tries to Test them.

- The Dark Side is Hierarchical. The Light Side is Heterarchical.

- The Dark Side is Clean. The Light Side is Noisy.

- The Dark Side operates in Closed Worlds. The Light Side operates in Open Worlds.

- The Dark Side is Rigid. The Light Side is Flexible.

- The Dark Side is Sequential. The Light Side is Realtime.

- The Dark Side demands Control and Uniformity. The Light Side champions Freedom and Individuality.

- The Dark Side is Lawful. The Light Side, on the other hand, is Creative.

By means of this tremendous package deal fallacy, lots of good feelings are generated about the New Idea (even if it's thirty years old). Enough nice words may even manage to start an affective death spiral. Until finally, via the standard channels of affect heuristic and halo effect, it seems that the New Idea will surely be able to accomplish some extremely difficult end -

- like, say, true general intelligence -

- even if you can't quite give a walkthrough of the internal mechanisms which are going to produce that output.

(Why yes, I have seen AGIfolk trying to pull this on Friendly AI - as they explain how all we need to do is stamp the AI with the properties of Democracy and Love and Joy and Apple Pie and paint an American Flag on the case, and surely it will be Friendly as well - though they can't quite walk through internal cognitive mechanisms.)

From such reasoning as this (and this), came the string of false promises that were published in the newspapers (and led futurists who grew up during that era to be very disappointed in AI, leading them to feel negative affect that now causes them to put AI a hundred years in the future).

Let's say it again: Reversed stupidity is not intelligence - if people are making bad predictions you should just discard them, not reason from their failure.

But there is a certain lesson to be learned. A bounded rationalist cannot do all things, but the true Way should not overpromise - it should not (systematically/regularly/on average) hold out the prospect of success, and then deliver failure. Even a bounded rationalist can aspire to be well calibrated, to not assign 90% probability unless they really do have good enough information to be right nine times out of ten. If you only have good enough information to be right 6 times out of 10, just say 60% instead. A bounded rationalist cannot do all things, but the true Way does not overpromise.

If you want to avoid failed promises of AI... then history suggests, I think, that you should not expect good things out of your AI system unless you have a good idea of how specifically it is going to happen. I don't mean writing out the exact internal program state in advance. But I also don't mean saying that the refrigeration unit will cool down the AI and make it more contemplative. For myself, I seek to know the laws governing the AI's lawful uncertainty and lawful creativity - though I don't expect to know the full content of its future knowledge, or the exact design of its future inventions.

Don't want to be disappointed? Don't hope!

Don't ask yourself if you're allowed to believe that your AI design will work.

Don't guess. Know.

For this much I do know - if I don't know that my AI design will work, it won't.

There are various obvious caveats that need to be attached here, and various obvious stupid interpretations of this principle not to make. You can't be sure a search will return successfully before you have run it -

- but you should understand on a gut level: If you are hoping that your AI design will work, it will fail. If you know that your AI design will work, then it might work.

And on the Friendliness part of that you should hold yourself to an even higher standard - ask yourself if you are forced to believe the AI will be Friendly - because in that aspect, above all, you must constrain Reality so tightly that even Reality is not allowed to say, "So what?" This is a very tough test, but if you do not apply it, you will just find yourself trying to paint a flag on the case, and hoping.

Referenced by: Artificial Mysterious Intelligence • Getting Nearer

Original with comments: Failure By Affective Analogy

The Weak Inside View

Eliezer Yudkowsky, 18 November 2008 06:37PM

Followup to: The Outside View's Domain

When I met Robin in Oxford for a recent conference, we had a preliminary discussion on the Singularity - this is where Robin suggested using production functions. And at one point Robin said something like, "Well, let's see whether your theory's predictions fit previously observed growth rate curves," which surprised me, because I'd never thought of that at all.

It had never occurred to me that my view of optimization ought to produce quantitative predictions. It seemed like something only an economist would try to do, as 'twere. (In case it's not clear, sentence 1 is self-deprecating and sentence 2 is a compliment to Robin. --EY)

Looking back, it's not that I made a choice to deal only in qualitative predictions, but that it didn't really occur to me to do it any other way.

Perhaps I'm prejudiced against the Kurzweilian crowd, and their Laws of Accelerating Change and the like. Way back in the distant beginning that feels like a different person, I went around talking about Moore's Law and the extrapolated arrival time of "human-equivalent hardware" a la Moravec. But at some point I figured out that if you weren't exactly reproducing the brain's algorithms, porting cognition to fast serial hardware and to human design instead of evolved adaptation would toss the numbers out the window - and that how much hardware you needed depended on how smart you were - and that sort of thing.

Betrayed, I decided that the whole Moore's Law thing was silly and a corruption of futurism, and I restrained myself to qualitative predictions (and retrodictions) thenceforth.

Though this is to some extent an argument produced after the conclusion, I would explain my reluctance to venture into quantitative futurism, via the following trichotomy:

- On problems whose pieces are individually precisely predictable, you can use the Strong Inside View to calculate a final outcome that has never been seen before - plot the trajectory of the first moon rocket before it is ever launched, or verify a computer chip before it is ever manufactured.

- On problems that are drawn from a barrel of causally similar problems, where human optimism runs rampant and unforeseen troubles are common, the Outside View beats the Inside View. Trying to visualize the course of history piece by piece, will turn out to not (for humans) work so well, and you'll be better off assuming a probable distribution of results similar to previous historical occasions - without trying to adjust for all the reasons why this time will be different and better.

- But on problems that are new things under the Sun, where there's a huge change of context and a structural change in underlying causal forces, the Outside View also fails - try to use it, and you'll just get into arguments about what is the proper domain of "similar historical cases" or what conclusions can be drawn therefrom. In this case, the best we can do is use the Weak Inside View - visualizing the causal process - to produce loose qualitative conclusions about only those issues where there seems to be lopsided support.

So to me it seems "obvious" that my view of optimization is only strong enough to produce loose qualitative conclusions, and that it can only be matched to its retrodiction of history, or wielded to produce future predictions, on the level of qualitative physics.

"Things should speed up here", I could maybe say. But not "The doubling time of this exponential should be cut in half."

I aspire to a deeper understanding of intelligence than this, mind you. But I'm not sure that even perfect Bayesian enlightenment, would let me predict quantitatively how long it will take an AI to solve various problems in advance of it solving them. That might just rest on features of an unexplored solution space which I can't guess in advance, even though I understand the process that searches.

Robin keeps asking me what I'm getting at by talking about some reasoning as "deep" while other reasoning is supposed to be "surface". One thing which makes me worry that something is "surface", is when it involves generalizing a level N feature across a shift in level N-1 causes.

For example, suppose you say "Moore's Law has held for the last sixty years, so it will hold for the next sixty years, even after the advent of superintelligence" (as Kurzweil seems to believe, since he draws his graphs well past the point where you're buying "a billion times human brainpower for $1000").

Now, if the Law of Accelerating Change were an exogenous, ontologically fundamental, precise physical law, then you wouldn't expect it to change with the advent of superintelligence.

But to the extent that you believe Moore's Law depends on human engineers, and that the timescale of Moore's Law has something to do with the timescale on which human engineers think, then extrapolating Moore's Law across the advent of superintelligence is extrapolating it across a shift in the previous causal generator of Moore's Law.

So I'm worried when I see generalizations extrapolated across a change in causal generators not themselves described - i.e. the generalization itself is on a level of the outputs of those generators and doesn't describe the generators directly.

If, on the other hand, you extrapolate Moore's Law out to 2015 because it's been reasonably steady up until 2008 - well, Reality is still allowed to say "So what?", to a greater extent than we can expect to wake up one morning and find Mercury in Mars's orbit. But I wouldn't bet against you, if you just went ahead and drew the graph.

So what's "surface" or "deep" depends on what kind of context shifts you try to extrapolate past.

Robin Hanson said:

Taking a long historical long view, we see steady total growth rates punctuated by rare transitions when new faster growth modes appeared with little warning. We know of perhaps four such "singularities": animal brains (~600MYA), humans (~2MYA), farming (~1OKYA), and industry (~0.2KYA). The statistics of previous transitions suggest we are perhaps overdue for another one, and would be substantially overdue in a century. The next transition would change the growth rate rather than capabilities directly, would take a few years at most, and the new doubling time would be a week to a month.

Why do these transitions occur? Why have they been similar to each other? Are the same causes still operating? Can we expect the next transition to be similar for the same reasons?

One may of course say, "I don't know, I just look at the data, extrapolate the line, and venture this guess - the data is more sure than any hypotheses about causes." And that will be an interesting projection to make, at least.

But you shouldn't be surprised at all if Reality says "So what?" I mean - real estate prices went up for a long time, and then they went down. And that didn't even require a tremendous shift in the underlying nature and causal mechanisms of real estate.

To stick my neck out further: I am liable to trust the Weak Inside View over a "surface" extrapolation, if the Weak Inside View drills down to a deeper causal level and the balance of support is sufficiently lopsided.

I will go ahead and say, "I don't care if you say that Moore's Law has held for the last hundred years. Human thought was a primary causal force in producing Moore's Law, and your statistics are all over a domain of human neurons running at the same speed. If you substitute better-designed minds running at a million times human clock speed, the rate of progress ought to speed up - qualitatively speaking."

That is, the prediction is without giving precise numbers, or supposing that it's still an exponential curve; computation might spike to the limits of physics and then stop forever, etc. But I'll go ahead and say that the rate of technological progress ought to speed up, given the said counterfactual intervention on underlying causes to increase the thought speed of engineers by a factor of a million. I'll be downright indignant if Reality says "So what?" and has the superintelligence make slower progress than human engineers instead. It really does seem like an argument so strong that even Reality ought to be persuaded.

It would be interesting to ponder what kind of historical track records have prevailed in such a clash of predictions - trying to extrapolate "surface" features across shifts in underlying causes without speculating about those underlying causes, versus trying to use the Weak Inside View on those causes and arguing that there is "lopsided" support for a qualitative conclusion; in a case where the two came into conflict...

...kinda hard to think of what that historical case would be, but perhaps I only lack history.

Robin, how surprised would you be if your sequence of long-term exponentials just... didn't continue? If the next exponential was too fast, or too slow, or something other than an exponential? To what degree would you be indignant, if Reality said "So what?"

Referenced by: Underconstrained Abstractions • Disjunctions, Antipredictions, Etc. • Seduced by Imagination • "Outside View!" as Conversation-Halter

Original with comments: The Weak Inside View

The First World Takeover

Eliezer Yudkowsky, 19 November 2008 03:00PM

Before Robin and I move on to talking about the Future, it seems to me wise to check if we have disagreements in our view of the Past. Which might be much easier to discuss - and maybe even resolve... So...

In the beginning was the Bang. For nine billion years afterward, nothing much happened.

Stars formed, and burned for long periods or short periods depending on their structure; but "successful" stars that burned longer or brighter did not pass on their characteristics to other stars. The first replicators were yet to come.

It was the Day of the Stable Things, when your probability of seeing something was given by its probability of accidental formation times its duration. Stars last a long time; there are many helium atoms.

It was the Era of Accidents, before the dawn of optimization. You'd only expect to see something with 40 bits of optimization if you looked through a trillion samples. Something with 1000 bits' worth of functional complexity? You wouldn't expect to find that in the whole universe.

I would guess that, if you were going to be stuck on a desert island and you wanted to stay entertained as long as possible, then you should sooner choose to examine the complexity of the cells and biochemistry of a single Earthly butterfly, over all the stars and astrophysics in the visible universe beyond Earth.

It was the Age of Boredom.

The hallmark of the Age of Boredom was not lack of natural resources - it wasn't that the universe was low on hydrogen - but, rather, the lack of any cumulative search. If one star burned longer or brighter, that didn't affect the probability distribution of the next star to form. There was no search but blind search. Everything from scratch, not even looking at the neighbors of previously successful points. Not hill-climbing, not mutation and selection, not even discarding patterns already failed. Just a random sample from the same distribution, over and over again.

The Age of Boredom ended with the first replicator.

(Or the first replicator to catch on, if there were failed alternatives lost to history - but this seems unlikely, given the Fermi Paradox; a replicator should be more improbable than that, or the stars would teem with life already.)

Though it might be most dramatic to think of a single RNA strand a few dozen bases long, forming by pure accident after who-knows-how-many chances on who-knows-how-many planets, another class of hypotheses deals with catalytic hypercycles - chemicals whose presence makes it more likely for other chemicals to form, with the arrows happening to finally go around in a circle. If so, RNA would just be a crystallization of that hypercycle into a single chemical that can both take on enzymatic shapes, and store information in its sequence for easy replication.

The catalytic hypercycle is worth pondering, since it reminds us that the universe wasn't quite drawing its random patterns from the same distribution every time - the formation of a longlived star made it more likely for a planet to form (if not another star to form), the formation of a planet made it more likely for amino acids and RNA bases to form in a pool of muck somewhere (if not more likely for planets to form).

In this flow of probability, patterns in one attractor leading to other attractors becoming stronger, there was finally born a cycle - perhaps a single strand of RNA, perhaps a crystal in clay, perhaps a catalytic hypercycle - and that was the dawn.

What makes this cycle significant? Is it the large amount of material that the catalytic hypercycle or replicating RNA strand could absorb into its pattern?

Well, but any given mountain on Primordial Earth would probably weigh vastly more than the total mass devoted to copies of the first replicator. What effect does mere mass have on optimization?

Suppose the first replicator had a probability of formation of 10-30. If that first replicator managed to make 10,000,000,000 copies of itself (I don't know if this would be an overestimate or underestimate for a tidal pool) then this would increase your probability of encountering the replicator-pattern by a factor of 10^10, the total probability going up to 10-20. (If you were observing "things" at random, that is, and not just on Earth but on all the planets with tidal pools.) So that was a kind of optimization-directed probability flow.

But vastly more important, in the scheme of things, was this - that the first replicator made copies of itself, and some of those copies were errors.

That is, it explored the neighboring regions of the search space - some of which contained better replicators - and then those replicators ended up with more probability flowing into them, which explored their neighborhoods.

Even in the Age of Boredom there were always regions of attractor-space that were the gateways to other regions of attractor-space. Stars begot planets, planets begot tidal pools. But that's not the same as a replicator begetting a replicator - it doesn't search a neighborhood, find something that better matches a criterion (in this case, the criterion of effective replication) and then search that neighborhood, over and over.

This did require a certain amount of raw material to act as replicator feedstock. But the significant thing was not how much material was recruited into the world of replication; the significant thing was the search, and the material just carried out that search. If, somehow, there'd been some way of doing the same search without all that raw material - if there'd just been a little beeping device that determined how well a pattern would replicate, and incremented a binary number representing "how much attention" to pay to that pattern, and then searched neighboring points in proportion to that number - well, that would have searched just the same. It's not something that evolution can do, but if it happened, it would generate the same information.

Human brains routinely outthink the evolution of whole species, species whose net weights of biological material outweigh a human brain a million times over - the gun against a lion's paws. It's not the amount of raw material, it's the search.

In the evolution of replicators, the raw material happens to carry out the search - but don't think that the key thing is how much gets produced, how much gets consumed. The raw material is just a way of keeping score. True, even in principle, you do need some negentropy and some matter to perform the computation. But the same search could theoretically be performed with much less material - examining fewer copies of a pattern, to draw the same conclusions, using more efficient updating on the evidence. Replicators happen to use the number of copies produced of themselves, as a way of keeping score.

But what really matters isn't the production, it's the search.

If, after the first primitive replicators had managed to produce a few tons of themselves, you deleted all those tons of biological material, and substituted a few dozen cells here and there from the future - a single algae, a single bacterium - to say nothing of a whole multicellular C. elegans earthworm with a 302-neuron brain - then Time would leap forward by billions of years, even if the total mass of Life had just apparently shrunk. The search would have leapt ahead, and production would recover from the apparent "setback" in a handful of easy doublings.

The first replicator was the first great break in History - the first Black Swan that would have been unimaginable by any surface analogy. No extrapolation of previous trends could have spotted it - you'd have had to dive down into causal modeling, in enough detail to visualize the unprecedented search.

Not that I'm saying I would have guessed, without benefit of hindsight - if somehow I'd been there as a disembodied and unreflective spirit, knowing only the previous universe as my guide - having no highfalutin' concepts of "intelligence" or "natural selection" because those things didn't exist in my environment, and I had no mental mirror in which to see myself - and indeed, who should have guessed it with short of godlike intelligence? When all the previous history of the universe contained no break in History that sharp? The replicator was the first Black Swan.

Maybe I, seeing the first replicator as a disembodied unreflective spirit, would have said, "Wow, what an amazing notion - some of the things I see won't form with high probability, or last for long times - they'll be things that are good at copying themselves, instead. It's the new, third reason for seeing a lot of something!" But would I have been imaginative enough to see the way to amoebas, to birds, to humans? Or would I have just expected it to hit the walls of the tidal pool and stop?

Try telling a disembodied spirit who had watched the whole history of the universe up to that point about the birds and the bees, and they would think you were absolutely and entirely out to lunch. For nothing remotely like that would have been found anywhere else in the universe - and it would obviously take an exponential and ridiculous amount of time to accidentally form a pattern like that, no matter how good it was at replicating itself once formed - and as for it happening many times over in a connected ecology, when the first replicator in the tidal pool took such a long time to happen - why, that would just be madness. The Absurdity Heuristic would come into play. Okay, it's neat that a little molecule can replicate itself - but this notion of a "squirrel" is insanity. So far beyond a Black Swan that you can't even call it a swan anymore.

That first replicator took over the world - in what sense? Earth's crust, Earth's magma, far outweighs its mass of Life. But Robin and I both suspect, I think, that the fate of the universe, and all those distant stars that outweigh us, will end up shaped by Life. So that the universe ends up hanging quite heavily on the existence of that first replicator, and not on the counterfactual states of any particular other molecules nearby... In that sense, a small handful of atoms once seized the reins of Destiny.

How? How did the first replicating pattern take over the world? Why didn't all those other molecules get an equal vote in the process?

Well, that initial replicating pattern was doing some kind of search - some kind of optimization - and nothing else in the Universe was even trying. Really it was evolution that took over the world, not the first replicating pattern per se - you don't see many copies of it around any more. But still, once upon a time the thread of Destiny was seized and concentrated and spun out from a small handful of atoms.

The first replicator did not set in motion a clever optimization process. Life didn't even have sex yet, or DNA to store information at very high fidelity. But the rest of the Universe had zip. In the kingdom of blind chance, the myopic optimization process is king.

Issues of "sharing improvements" or "trading improvements" wouldn't even arise - there were no partners from outside. All the agents, all the actors of our modern world, are descended from that first replicator, and none from the mountains and hills.

And that was the story of the First World Takeover, when a shift in the structure of optimization - namely, moving from no optimization whatsoever, to natural selection - produced a stark discontinuity with previous trends; and squeezed the flow of the whole universe's destiny through the needle's eye of a single place and time and pattern.

That's Life.

Referenced by: Life's Story Continues • True Sources of Disagreement • "Outside View!" as Conversation-Halter

Original with comments: The First World Takeover

Whence Your Abstractions?

Eliezer Yudkowsky, 20 November 2008 01:07AM

Reply to: Abstraction, Not Analogy

Robin asks:

Eliezer, have I completely failed to communicate here? You have previously said nothing is similar enough to this new event for analogy to be useful, so all we have is "causal modeling" (though you haven't explained what you mean by this in this context). This post is a reply saying, no, there are more ways using abstractions; analogy and causal modeling are two particular ways to reason via abstractions, but there are many other ways.

Well... it shouldn't be surprising if you've communicated less than you thought. Two people, both of whom know that disagreement is not allowed, have a persistent disagreement. It doesn't excuse anything, but - wouldn't it be more surprising if their disagreement rested on intuitions that were easy to convey in words, and points readily dragged into the light?

I didn't think from the beginning that I was succeeding in communicating. Analogizing Doug Engelbart's mouse to a self-improving AI is for me such a flabbergasting notion - indicating such completely different ways of thinking about the problem - that I am trying to step back and find the differing sources of our differing intuitions.

(Is that such an odd thing to do, if we're really following down the path of not agreeing to disagree?)

"Abstraction", for me, is a word that means a partitioning of possibility - a boundary around possible things, events, patterns. They are in no sense neutral; they act as signposts saying "lump these things together for predictive purposes". To use the word "singularity" as ranging over human brains, farming, industry, and self-improving AI, is very nearly to finish your thesis right there.

I wouldn't be surprised to find that, in a real AI, 80% of the actual computing crunch goes into drawing the right boundaries to make the actual reasoning possible. The question "Where do abstractions come from?" cannot be taken for granted.

Boundaries are drawn by appealing to other boundaries. To draw the boundary "human" around things that wear clothes and speak language and have a certain shape, you must have previously noticed the boundaries around clothing and language. And your visual cortex already has a (damned sophisticated) system for categorizing visual scenes into shapes, and the shapes into categories.

It's very much worth distinguishing between boundaries drawn by noticing a set of similarities, and boundaries drawn by reasoning about causal interactions.

There's a big difference between saying "I predict that Socrates, like other humans I've observed, will fall into the class of 'things that die when drinking hemlock'" and saying "I predict that Socrates, whose biochemistry I've observed to have this-and-such characteristics, will have his neuromuscular junction disrupted by the coniine in the hemlock - even though I've never seen that happen, I've seen lots of organic molecules and I know how they behave."

But above all - ask where the abstraction comes from!

To see a hammer is not good to hold high in a lightning storm, we draw on pre-existing objects that you're not supposed to hold electrically conductive things to high altitudes - this is a predrawn boundary, found by us in books; probably originally learned from experience and then further explained by theory. We just test the hammer to see if it fits in a pre-existing boundary, that is, a boundary we drew before we ever thought about the hammer.

To evaluate the cost to carry a hammer in a tool kit, you probably visualized the process of putting the hammer in the kit, and the process of carrying it. Its mass determines the strain on your arm muscles. Its volume and shape - not just "volume", as you can see as soon as that is pointed out - determine the difficulty of fitting it into the kit. You said "volume and mass" but that was an approximation, and as soon as I say "volume and mass and shape" you say, "Oh, of course that's what I meant" - based on a causal visualization of trying to fit some weirdly shaped object into a toolkit, or e.g. a thin ten-foot thin pin of low volume and high annoyance. So you're redrawing the boundary based on a causal visualization which shows that other characteristics can be relevant to the consequence you care about.

None of your examples talk about drawing new conclusions about the hammer by analogizing it to other things rather than directly assessing its characteristics in their own right, so it's not all that good an example when it comes to making predictions about self-improving AI by putting it into a group of similar things that includes farming or industry.

But drawing that particular boundary would already rest on causal reasoning that tells you which abstraction to use. Very much an Inside View, and a Weak Inside View, even if you try to go with an Outside View after that.

Using an "abstraction" that covers such massively different things, will often be met by a differing intuition that makes a different abstraction, based on a different causal visualization behind the scenes. That's what you want to drag into the light - not just say, "Well, I expect this Singularity to resemble past Singularities."

Robin said:

I am of course open to different way to conceive of "the previous major singularities". I have previously tried to conceive of them in terms of sudden growth speedups.

Is that the root source for your abstraction - "things that do sudden growth speedups"? I mean... is that really what you want to go with here?

Referenced by: Life's Story Continues

Original with comments: Whence Your Abstractions?

Observing Optimization

Eliezer Yudkowsky, 21 November 2008 05:39AM

Followup to: Optimization and the Singularity

In "Optimization and the Singularity" I pointed out that history since the first replicator, including human history to date, has mostly been a case of nonrecursive optimization - where you've got one thingy doing the optimizing, and another thingy getting optimized. When evolution builds a better amoeba, that doesn't change the structure of evolution - the mutate-reproduce-select cycle.

But there are exceptions to this rule, such as the invention of sex, which affected the structure of natural selection itself - transforming it to mutate-recombine-mate-reproduce-select.

I was surprised when Robin, in "Eliezer's Meta-Level Determinism" took that idea and ran with it and said:

...his view does seem to make testable predictions about history. It suggests the introduction of natural selection and of human culture coincided with the very largest capability growth rate increases. It suggests that the next largest increases were much smaller and coincided in biology with the introduction of cells and sex, and in humans with the introduction of writing and science. And it suggests other rate increases were substantially smaller.

It hadn't occurred to me to try to derive that kind of testable prediction. Why? Well, partially because I'm not an economist. (Don't get me wrong, it was a virtuous step to try.) But also because the whole issue looked to me like it was a lot more complicated than that, so it hadn't occurred to me to try to directly extract predictions.

What is this "capability growth rate" of which you speak, Robin? There are old, old controversies in evolutionary biology involved here.

Just to start by pointing out the obvious - if there are fixed resources available, only so much grass to be eaten or so many rabbits to consume, then any evolutionary "progress" that we would recognize as producing a better-designed organism, may just result in the displacement of the old allele by the new allele - not any increase in the population as a whole. It's quite possible to have a new wolf that expends 10% more energy per day to be 20% better at hunting, and in this case the sustainable wolf population will decrease as new wolves replace old.

If I was going to talk about the effect that a meta-level change might have on the "optimization velocity" of natural selection, I would talk about the time for a new adaptation to replace an old adaptation after a shift in selection pressures - not the total population or total biomass or total morphological complexity (see below).

Likewise in human history - farming was an important innovation for purposes of optimization, not because it changed the human brain all that much, but because it meant that there were a hundred times as many brains around; and even more importantly, that there were surpluses that could support specialized professions. But many innovations in human history may have consisted of new, improved, more harmful weapons - which would, if anything, have decreased the sustainable population size (though "no effect" is more likely - fewer people means more food means more people).

Or similarly: there's a talk somewhere where either Warren Buffett or Charles Munger mentions how they hate to hear about technological improvements in certain industries - because even if investing a few million can cut the cost of production by 30% or whatever, the barriers to competition are so low that the consumer captures all the gain. So they have to invest to keep up with competitors, and the investor doesn't get much return.

I'm trying to measure the optimization velocity of information, not production or growth rates. At the tail end of a very long process, knowledge finally does translate into power - guns or nanotechnology or whatever. But along that long way, if you're measuring the number of material copies of the same stuff (how many wolves, how many people, how much grain), you may not be getting much of a glimpse at optimization velocity. Too many complications along the causal chain.

And this is not just my problem.

Back in the bad old days of pre-1960s evolutionary biology, it was widely taken for granted that there was such a thing as progress, that it proceeded forward over time, and that modern human beings were at the apex.

George Williams's Adaptation and Natural Selection, marking the so-called "Williams Revolution" in ev-bio that flushed out a lot of the romanticism and anthropomorphism, spent most of one chapter questioning the seemingly common-sensical metrics of "progress".

Biologists sometimes spoke of "morphological complexity" increasing over time. But how do you measure that, exactly? And at what point in life do you measure it if the organism goes through multiple stages? Is an amphibian more advanced than a mammal, since its genome has to store the information for multiple stages of life?

"There are life cycles enormously more complex than that of a frog," Williams wrote. "The lowly and 'simple' liver fluke..." goes through stages that include a waterborne stage that swims using cilia; finds and burrows into a snail and then transforms into a sporocyst; that reproduces by budding to produce redia; that migrate in the snail and reproduce asexually; then transform into cercaria, that, by wiggling a tail, burrows out of the snail and swims to a blade of grass; where they transform into dormant metacercaria; that are eaten by sheep and then hatch into a young fluke inside the sheep; then transform into adult flukes; which spawn fluke zygotes... So how "advanced" is that?

Williams also pointed out that there would be a limit to how much information evolution could maintain in the genome against degenerative pressures - which seems like a good principle in practice, though I made some mistakes on OB in trying to describe the theory. Taxonomists often take a current form and call the historical trend toward it "progress", but is that upward motion, or just substitution of some adaptations for other adaptations in response to changing selection pressures?

"Today the fishery biologists greatly fear such archaic fishes as the bowfin, garpikes , and lamprey, because they are such outstandingly effective competitors," Williams noted.

So if I were talking about the effect of e.g. sex as a meta-level innovation, then I would expect e.g. an increase in the total biochemical and morphological complexity that could be maintained - the lifting of a previous upper bound, followed by an accretion of information. And I might expect a change in the velocity of new adaptations replacing old adaptations.

But to get from there, to something that shows up in the fossil record - that's not a trivial step.

I recall reading, somewhere or other, about an ev-bio controversy that ensued when one party spoke of the "sudden burst of creativity" represented by the Cambrian explosion, and wondered why evolution was proceeding so much more slowly nowadays. And another party responded that the Cambrian differentiation was mainly visible post hoc - that the groups of animals we have now, first differentiated from one another then, but that at the time the differences were not as large as they loom nowadays. That is, the actual velocity of adaptational change wasn't remarkable by comparison to modern times, and only hindsight causes us to see those changes as "staking out" the ancestry of the major animal groups.

I'd be surprised to learn that sex had no effect on the velocity of evolution. It looks like it should increase the speed and number of substituted adaptations, and also increase the complexity bound on the total genetic information that can be maintained against mutation. But to go from there, to just looking at the fossil record and seeing faster progress - it's not just me who thinks that this jump to phenomenology is tentative, difficult, and controversial.

Should you expect more speciation after the invention of sex, or less? The first impulse is to say "more", because sex seems like it should increase the optimization velocity and speed up time. But sex also creates mutually reproducing populations, that share genes among themselves, as opposed to asexual lineages - so might that act as a centripetal force?

I don't even propose to answer this question, just point out that it is actually quite standard for the phenomenology of evolutionary theories - the question of which observables are predicted - to be a major difficulty. Unless you're dealing with really easy qualitative questions like "Should I find rabbit fossils in the pre-Cambrian?" (I try to only make predictions about AI, using my theory of optimization, when it looks like an easy question.)

Yes, it's more convenient for scientists when theories make easily testable, readily observable predictions. But when I look back at the history of life, and the history of humanity, my first priority is to ask "What's going on here?", and only afterward see if I can manage to make non-obvious retrodictions. I can't just start with the goal of having a convenient phenomenology. Or similarly: the theories I use to organize my understanding of the history of optimization to date, have lots of parameters, e.g. the optimization-efficiency curve that describes optimization output as a function of resource input, or the question of how many low-hanging fruit exist in the neighborhood of a given search point. Does a larger population of wolves increase the velocity of natural selection, by covering more of the search neighborhood for possible mutations? If so, is that a logarithmic increase with population size, or what? - But I can't just wish my theories into being simpler.

If Robin has a simpler causal model, with fewer parameters, that stands directly behind observables and easily coughs up testable predictions, which fits the data well, and obviates the need for my own abstractions like "optimization efficiency" -

- then I may have to discard my own attempts at theorizing. But observing a series of material growth modes doesn't contradict a causal model of optimization behind the scenes, because it's a pure phenomenology, not itself a causal model - it doesn't say whether a given innovation had any effect on the optimization velocity of the process that produced future object-level innovations that actually changed growth modes, etcetera.

Original with comments: Observing Optimization

Life's Story Continues

Eliezer Yudkowsky, 21 November 2008 11:05PM

Followup to: The First World Takeover

As last we looked at the planet, Life's long search in organism-space had only just gotten started.

When I try to structure my understanding of the unfolding process of Life, it seems to me that, to understand the optimization velocity at any given point, I want to break down that velocity using the following abstractions:

- The searchability of the neighborhood of the current location, and the availability of good/better alternatives in that rough region. Maybe call this the optimization slope. Are the fruit low-hanging or high-hanging, and how large are the fruit?

- The optimization resources, like the amount of computing power available to a fixed program, or the number of individuals in a population pool.

- The optimization efficiency, a curve that gives the amount of searchpower generated by a given investiture of resources, which is presumably a function of the optimizer's structure at that point in time.

Example: If an object-level adaptation enables more efficient extraction of resources, and thereby increases the total population that can be supported by fixed available resources, then this increases the optimization resources and perhaps the optimization velocity.

How much does optimization velocity increase - how hard does this object-level innovation hit back to the meta-level?

If a population is small enough that not all mutations are occurring in each generation, then a larger population decreases the time for a given mutation to show up. If the fitness improvements offered by beneficial mutations follow an exponential distribution, then - I'm not actually doing the math here, just sort of eyeballing - I would expect the optimization velocity to go as log population size, up to a maximum where the search neighborhood is explored thoroughly. (You could test this in the lab, though not just by eyeballing the fossil record.)

This doesn't mean all optimization processes would have a momentary velocity that goes as the log of momentary resource investment up to a maximum. Just one mode of evolution would have this character. And even under these assumptions, evolution's cumulative optimization wouldn't go as log of cumulative resources - the log-pop curve is just the instantaneous velocity. If we assume that the variance of the neighborhood remains the same over the course of exploration (good points have better neighbors with same variance ad infinitum), and that the population size remains the same, then we should see linearly cumulative optimization over time. At least until we start to hit the information bound on maintainable genetic information...

These are the sorts of abstractions that I think are required to describe the history of life on Earth in terms of optimization. And I also think that if you don't talk optimization, then you won't be able to understand the causality - there'll just be these mysterious unexplained progress modes that change now and then. In the same way you have to talk natural selection to understand observed evolution, you have to talk optimization velocity to understand observed evolutionary speeds.

The first thing to realize is that meta-level changes are rare, so most of what we see in the historical record will be structured by the search neighborhoods - the way that one innovation opens up the way for additional innovations. That's going to be most of the story, not because meta-level innovations are unimportant, but because they are rare.

In "Eliezer's Meta-Level Determinism", Robin lists the following dramatic events traditionally noticed in the fossil record:

Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual Eukaryotes, Metazoans

And he describes "the last three strong transitions" as:

Humans, farming, and industry

So let me describe what I see when I look at these events, plus some others, through the lens of my abstractions:

Cells: Force a set of genes, RNA strands, or catalytic chemicals to share a common reproductive fate. (This is the real point of the cell boundary, not "protection from the environment" - it keeps the fruits of chemical labor inside a spatial boundary.) But, as we've defined our abstractions, this is mostly a matter of optimization slope - the quality of the search neighborhood. The advent of cells opens up a tremendously rich new neighborhood defined by specialization and division of labor. It also increases the slope by ensuring that chemicals get to keep the fruits of their own labor in a spatial boundary, so that fitness advantages increase. But does it hit back to the meta-level? How you define that seems to me like a matter of taste. Cells don't quite change the mutate-reproduce-select cycle. But if we're going to define sexual recombination as a meta-level innovation, then we should also define cellular isolation as a meta-level innovation.

It's worth noting that modern genetic algorithms have not, to my knowledge, reached anything like the level of intertwined complexity that characterizes modern unicellular organisms. Modern genetic algorithms seem more like they're producing individual chemicals, rather than being able to handle individually complex modules. So the cellular transition may be a hard one.

DNA: I haven't yet looked up the standard theory on this, but I would sorta expect it to come after cells, since a ribosome seems like the sort of thing you'd have to keep around in a defined spatial location. DNA again opens up a huge new search neighborhood by separating the functionality of chemical shape from the demands of reproducing the pattern. Maybe we should rule that anything which restructures the search neighborhood this drastically should count as a hit back to the meta-level. (Whee, our abstractions are already breaking down.) Also, DNA directly hits back to the meta-level by carrying information at higher fidelity, which increases the total storable information.

Filamentous prokaryotes, unicellular eukaryotes: Meh, so what.

Sex: The archetypal example of a rare meta-level innovation. Evolutionary biologists still puzzle over how exactly this one managed to happen.

Metazoans: The key here is not cells aggregating into colonies with similar genetic heritages; the key here is the controlled specialization of cells with an identical genetic heritage. This opens up a huge new region of the search space, but does not particularly change the nature of evolutionary optimization.

Note that opening a sufficiently huge gate in the search neighborhood, may result in a meta-level innovation being uncovered shortly thereafter. E.g. if cells make ribosomes possible. One of the main lessons in this whole history is that one thing leads to another.

Neurons, for example, may have been the key enabling factor in enabling large-motile-animal body plans, because they enabled one side of the organism to talk with the other.

This brings us to the age of brains, which will be the topic of the next post.

But in the meanwhile, I just want to note that my view is nothing as simple as "meta-level determinism" or "the impact of something is proportional to how meta it is; non-meta things must have small impacts". Nothing much meta happened between the age of sexual metazoans and the age of humans - brains were getting more sophisticated over that period, but that didn't change the nature of evolution.

Some object-level innovations are small, some are medium-sized, some are huge. It's no wonder if you look at the historical record and see a Big Innovation that doesn't look the least bit meta, but had a huge impact by itself and led to lots of other innovations by opening up a new neighborhood picture of search space. This is allowed. Why wouldn't it be?

You can even get exponential acceleration without anything meta - if, for example, the more knowledge you have, or the more genes you have, the more opportunities you have to make good improvements to them. Without any increase in optimization pressure, the neighborhood gets higher-sloped as you climb it.

My thesis is more along the lines of, "If this is the picture without recursion, just imagine what's going to happen when we add recursion."

To anticipate one possible objection: I don't expect Robin to disagree that modern civilizations underinvest in meta-level improvements because they take time to yield cumulative effects, are new things that don't have certain payoffs, and worst of all, tend to be public goods. That's why we don't have billions of dollars flowing into prediction markets, for example. I, Robin, or Michael Vassar could probably think for five minutes and name five major probable-big-win meta-level improvements that society isn't investing in.

So if meta-level improvements are rare in the fossil record, it's not necessarily because it would be hard to improve on evolution, or because meta-level improving doesn't accomplish much. Rather, evolution doesn't do anything because it will have a long-term payoff a thousand generations later. Any meta-level improvement also has to grant an object-level fitness advantage in, say, the next two generations, or it will go extinct. This is why we can't solve the puzzle of how sex evolved by pointing directly to how it speeds up evolution. "This speeds up evolution" is just not a valid reason for something to evolve.

Any creative evolutionary biologist could probably think for five minutes and come up with five great ways that evolution could have improved on evolution - but which happen to be more complicated than the wheel, which evolution evolved on only three known occasions - or don't happen to grant an immediate fitness benefit to a handful of implementers.

Referenced by: Surprised by Brains • Recursive Self-Improvement

Original with comments: Life's Story Continues

Surprised by Brains

Eliezer Yudkowsky, 23 November 2008 07:26AM

Followup to: Life's Story Continues

Imagine two agents who've never seen an intelligence - including, somehow, themselves - but who've seen the rest of the universe up until now, arguing about what these newfangled "humans" with their "language" might be able to do...

Believer: Previously, evolution has taken hundreds of thousands of years to create new complex adaptations with many working parts. I believe that, thanks to brains and language, we may see a new era, an era of intelligent design. In this era, complex causal systems - with many interdependent parts that collectively serve a definite function - will be created by the cumulative work of many brains building upon each others' efforts.

Skeptic: I see - you think that brains might have something like a 50% speed advantage over natural selection? So it might take a while for brains to catch up, but after another eight billion years, brains will be in the lead. But this planet's Sun will swell up by then, so -

Believer: Thirty percent? I was thinking more like three orders of magnitude. With thousands of brains working together and building on each others' efforts, whole complex machines will be designed on the timescale of mere millennia - no, centuries!

Skeptic: What?

Believer: You heard me.

Skeptic: Oh, come on! There's absolutely no empirical evidence for an assertion like that! Animal brains have been around for hundreds of millions of years without doing anything like what you're saying. I see no reason to think that life-as-we-know-it will end, just because these hominid brains have learned to send low-bandwidth signals over their vocal cords. Nothing like what you're saying has happened before in my experience -

Believer: That's kind of the point, isn't it? That nothing like this has happened before? And besides, there is precedent for that kind of Black Swan - namely, the first replicator.

Skeptic: Yes, there is precedent in the replicators. Thanks to our observations of evolution, we have extensive knowledge and many examples of how optimization works. We know, in particular, that optimization isn't easy - it takes millions of years to climb up through the search space. Why should "brains", even if they optimize, produce such different results?

Believer: Well, natural selection is just the very first optimization process that got started accidentally. These newfangled brains were designed by evolution, rather than, like evolution itself, being a natural process that got started by accident. So "brains" are far more sophisticated - why, just look at them. Once they get started on cumulative optimization - FOOM!

Skeptic: So far, brains are a lot less impressive than natural selection. These "hominids" you're so interested in - can these creatures' handaxes really be compared to the majesty of a dividing cell?

Believer: That's because they only just got started on language and cumulative optimization.

Skeptic: Really? Maybe it's because the principles of natural selection are simple and elegant for creating complex designs, and all the convolutions of brains are only good for chipping handaxes in a hurry. Maybe brains simply don't scale to detail work. Even if we grant the highly dubious assertion that brains are more efficient than natural selection - which you seem to believe on the basis of just looking at brains and seeing the convoluted folds - well, there still has to be a law of diminishing returns.

Believer: Then why have brains been getting steadily larger over time? That doesn't look to me like evolution is running into diminishing returns. If anything, the recent example of hominids suggests that once brains get large and complicated enough, the fitness advantage for further improvements is even greater -

Skeptic: Oh, that's probably just sexual selection! I mean, if you think that a bunch of brains will produce new complex machinery in just a hundred years, then why not suppose that a brain the size of a whole planet could produce a de novo complex causal system with many interdependent elements in a single day?

Believer: You're attacking a strawman here - I never said anything like that.

Skeptic: Yeah? Let's hear you assign a probability that a brain the size of a planet could produce a new complex design in a single day.

Believer: The size of a planet? (Thinks.) Um... ten percent.

Skeptic: (Muffled choking sounds.)

Believer: Look, brains are fast. I can't rule it out in principle -

Skeptic: Do you understand how long a day is? It's the amount of time for the Earth to spin on its own axis, once. One sunlit period, one dark period. There are 365,242 of them in a single millennium.

Believer: Do you understand how long a second is? That's how long it takes a brain to see a fly coming in, target it in the air, and eat it. There's 86,400 of them in a day.

Skeptic: Pffft, and chemical interactions in cells happen in nanoseconds. Speaking of which, how are these brains going to build any sort of complex machinery without access to ribosomes? They're just going to run around on the grassy plains in really optimized patterns until they get tired and fall over. There's nothing they can use to build proteins or even control tissue structure.

Believer: Well, life didn't always have ribosomes, right? The first replicator didn't.

Skeptic: So brains will evolve their own ribosomes?

Believer: Not necessarily ribosomes. Just some way of making things.

Skeptic: Great, so call me in another hundred million years when that evolves, and I'll start worrying about brains.

Believer: No, the brains will think of a way to get their own ribosome-analogues.

Skeptic: No matter what they think, how are they going to make anything without ribosomes?

Believer: They'll think of a way.

Skeptic: Now you're just treating brains as magic fairy dust.

Believer: The first replicator would have been magic fairy dust by comparison with anything that came before it -

Skeptic: That doesn't license throwing common sense out the window.

Believer: What you call "common sense" is exactly what would have caused you to assign negligible probability to the actual outcome of the first replicator. Ergo, not so sensible as it seems, if you want to get your predictions actually right, instead of sounding reasonable.

Skeptic: And your belief that in the Future it will only take a hundred years to optimize a complex causal system with dozens of interdependent parts - you think this is how you get it right?

Believer: Yes! Sometimes, in the pursuit of truth, you have to be courageous - to stop worrying about how you sound in front of your friends - to think outside the box - to imagine futures fully as absurd as the Present would seem without benefit of hindsight - and even, yes, say things that sound completely ridiculous and outrageous by comparison with the Past. That is why I boldly dare to say - pushing out my guesses to the limits of where Truth drives me, without fear of sounding silly - that in the far future, a billion years from now when brains are more highly evolved, they will find it possible to design a complete machine with a thousand parts in as little as one decade!

Skeptic: You're just digging yourself deeper. I don't even understand how brains are supposed to optimize so much faster. To find out the fitness of a mutation, you've got to run millions of real-world tests, right? And even then, an environmental shift can make all your optimization worse than nothing, and there's no way to predict that no matter how much you test -

Believer: Well, a brain is complicated, right? I've been looking at them for a while and even I'm not totally sure I understand what goes on in there.

Skeptic: Pffft! What a ridiculous excuse.

Believer: I'm sorry, but it's the truth - brains are harder to understand.

Skeptic: Oh, and I suppose evolution is trivial?

Believer: By comparison... yeah, actually.

Skeptic: Name me one factor that explains why you think brains will run so fast.

Believer: Abstraction.

Skeptic: Eh? Abstrah-shun?

Believer: It... um... lets you know about parts of the search space you haven't actually searched yet, so you can... sort of... skip right to where you need to be -

Skeptic: I see. And does this power work by clairvoyance, or by precognition? Also, do you get it from a potion or an amulet?

Believer: The brain looks at the fitness of just a few points in the search space - does some complicated processing - and voila, it leaps to a much higher point!

Skeptic: Of course. I knew teleportation had to fit in here somewhere.

Believer: See, the fitness of one point tells you something about other points -

Skeptic: Eh? I don't see how that's possible without running another million tests.

Believer: You just look at it, dammit!

Skeptic: With what kind of sensor? It's a search space, not a bug to eat!

Believer: The search space is compressible -

Skeptic: Whaa? This is a design space of possible genes we're talking about, not a folding bed -

Believer: Would you stop talking about genes already! Genes are on the way out! The future belongs to ideas!

Skeptic: Give. Me. A. Break.

Believer: Hominids alone shall carry the burden of destiny!

Skeptic: They'd die off in a week without plants to eat. You probably don't know this, because you haven't studied ecology, but ecologies are complicated - no single species ever "carries the burden of destiny" by itself. But that's another thing - why are you postulating that it's just the hominids who go FOOM? What about the other primates? These chimpanzees are practically their cousins - why wouldn't they go FOOM too?

Believer: Because it's all going to shift to the level of ideas, and the hominids will build on each other's ideas without the chimpanzees participating -

Skeptic: You're begging the question. Why won't chimpanzees be part of the economy of ideas? Are you familiar with Ricardo's Law of Comparative Advantage? Even if chimpanzees are worse at everything than hominids, the hominids will still trade with them and all the other brainy animals.

Believer: The cost of explaining an idea to a chimpanzee will exceed any benefit the chimpanzee can provide.

Skeptic: But why should that be true? Chimpanzees only forked off from hominids a few million years ago. They have 95% of their genome in common with the hominids. The vast majority of optimization that went into producing hominid brains also went into producing chimpanzee brains. If hominids are good at trading ideas, chimpanzees will be 95% as good at trading ideas. Not to mention that all of your ideas belong to the far future, so that both hominids, and chimpanzees, and many other species will have evolved much more complex brains before anyone starts building their own cells -

Believer: I think we could see as little as a million years pass between when these creatures first invent a means of storing information with persistent digital accuracy - their equivalent of DNA - and when they build machines as complicated as cells.

Skeptic: Too many assumptions... I don't even know where to start... Look, right now brains are nowhere near building cells. It's going to take a lot more evolution to get to that point, and many other species will be much further along the way by the time hominids get there. Chimpanzees, for example, will have learned to talk -

Believer: It's the ideas that will accumulate optimization, not the brains.

Skeptic: Then I say again that if hominids can do it, chimpanzees will do it 95% as well.

Believer: You might get discontinuous returns on brain complexity. Like... even though the hominid lineage split off from chimpanzees very recently, and only a few million years of evolution have occurred since then, the chimpanzees won't be able to keep up.

Skeptic: Why?

Believer: Good question.

Skeptic: Does it have a good answer?

Believer: Well, there might be compound interest on learning during the maturational period... or something about the way a mind flies through the search space, so that slightly more powerful abstracting-machinery can create abstractions that correspond to much faster travel... or some kind of feedback loop involving a brain powerful enough to control itself... or some kind of critical threshold built into the nature of cognition as a problem, so that a single missing gear spells the difference between walking and flying... or the hominids get started down some kind of sharp slope in the genetic fitness landscape, involving many changes in sequence, and the chimpanzees haven't gotten started down it yet... or all these statements are true and interact multiplicatively... I know that a few million years doesn't seem like much time, but really, quite a lot can happen. It's hard to untangle.

Skeptic: I'd say it's hard to believe.

Believer: Sometimes it seems that way to me too! But I think that in a mere ten or twenty million years, we won't have a choice.

Referenced by: Cascades, Cycles, Insight... • Recursive Self-Improvement • Hard Takeoff • True Sources of Disagreement • "Outside View!" as Conversation-Halter

Original with comments: Surprised by Brains

Cascades, Cycles, Insight...

Eliezer Yudkowsky, 24 November 2008 09:33AM

Followup to: Surprised by Brains

Five sources of discontinuity: 1, 2, and 3...

Cascades are when one thing leads to another. Human brains are effectively discontinuous with chimpanzee brains due to a whole bag of design improvements, even though they and we share 95% genetic material and only a few million years have elapsed since the branch. Why this whole series of improvements in us, relative to chimpanzees? Why haven't some of the same improvements occurred in other primates?

Well, this is not a question on which one may speak with authority (so far as I know). But I would venture an unoriginal guess that, in the hominid line, one thing led to another.

The chimp-level task of modeling others, in the hominid line, led to improved self-modeling which supported recursion which enabled language which birthed politics that increased the selection pressure for outwitting which led to sexual selection on wittiness...

...or something. It's hard to tell by looking at the fossil record what happened in what order and why. The point being that it wasn't one optimization that pushed humans ahead of chimps, but rather a cascade of optimizations that, in Pan, never got started.

We fell up the stairs, you might say. It's not that the first stair ends the world, but if you fall up one stair, you're more likely to fall up the second, the third, the fourth...

I will concede that farming was a watershed invention in the history of the human species, though it intrigues me for a different reason than Robin. Robin, presumably, is interested because the economy grew by two orders of magnitude, or something like that. But did having a hundred times as many humans, lead to a hundred times as much thought-optimization accumulating per unit time? It doesn't seem likely, especially in the age before writing and telephones. But farming, because of its sedentary and repeatable nature, led to repeatable trade, which led to debt records. Aha! - now we have writing. There's a significant invention, from the perspective of cumulative optimization by brains. Farming isn't writing but it cascaded to writing.

Farming also cascaded (by way of surpluses and cities) to support professional specialization. I suspect that having someone spend their whole life thinking about topic X instead of a hundred farmers occasionally pondering it, is a more significant jump in cumulative optimization than the gap between a hundred farmers and one hunter-gatherer pondering something.

Farming is not the same trick as professional specialization or writing, but it cascaded to professional specialization and writing, and so the pace of human history picked up enormously after agriculture. Thus I would interpret the story.

From a zoomed-out perspective, cascades can lead to what look like discontinuities in the historical record, even given a steady optimization pressure in the background. It's not that natural selection sped up during hominid evolution. But the search neighborhood contained a low-hanging fruit of high slope... that led to another fruit... which led to another fruit... and so, walking at a constant rate, we fell up the stairs. If you see what I'm saying.

Predicting what sort of things are likely to cascade, seems like a very difficult sort of problem.

But I will venture the observation that - with a sample size of one, and an optimization process very different from human thought - there was a cascade in the region of the transition from primate to human intelligence.

Cycles happen when you connect the output pipe to the input pipe in a repeatable transformation. You might think of them as a special case of cascades with very high regularity. (From which you'll note that in the cases above, I talked about cascades through differing events: farming -> writing.)

The notion of cycles as a source of discontinuity might seem counterintuitive, since it's so regular. But consider this important lesson of history:

Once upon a time, in a squash court beneath Stagg Field at the University of Chicago, physicists were building a shape like a giant doorknob out of alternate layers of graphite and uranium...

The key number for the "pile" is the effective neutron multiplication factor. When a uranium atom splits, it releases neutrons - some right away, some after delay while byproducts decay further. Some neutrons escape the pile, some neutrons strike another uranium atom and cause an additional fission. The effective neutron multiplication factor, denoted k, is the average number of neutrons from a single fissioning uranium atom that cause another fission. At k less than 1, the pile is "subcritical". At k >= 1, the pile is "critical". Fermi calculates that the pile will reach k=1 between layers 56 and 57.

On December 2nd in 1942, with layer 57 completed, Fermi orders the final experiment to begin. All but one of the control rods (strips of wood covered with neutron-absorbing cadmium foil) are withdrawn. At 10:37am, Fermi orders the final control rod withdrawn about half-way out. The geiger counters click faster, and a graph pen moves upward. "This is not it," says Fermi, "the trace will go to this point and level off," indicating a spot on the graph. In a few minutes the graph pen comes to the indicated point, and does not go above it. Seven minutes later, Fermi orders the rod pulled out another foot. Again the radiation rises, then levels off. The rod is pulled out another six inches, then another, then another.

At 11:30, the slow rise of the graph pen is punctuated by an enormous CRASH - an emergency control rod, triggered by an ionization chamber, activates and shuts down the pile, which is still short of criticality.

Fermi orders the team to break for lunch.

At 2pm the team reconvenes, withdraws and locks the emergency control rod, and moves the control rod to its last setting. Fermi makes some measurements and calculations, then again begins the process of withdrawing the rod in slow increments. At 3:25pm, Fermi orders the rod withdrawn another twelve inches. "This is going to do it," Fermi says. "Now it will become self-sustaining. The trace will climb and continue to climb. It will not level off."

Herbert Anderson recounted (as told in Rhodes's The Making of the Atomic Bomb):

"At first you could hear the sound of the neutron counter, clickety-clack, clickety-clack. Then the clicks came more and more rapidly, and after a while they began to merge into a roar; the counter couldn't follow anymore. That was the moment to switch to the chart recorder. But when the switch was made, everyone watched in the sudden silence the mounting deflection of the recorder's pen. It was an awesome silence. Everyone realized the significance of that switch; we were in the high intensity regime and the counters were unable to cope with the situation anymore. Again and again, the scale of the recorder had to be changed to accomodate the neutron intensity which was increasing more and more rapidly. Suddenly Fermi raised his hand. 'The pile has gone critical,' he announced. No one present had any doubt about it."

Fermi kept the pile running for twenty-eight minutes, with the neutron intensity doubling every two minutes.

That first critical reaction had k of 1.0006.

It might seem that a cycle, with the same thing happening over and over again, ought to exhibit continuous behavior. In one sense it does. But if you pile on one more uranium brick, or pull out the control rod another twelve inches, there's one hell of a big difference between k of 0.9994 and k of 1.0006.

If, rather than being able to calculate, rather than foreseeing and taking cautions, Fermi had just reasoned that 57 layers ought not to behave all that differently from 56 layers - well, it wouldn't have been a good year to be a student at the University of Chicago.

The inexact analogy to the domain of self-improving AI is left as an exercise for the reader, at least for now.

Economists like to measure cycles because they happen repeatedly. You take a potato and an hour of labor and make a potato clock which you sell for two potatoes; and you do this over and over and over again, so an economist can come by and watch how you do it.

As I noted here at some length, economists are much less likely to go around measuring how many scientific discoveries it takes to produce a new scientific discovery. All the discoveries are individually dissimilar and it's hard to come up with a common currency for them. The analogous problem will prevent a self-improving AI from being directly analogous to a uranium heap, with almost perfectly smooth exponential increase at a calculable rate. You can't apply the same software improvement to the same line of code over and over again, you've got to invent a new improvement each time. But if self-improvements are triggering more self-improvements with great regularity, you might stand a long way back from the AI, blur your eyes a bit, and ask: What is the AI's average neutron multiplication factor?

Economics seems to me to be largely the study of production cycles - highly regular repeatable value-adding actions. This doesn't seem to me like a very deep abstraction so far as the study of optimization goes, because it leaves out the creation of novel knowledge and novel designs - further informational optimizations. Or rather, treats productivity improvements as a mostly exogenous factor produced by black-box engineers and scientists. (If I underestimate your power and merely parody your field, by all means inform me what kind of economic study has been done of such things.) (Answered: This literature goes by the name "endogenous growth". See comments starting here.) So far as I can tell, economists do not venture into asking where discoveries come from, leaving the mysteries of the brain to cognitive scientists.

(Nor do I object to this division of labor - it just means that you may have to drag in some extra concepts from outside economics if you want an account of self-improving Artificial Intelligence. Would most economists even object to that statement? But if you think you can do the whole analysis using standard econ concepts, then I'm willing to see it...)

Insight is that mysterious thing humans do by grokking the search space, wherein one piece of highly abstract knowledge (e.g. Newton's calculus) provides the master key to a huge set of problems. Since humans deal in the compressibility of compressible search spaces (at least the part we can compress) we can bite off huge chunks in one go. This is not mere cascading, where one solution leads to another:

Rather, an "insight" is a chunk of knowledge which, if you possess it, decreases the cost of solving a whole range of governed problems.

There's a parable I once wrote - I forget what for, I think ev-bio - which dealt with creatures who'd evolved addition in response to some kind of environmental problem, and not with overly sophisticated brains - so they started with the ability to add 5 to things (which was a significant fitness advantage because it let them solve some of their problems), then accreted another adaptation to add 6 to odd numbers. Until, some time later, there wasn't a reproductive advantage to "general addition", because the set of special cases covered almost everything found in the environment.

There may be even be a real-world example of this. If you glance at a set, you should be able to instantly distinguish the numbers one, two, three, four, and five, but seven objects in an arbitrary (non-canonical pattern) will take at least one noticeable instant to count. IIRC, it's been suggested that we have hardwired numerosity-detectors but only up to five.

I say all this, to note the difference between evolution nibbling bits off the immediate search neighborhood, versus the human ability to do things in one fell swoop.

Our compression of the search space is also responsible for ideas cascading much more easily than adaptations. We actively examine good ideas, looking for neighbors.

But an insight is higher-level than this; it consists of understanding what's "good" about an idea in a way that divorces it from any single point in the search space. In this way you can crack whole volumes of the solution space in one swell foop. The insight of calculus apart from gravity is again a good example, or the insight of mathematical physics apart from calculus, or the insight of math apart from mathematical physics.

Evolution is not completely barred from making "discoveries" that decrease the cost of a very wide range of further discoveries. Consider e.g. the ribosome, which was capable of manufacturing a far wider range of proteins than whatever it was actually making at the time of its adaptation: this is a general cost-decreaser for a wide range of adaptations. It likewise seems likely that various types of neuron have reasonably-general learning paradigms built into them (gradient descent, Hebbian learning, more sophisticated optimizers) that have been reused for many more problems than they were originally invented for.

A ribosome is something like insight: an item of "knowledge" that tremendously decreases the cost of inventing a wide range of solutions. But even evolution's best "insights" are not quite like the human kind. A sufficiently powerful human insight often approaches a closed form - it doesn't feel like you're exploring even a compressed search space. You just apply the insight-knowledge to whatever your problem, and out pops the now-obvious solution.

Insights have often cascaded, in human history - even major insights. But they don't quite cycle - you can't repeat the identical pattern Newton used originally to get a new kind of calculus that's twice and then three times as powerful.

Human AI programmers who have insights into intelligence may acquire discontinuous advantages over others who lack those insights. AIs themselves will experience discontinuities in their growth trajectory associated with becoming able to do AI theory itself - a watershed moment in the FOOM.

Referenced by: ...Recursion, Magic • Engelbart: Insufficiently Recursive • Total Nano Domination • Recursive Self-Improvement • Hard Takeoff • Underconstrained Abstractions • Sustained Strong Recursion • Artificial Mysterious Intelligence • What I Think, If Not Why

Original with comments: Cascades, Cycles, Insight...

...Recursion, Magic

Eliezer Yudkowsky, 25 November 2008 09:10AM

Followup to: Cascades, Cycles, Insight...

...4, 5 sources of discontinuity.

Recursion is probably the most difficult part of this topic. We have historical records aplenty of cascades, even if untangling the causality is difficult. Cycles of reinvestment are the heartbeat of the modern economy. An insight that makes a hard problem easy, is something that I hope you've experienced at least once in your life...

But we don't have a whole lot of experience redesigning our own neural circuitry.

We have these wonderful things called "optimizing compilers". A compiler translates programs in a high-level language, into machine code (though these days it's often a virtual machine). An "optimizing compiler", obviously, is one that improves the program as it goes.

So why not write an optimizing compiler in its own language, and then run it on itself? And then use the resulting optimized optimizing compiler, to recompile itself yet again, thus producing an even more optimized optimizing compiler -

Halt! Stop! Hold on just a minute! An optimizing compiler is not supposed to change the logic of a program - the input/output relations. An optimizing compiler is only supposed to produce code that does the same thing, only faster. A compiler isn't remotely near understanding what the program is doing and why, so it can't presume to construct a better input/output function. We just presume that the programmer wants a fixed input/output function computed as fast as possible, using as little memory as possible.

So if you run an optimizing compiler on its own source code, and then use the product to do the same again, it should produce the same output on both occasions - at most, the first-order product will run faster than the original compiler.

If we want a computer program that experiences cascades of self-improvement, the path of the optimizing compiler does not lead there - the "improvements" that the optimizing compiler makes upon itself, do not improve its ability to improve itself.

Now if you are one of those annoying nitpicky types, like me, you will notice a flaw in this logic: suppose you built an optimizing compiler that searched over a sufficiently wide range of possible optimizations, that it did not ordinarily have time to do a full search of its own space - so that, when the optimizing compiler ran out of time, it would just implement whatever speedups it had already discovered. Then the optimized optimizing compiler, although it would only implement the same logic faster, would do more optimizations in the same time - and so the second output would not equal the first output.

Well... that probably doesn't buy you much. Let's say the optimized program is 20% faster, that is, it gets 20% more done in the same time. Then, unrealistically assuming "optimization" is linear, the 2-optimized program will be 24% faster, the 3-optimized program will be 24.8% faster, and so on until we top out at a 25% improvement. k < 1.

So let us turn aside from optimizing compilers, and consider a more interesting artifact, EURISKO.

To the best of my inexhaustive knowledge, EURISKO may still be the most sophisticated self-improving AI ever built - in the 1980s, by Douglas Lenat before he started wasting his life on Cyc. EURISKO was applied in domains ranging from the Traveller war game (EURISKO became champion without having ever before fought a human) to VLSI circuit design.

EURISKO used "heuristics" to, for example, design potential space fleets. It also had heuristics for suggesting new heuristics, and metaheuristics could apply to any heuristic, including metaheuristics. E.g. EURISKO started with the heuristic "investigate extreme cases" but moved on to "investigate cases close to extremes". The heuristics were written in RLL, which stands for Representation Language Language. According to Lenat, it was figuring out how to represent the heuristics in such fashion that they could usefully modify themselves without always just breaking, that consumed most of the conceptual effort in creating EURISKO.

But EURISKO did not go foom.

EURISKO could modify even the metaheuristics that modified heuristics. EURISKO was, in an important sense, more recursive than either humans or natural selection - a new thing under the Sun, a cycle more closed than anything that had ever existed in this universe.

Still, EURISKO ran out of steam. Its self-improvements did not spark a sufficient number of new self-improvements. This should not really be too surprising - it's not as if EURISKO started out with human-level intelligence plus the ability to modify itself - its self-modifications were either evolutionarily blind, or produced by the simple procedural rules of some heuristic or other. That's not going to navigate the search space very fast on an atomic level. Lenat did not stand dutifully apart from his creation, but stepped in and helped EURISKO prune its own heuristics. But in the end EURISKO ran out of steam, and Lenat couldn't push it any further.

EURISKO lacked what I called "insight" - that is, the type of abstract knowledge that lets humans fly through the search space. And so its recursive access to its own heuristics proved to be for nought.

Unless, y'know, you're counting becoming world champion at Traveller without ever previously playing a human, as some sort of accomplishment.

But it is, thankfully, a little harder than that to destroy the world - as Lenat's experimental test informed us.

Robin previously asked why Douglas Engelbart did not take over the world, despite his vision of a team building tools to improve tools, and his anticipation of tools like computer mice and hypertext.

One reply would be, "Sure, a computer gives you a 10% advantage in doing various sorts of problems, some of which include computers - but there's still a lot of work that the computer doesn't help you with - and the mouse doesn't run off and write better mice entirely on its own - so k < 1, and it still takes large amounts of human labor to advance computer technology as a whole - plus a lot of the interesting knowledge is nonexcludable so it's hard to capture the value you create - and that's why Buffett could manifest a better take-over-the-world-with-sustained-higher-interest-rates than Engelbart."

But imagine that Engelbart had built a computer mouse, and discovered that each click of the mouse raised his IQ by one point. Then, perhaps, we would have had a situation on our hands.

Maybe you could diagram it something like this:

1. Metacognitive level: Evolution is the metacognitive algorithm which produced the wiring patterns and low-level developmental rules for human brains.

2. Cognitive level: The brain processes its knowledge (including procedural knowledge) using algorithms that quite mysterious to the user within them. Trying to program AIs with the sort of instructions humans give each other usually proves not to do anything: the machinery activated by the levers is missing.

3. Metaknowledge level: Knowledge and skills associated with e.g. "science" as an activity to carry out using your brain - instructing you when to try to think of new hypotheses using your mysterious creative abilities.

4. Knowledge level: Knowing how gravity works, or how much weight steel can support.

5. Object level: Specific actual problems, like building a bridge or something.

This is a causal tree, and changes at levels closer to root have greater impacts as the effects cascade downward.

So one way of looking at it is: "A computer mouse isn't recursive enough."

This is an issue that I need to address at further length, but for today I'm out of time.

Magic is the final factor I'd like to point out, at least for now, in considering sources of discontinuity for self-improving minds. By "magic" I naturally do not refer to this. Rather, "magic" in the sense that if you asked 19th-century Victorians what they thought the future would bring, they would have talked about flying machines or gigantic engines, and a very few true visionaries would have suggested space travel or Babbage computers. Nanotechnology, not so much.

The future has a reputation for accomplishing feats which the past thought impossible. Future civilizations have even broken what past civilizations thought (incorrectly, of course) to be the laws of physics. If prophets of 1900 AD - never mind 1000 AD - had tried to bound the powers of human civilization a billion years later, some of those impossibilities would have been accomplished before the century was out; transmuting lead into gold, for example. Because we remember future civilizations surprising past civilizations, it has become cliche that we can't put limits on our great-grandchildren.

And yet everyone in the 20th century, in the 19th century, and in the 11th century, was human. There is also the sort of magic that a human gun is to a wolf, or the sort of magic that human genetic engineering is to natural selection.

To "improve your own capabilities" is an instrumental goal, and if a smarter intelligence than my own is focused on that goal, I should expect to be surprised. The mind may find ways to produce larger jumps in capability than I can visualize myself. Where higher creativity than mine is at work and looking for shorter shortcuts, the discontinuities that I imagine may be dwarfed by the discontinuities that it can imagine.

And remember how little progress it takes - just a hundred years of human time, with everyone still human - to turn things that would once have been "unimaginable" into heated debates about feasibility. So if you build a mind smarter than you, and it thinks about how to go FOOM quickly, and it goes FOOM faster than you imagined possible, you really have no right to complain - based on the history of mere human history, you should have expected a significant probability of being surprised. Not, surprised that the nanotech is 50% faster than you thought it would be. Surprised the way the Victorians would have been surprised by nanotech.

Thus the last item on my (current, somewhat ad-hoc) list of reasons to expect discontinuity: Cascades, cycles, insight, recursion, magic.

Referenced by: Engelbart: Insufficiently Recursive • Total Nano Domination • Recursive Self-Improvement • Sustained Strong Recursion • What I Think, If Not Why

Original with comments: ...Recursion, Magic

The Complete Idiot's Guide to Ad Hominem

Eliezer Yudkowsky, 25 November 2008 09:47PM

Stephen Bond writes the definitive word on ad hominem in "the ad hominem fallacy fallacy":

In reality, ad hominem is unrelated to sarcasm or personal abuse. Argumentum ad hominem is the logical fallacy of attempting to undermine a speaker's argument by attacking the speaker instead of addressing the argument. The mere presence of a personal attack does not indicate ad hominem: the attack must be used for the purpose of undermining the argument, or otherwise the logical fallacy isn't there.

[...]

A: "All rodents are mammals, but a weasel isn't a rodent, so it can't be a mammal."
B: "You evidently know nothing about logic. This does not logically follow."

B's argument is still not ad hominem. B does not imply that A's sentence does not logically follow because A knows nothing about logic. B is still addressing the substance of A's argument...

This is too beautiful, thorough, and precise to not post. HT to sfk on HN.

Original with comments: The Complete Idiot's Guide to Ad Hominem

Engelbart: Insufficiently Recursive

Eliezer Yudkowsky, 26 November 2008 08:31AM

Followup to: Cascades, Cycles, Insight, Recursion, Magic
Reply to: Engelbart As Ubertool?

When Robin originally suggested that Douglas Engelbart, best known as the inventor of the computer mouse, would have been a good candidate for taking over the world via compound interest on tools that make tools, my initial reaction was "What on Earth? With a mouse?"

On reading the initial portions of Engelbart's "Augmenting Human Intellect: A Conceptual Framework", it became a lot clearer where Robin was coming from.

Sometimes it's hard to see through the eyes of the past. Engelbart was a computer pioneer, and in the days when all these things were just getting started, he had a vision of using computers to systematically augment human intelligence. That was what he thought computers were for. That was the ideology lurking behind the mouse. Something that makes its users smarter - now that sounds a bit more plausible as an UberTool.

Looking back at Engelbart's plans with benefit of hindsight, I see two major factors that stand out:

1. Engelbart committed the Classic Mistake of AI: underestimating how much cognitive work gets done by hidden algorithms running beneath the surface of introspection, and overestimating what you can do by fiddling with the visible control levers.

2. Engelbart anchored on the way that someone as intelligent as Engelbart would use computers, but there was only one of him - and due to point 1 above, he couldn't use computers to make other people as smart as him.

To start with point 2: They had more reverence for computers back in the old days. Engelbart visualized a system carefully designed to flow with every step of a human's work and thought, assisting every iota it could manage along the way. And the human would be trained to work with the computer, the two together dancing a seamless dance.

And the problem with this, was not just that computers got cheaper and that programmers wrote their software more hurriedly.

There's a now-legendary story about the Windows Vista shutdown menu, a simple little feature into which 43 different Microsoft people had input. The debate carried on for over a year. The final product ended up as the lowest common denominator - a couple of hundred lines of code and a very visually unimpressive menu.

So even when lots of people spent a tremendous amount of time thinking about a single feature of the system - it still didn't end up very impressive. Jef Raskin could have done better than that, I bet. But Raskins and Engelbarts are rare.

You see the same effect in Eric Drexler's chapter on hypertext in Engines of Creation: Drexler imagines the power of the Web to... use two-way links and user annotations to promote informed criticism. (As opposed to the way we actually use it.) And if the average Web user were Eric Drexler, the Web probably would work that way by now.

But no piece of software that has yet been developed, by mouse or by Web, can turn an average human user into Engelbart or Raskin or Drexler. You would very probably have to reach into the brain and rewire neural circuitry directly; I don't think any sense input or motor interaction would accomplish such a thing.

Which brings us to point 1.

It does look like Engelbart was under the spell of the "logical" paradigm that prevailed in AI at the time he made his plans. (Should he even lose points for that? He went with the mainstream of that science.) He did not see it as an impossible problem to have computers help humans think - he seems to have underestimated the difficulty in much the same way that the field of AI once severely underestimated the work it would take to make computers themselves solve cerebral-seeming problems. (Though I am saying this, reading heavily between the lines of one single paper that he wrote.) He talked about how the core of thought is symbols, and speculated on how computers could help people manipulate those symbols.

I have already said much on why people tend to underestimate the amount of serious heavy lifting that gets done by cognitive algorithms hidden inside black boxes that run out of your introspective vision, and overestimating what you can do by duplicating the easily visible introspective control levers. The word "apple", for example, is a visible lever; you can say it or not say it, its presence or absence is salient. The algorithms of a visual cortex that let you visualize what an apple would look like upside-down - we all have these in common, and they are not introspectively accessible. Human beings knew about apples a long, long time before they knew there was even such a thing as the visual cortex, let alone beginning to unravel the algorithms by which it operated.

Robin Hanson asked me:

"You really think an office worker with modern computer tools is only 10% more productive than one with 1950-era non-computer tools? Even at the task of creating better computer tools?"

But remember the parable of the optimizing compiler run on its own source code - maybe it makes itself 50% faster, but only once; the changes don't increase its ability to make future changes. So indeed, we should not be too impressed by a 50% increase in office worker productivity - not for purposes of asking about FOOMs. We should ask whether that increase in productivity translates into tools that create further increases in productivity.

And this is where the problem of underestimating hidden labor, starts to bite. Engelbart rhapsodizes (accurately!) on the wonders of being able to cut and paste text while writing, and how superior this should be compared to the typewriter. But suppose that Engelbart overestimates, by a factor of 10, how much of the intellectual labor of writing goes into fighting the typewriter. Then because Engelbart can only help you cut and paste more easily, and cannot rewrite those hidden portions of your brain that labor to come up with good sentences and good arguments, the actual improvement he delivers is a tenth of what he thought it would be. An anticipated 20% improvement becomes an actual 2% improvement. k way less than 1.

This will hit particularly hard if you think that computers, with some hard work on the user interface, and some careful training of the humans, ought to be able to help humans with the type of "creative insight" or "scientific labor" that goes into inventing new things to do with the computer. If you thought that the surface symbols were where most of the intelligence resided, you would anticipate that computer improvements would hit back hard to this meta-level, and create people who were more scientifically creative and who could design even better computer systems.

But if really you can only help people type up their ideas, while all the hard creative labor happens in the shower thanks to very-poorly-understood cortical algorithms - then you are much less like neutrons cascading through uranium, and much more like an optimizing compiler that gets a single speed boost and no more. It looks like the person is 20% more productive, but in the aspect of intelligence that potentially cascades to further improvements they're only 2% more productive, if that.

(Incidentally... I once met a science-fiction author of a previous generation, and mentioned to him that the part of my writing I most struggled with, was my tendency to revise and revise and revise things I had already written, instead of writing new things. And he said, "Yes, that's why I went back to the typewriter. The word processor made it too easy to revise things; I would do too much polishing, and writing stopped being fun for me." It made me wonder if there'd be demand for an author's word processor that wouldn't let you revise anything until you finished your first draft.

But this could be chalked up to the humans not being trained as carefully, nor the software designed as carefully, as in the process Engelbart envisioned.)

Engelbart wasn't trying to take over the world in person, or with a small group. Yet had he tried to go the UberTool route, we can reasonably expect he would have failed - that is, failed at advancing far beyond the outside world in internal computer technology, while selling only UberTool's services to outsiders.

Why? Because it takes too much human labor to develop computer software and computer hardware, and this labor cannot be automated away as a one-time cost. If the world outside your window has a thousand times as many brains, a 50% productivity boost that only cascades to a 10% and then a 1% additional productivity boost, will not let you win against the world. If your UberTool was itself a mind, if cascades of self-improvement could fully automate away more and more of the intellectual labor performed by the outside world - then it would be a different story. But while the development path wends inexorably through thousands and millions of engineers, and you can't divert that path through an internal computer, you're not likely to pull far ahead of the world. You can just choose between giving your own people a 10% boost, or selling your product on the market to give lots of people a 10% boost.

You can have trade secrets, and sell only your services or products - many companies follow that business plan; any company that doesn't sell its source code does so. But this is just keeping one small advantage to yourself, and adding that as a cherry on top of the technological progress handed you by the outside world. It's not having more technological progress inside than outside.

If you're getting most of your technological progress handed to you - your resources not being sufficient to do it in-house - then you won't be able to apply your private productivity improvements to most of your actual velocity, since most of your actual velocity will come from outside your walls. If you only create 1% of the progress that you use, then a 50% improvement becomes a 0.5% improvement. The domain of potential recursion and potential cascades is much smaller, diminishing k. As if only 1% of the uranium generating your neutrons, were available for chain reactions to be fissioned further.

We don't live in a world that cares intensely about milking every increment of velocity out of scientific progress. A 0.5% improvement is easily lost in the noise. Corporations and universities routinely put obstacles in front of their internal scientists that cost them more than 10% of their potential. This is one of those problems where not everyone is Engelbart (and you can't just rewrite their source code either).

For completeness, I should mention that there are generic obstacles to pulling an UberTool. Warren Buffett has gotten a sustained higher interest rate than the economy at large, and is widely believed to be capable of doing so indefinitely. In principle, the economy could have invested hundreds of billions of dollars as soon as Berkshire Hathaway had a sufficiently long track record to rule out chance. Instead, Berkshire has grown mostly by compound interest. We could live in a world where asset allocations were ordinarily given as a mix of stocks, bonds, real estate, and Berkshire Hathaway. We don't live in that world for a number of reasons: financial advisors not wanting to make themselves appear irrelevant, strange personal preferences on the part of Buffett...

The economy doesn't always do the obvious thing, like flow money into Buffett until his returns approach the average return of the economy. Interest rate differences much higher than 0.5%, on matters that people care about far more intensely than Science, are ignored if they're not presented in exactly the right format to be seized.

And it's not easy for individual scientists or groups to capture the value created by scientific progress. Did Einstein die with 0.1% of the value that he created? Engelbart in particular doesn't seem to have tried to be Bill Gates, at least not as far as I know.

With that in mind - in one sense Engelbart succeeded at a good portion of what he actually set out to do: computer mice did take over the world.

But it was a broad slow cascade that mixed into the usual exponent of economic growth. Not a concentrated fast FOOM. To produce a concentrated FOOM, you've got to be able to swallow as much as possible of the processes driving the FOOM into the FOOM. Otherwise you can't improve those processes and you can't cascade through them and your k goes down. Then your interest rates won't even be as much higher than normal as, say, Warren Buffett's. And there's no grail to be won, only profits to be made: If you have no realistic hope of beating the world, you may as well join it.

Referenced by: Total Nano Domination • Recursive Self-Improvement

Original with comments: Engelbart: Insufficiently Recursive

Total Nano Domination

Eliezer Yudkowsky, 27 November 2008 09:54AM

Followup to: Engelbart: Insufficiently Recursive

The computer revolution had cascades and insights aplenty. Computer tools are routinely used to create tools, from using a C compiler to write a Python interpreter, to using theorem-proving software to help design computer chips. I would not yet rate computers as being very deeply recursive - I don't think they've improved our own thinking processes even so much as the Scientific Revolution - yet. But some of the ways that computers are used to improve computers, verge on being repeatable (cyclic).

Yet no individual, no localized group, nor even country, managed to get a sustained advantage in computing power, compound the interest on cascades, and take over the world. There was never a Manhattan moment when a computing advantage temporarily gave one country a supreme military advantage, like the US and its atomic bombs for that brief instant at the end of WW2. In computing there was no equivalent of "We've just crossed the sharp threshold of criticality, and now our pile doubles its neutron output every two minutes, so we can produce lots of plutonium and you can't."

Will the development of nanotechnology go the same way as computers - a smooth, steady developmental curve spread across many countries, no one project taking into itself a substantial fraction of the world's whole progress? Will it be more like the Manhattan Project, one country gaining a (temporary?) huge advantage at huge cost? Or could a small group with an initial advantage cascade and outrun the world?

Just to make it clear why we might worry about this for nanotech, rather than say car manufacturing - if you can build things from atoms, then the environment contains an unlimited supply of perfectly machined spare parts. If your molecular factory can build solar cells, it can acquire energy as well.

So full-fledged Drexlerian molecular nanotechnology can plausibly automate away much of the manufacturing in its material supply chain. If you already have nanotech, you may not need to consult the outside economy for inputs of energy or raw material.

This makes it more plausible that a nanotech group could localize off, and do its own compound interest, away from the global economy. If you're Douglas Engelbart building better software, you still need to consult Intel for the hardware that runs your software, and the electric company for the electricity that powers your hardware. It would be a considerable expense to build your own fab lab for your chips (that makes chips as good as Intel) and your own power station for electricity (that supplies electricity as cheaply as the utility company).

It's not just that this tends to entangle you with the fortunes of your trade partners, but also that - as an UberTool Corp keeping your trade secrets in-house - you can't improve the hardware you get, or drive down the cost of electricity, as long as these things are done outside. Your cascades can only go through what you do locally, so the more you do locally, the more likely you are to get a compound interest advantage. (Mind you, I don't think Engelbart could have gone FOOM even if he'd made his chips locally and supplied himself with electrical power - I just don't think the compound advantage on using computers to make computers is powerful enough to sustain k > 1.)

In general, the more capabilities are localized into one place, the less people will depend on their trade partners, the more they can cascade locally (apply their improvements to yield further improvements), and the more a "critical cascade" / FOOM sounds plausible.

Yet self-replicating nanotech is a very advanced capability. You don't get it right off the bat. Sure, lots of biological stuff has this capability, but this is a misleading coincidence - it's not that self-replication is easy, but that evolution, for its own alien reasons, tends to build it into everything. (Even individual cells, which is ridiculous.)

In the run-up to nanotechnology, it seems not implausible to suppose a continuation of the modern world. Today, many different labs work on small pieces of nanotechnology - fortunes entangled with their trade partners, and much of their research velocity coming from advances in other laboratories. Current nanotech labs are dependent on the outside world for computers, equipment, science, electricity, and food; any single lab works on a small fraction of the puzzle, and contributes small fractions of the progress.

In short, so far nanotech is going just the same way as computing.

But it is a tad premature - I would even say that it crosses the line into the "silly" species of futurism - to exhale a sigh of relief and say, "Ah, that settles it - no need to consider any further."

We all know how exponential multiplication works: 1 microscopic nanofactory, 2 microscopic nanofactories, 4 microscopic nanofactories... let's say there's 100 different groups working on self-replicating nanotechnology and one of those groups succeeds one week earlier than the others. Rob Freitas has calculated that some species of replibots could spread through the Earth in 2 days (even given what seem to me like highly conservative assumptions in a context where conservatism is not appropriate).

So, even if the race seems very tight, whichever group gets replibots first can take over the world given a mere week's lead time -

Yet wait! Just having replibots doesn't let you take over the world. You need fusion weapons, or surveillance bacteria, or some other way to actually govern. That's a lot of matterware - a lot of design and engineering work. A replibot advantage doesn't equate to a weapons advantage, unless, somehow, the planetary economy has already published the open-source details of fully debugged weapons that you can build with your newfound private replibots. Otherwise, a lead time of one week might not be anywhere near enough.

Even more importantly - "self-replication" is not a binary, 0-or-1 attribute. Things can be partially self-replicating. You can have something that manufactures 25% of itself, 50% of itself, 90% of itself, or 99% of itself - but still needs one last expensive computer chip to complete the set. So if you have twenty-five countries racing, sharing some of their results and withholding others, there isn't one morning where you wake up and find that one country has self-replication.

Bots become successively easier to manufacture; the factories get successively cheaper. By the time one country has bots that manufacture themselves from environmental materials, many other countries have bots that manufacture themselves from feedstock. By the time one country has bots that manufacture themselves entirely from feedstock, other countries have produced some bots using assembly lines. The nations also have all their old conventional arsenal, such as intercontinental missiles tipped with thermonuclear weapons, and these have deterrent effects against crude nanotechnology. No one ever gets a discontinuous military advantage, and the world is safe. (?)

At this point, I do feel obliged to recall the notion of "burdensome details", that we're spinning a story out of many conjunctive details, any one of which could go wrong. This is not an argument in favor of anything in particular, just a reminder not to be seduced by stories that are too specific. When I contemplate the sheer raw power of nanotechnology, I don't feel confident that the fabric of society can even survive the sufficiently plausible prospect of its near-term arrival. If your intelligence estimate says that Russia (the new belligerent Russia under Putin) is going to get self-replicating nanotechnology in a year, what does that do to Mutual Assured Destruction? What if Russia makes a similar intelligence assessment of the US? What happens to the capital markets? I can't even foresee how our world will react to the prospect of various nanotechnological capabilities as they promise to be developed in the future's near future. Let alone envision how society would actually change as full-fledged molecular nanotechnology was developed, even if it were developed gradually...

...but I suppose the Victorians might say the same thing about nuclear weapons or computers, and yet we still have a global economy - one that's actually lot more interdependent than theirs, thanks to nuclear weapons making small wars less attractive, and computers helping to coordinate trade.

I'm willing to believe in the possibility of a smooth, gradual ascent to nanotechnology, so that no one state - let alone any corporation or small group - ever gets a discontinuous advantage.

The main reason I'm willing to believe this is because of the difficulties of design and engineering, even after all manufacturing is solved. When I read Drexler's Nanosystems, I thought: "Drexler uses properly conservative assumptions everywhere I can see, except in one place - debugging. He assumes that any failed component fails visibly, immediately, and without side effects; this is not conservative."

In principle, we have complete control of our computers - every bit and byte is under human command - and yet it still takes an immense amount of engineering work on top of that to make the bits do what we want. This, and not any difficulties of manufacturing things once they are designed, is what takes an international supply chain of millions of programmers.

But we're still not out of the woods.

Suppose that, by a providentially incremental and distributed process, we arrive at a world of full-scale molecular nanotechnology - a world where designs, if not finished material goods, are traded among parties. In a global economy large enough that no one actor, or even any one state, is doing more than a fraction of the total engineering.

It would be a very different world, I expect; and it's possible that my essay may have already degenerated into nonsense. But even if we still have a global economy after getting this far - then we're still not out of the woods.

Remember those ems? The emulated humans-on-a-chip? The uploads?

Suppose that, with molecular nanotechnology already in place, there's an international race for reliable uploading - with some results shared, and some results private - with many state and some nonstate actors.

And suppose the race is so tight, that the first state to develop working researchers-on-a-chip, only has a one-day lead time over the other actors.

That is - one day before anyone else, they develop uploads sufficiently undamaged, or capable of sufficient recovery, that the ems can carry out research and development. In the domain of, say, uploading.

There are other teams working on the problem, but their uploads are still a little off, suffering seizures and having memory faults and generally having their cognition degraded to the point of not being able to contribute. (NOTE: I think this whole future is a wrong turn and we should stay away from it; I am not endorsing this.)

But this one team, though - their uploads still have a few problems, but they're at least sane enough and smart enough to start... fixing their problems themselves?

If there's already full-scale nanotechnology around when this happens, then even with some inefficiency built in, the first uploads may be running at ten thousand times human speed. Nanocomputers are powerful stuff.

And in an hour, or around a year of internal time, the ems may be able to upgrade themselves to a hundred thousand times human speed, and fix some of the remaining problems.

And in another hour, or ten years of internal time, the ems may be able to get the factor up to a million times human speed, and start working on intelligence enhancement...

One could, of course, voluntarily publish the improved-upload protocols to the world, and give everyone else a chance to join in. But you'd have to trust that not a single one of your partners were holding back a trick that lets them run uploads at ten times your own maximum speed (once the bugs were out of the process). That kind of advantage could snowball quite a lot, in the first sidereal day.

Now, if uploads are gradually developed at a time when computers are too slow to run them quickly - meaning, before molecular nanotech and nanofactories come along - then this whole scenario is averted; the first high-fidelity uploads, running at a hundredth of human speed, will grant no special advantage. (Assuming that no one is pulling any spectacular snowballing tricks with intelligence enhancement - but they would have to snowball fast and hard, to confer advantage on a small group running at low speeds. The same could be said of brain-computer interfaces, developed before or after nanotechnology, if running in a small group at merely human speeds. I would credit their world takeover, but I suspect Robin Hanson wouldn't at this point.)

Now, I don't really believe in any of this - this whole scenario, this whole world I'm depicting. In real life, I'd expect someone to brute-force an unFriendly AI on one of those super-ultimate-nanocomputers, followed in short order by the end of the world. But that's a separate issue. And this whole world seems too much like our own, after too much technological change, to be realistic to me. World government with an insuperable advantage? Ubiquitous surveillance? I don't like the ideas, but both of them would change the game dramatically...

But the real point of this essay is to illustrate a point more important than nanotechnology: as optimizers become more self-swallowing, races between them are more unstable.

If you sent a modern computer back in time to 1950 - containing many modern software tools in compiled form, but no future history or declaratively stored future science - I would guess that the recipient could not use it to take over the world. Even if the USSR got it. Our computing industry is a very powerful thing, but it relies on a supply chain of chip factories.

If someone got a future nanofactory with a library of future nanotech applications - including designs for things like fusion power generators and surveillance bacteria - they might really be able to take over the world. The nanofactory swallows its own supply chain; it incorporates replication within itself. If the owner fails, it won't be for lack of factories. It will be for lack of ability to develop new matterware fast enough, and apply existing matterware fast enough, to take over the world.

I'm not saying that nanotech will appear from nowhere with a library of designs - just making a point about concentrated power and the instability it implies.

Think of all the tech news that you hear about once - say, an article on Slashdot about yada yada 50% improved battery technology - and then you never hear about again, because it was too expensive or too difficult to manufacture.

Now imagine a world where the news of a 50% improved battery technology comes down the wire, and the head of some country's defense agency is sitting down across from engineers and intelligence officers and saying, "We have five minutes before all of our rival's weapons are adapted to incorporate this new technology; how does that affect our balance of power?" Imagine that happening as often as "amazing breakthrough" articles appear on Slashdot.

I don't mean to doomsay - the Victorians would probably be pretty surprised we haven't blown up the world with our ten-minute ICBMs, but we don't live in their world - well, maybe doomsay just a little - but the point is: It's less stable. Improvements cascade faster once you've swallowed your manufacturing supply chain.

And if you sent back in time a single nanofactory, and a single upload living inside it - then the world might end in five minutes or so, as we bios measure time.

The point being, not that an upload will suddenly appear, but that now you've swallowed your supply chain and your R&D chain.

And so this world is correspondingly more unstable, even if all the actors start out in roughly the same place. Suppose a state manages to get one of those Slashdot-like technology improvements - only this one lets uploads think 50% faster - and they get it fifty minutes before anyone else, at a point where uploads are running ten thousand times as fast as human (50 mins = ~1 year) - and in that extra half-year, the uploads manage to find another couple of 50% improvements...

Now, you can suppose that all the actors are all trading all of their advantages and holding nothing back, so everyone stays nicely synchronized.

Or you can suppose that enough trading is going on, that most of the research any group benefits from comes from outside that group, and so a 50% advantage for a local group doesn't cascade much.

But again, that's not the point. The point is that in modern times, with the modern computing industry, where commercializing an advance requires building a new computer factory, a bright idea that has gotten as far as showing a 50% improvement in the laboratory, is merely one more article on Slashdot.

If everything could instantly be rebuilt via nanotech, that laboratory demonstration could precipitate an instant international military crisis.

And if there are uploads around, so that a cute little 50% advancement in a certain kind of hardware, recurses back to imply 50% greater speed at all future research - then this Slashdot article could become the key to world domination.

As systems get more self-swallowing, they cascade harder; and even if all actors start out equivalent, races between them get much more unstable. I'm not claiming it's impossible for that world to be stable. The Victorians might have thought that about ICBMs. But that subjunctive world contains additional instability compared to our own, and would need additional centripetal forces to end up as stable as our own.

I expect Robin to disagree with some part of this essay, but I'm not sure which part or how.

Referenced by: Recursive Self-Improvement

Original with comments: Total Nano Domination

Thanksgiving Prayer

Eliezer Yudkowsky, 28 November 2008 04:17AM

At tonight's Thanksgiving, Erin remarked on how this was her first real Thanksgiving dinner away from her family, and that it was an odd feeling to just sit down and eat without any prayer beforehand. (Yes, she's a solid atheist in no danger whatsoever, thank you for asking.)

And as she said this, it reminded me of how wrong it is to give gratitude to God for blessings that actually come from our fellow human beings putting in a great deal of work.

So I at once put my hands together and said,

"Dear Global Economy, we thank thee for thy economies of scale, thy professional specialization, and thy international networks of trade under Ricardo's Law of Comparative Advantage, without which we would all starve to death while trying to assemble the ingredients for such a dinner as this. Amen."

Original with comments: Thanksgiving Prayer

Chaotic Inversion

Eliezer Yudkowsky, 29 November 2008 10:57AM

I was recently having a conversation with some friends on the topic of hour-by-hour productivity and willpower maintenance - something I've struggled with my whole life.

I can avoid running away from a hard problem the first time I see it (perseverance on a timescale of seconds), and I can stick to the same problem for years; but to keep working on a timescale of hours is a constant battle for me. It goes without saying that I've already read reams and reams of advice; and the most help I got from it was realizing that a sizable fraction other creative professionals had the same problem, and couldn't beat it either, no matter how reasonable all the advice sounds.

"What do you do when you can't work?" my friends asked me. (Conversation probably not accurate, this is a very loose gist.)

And I replied that I usually browse random websites, or watch a short video.

"Well," they said, "if you know you can't work for a while, you should watch a movie or something."

"Unfortunately," I replied, "I have to do something whose time comes in short units, like browsing the Web or watching short videos, because I might become able to work again at any time, and I can't predict when -"

And then I stopped, because I'd just had a revelation.

I'd always thought of my workcycle as something chaotic, something unpredictable. I never used those words, but that was the way I treated it.

But here my friends seemed to be implying - what a strange thought - that other people could predict when they would become able to work again, and structure their time accordingly.

And it occurred to me for the first time that I might have been committing that damned old chestnut the Mind Projection Fallacy, right out there in my ordinary everyday life instead of high abstraction.

Maybe it wasn't that my productivity was unusually chaotic; maybe I was just unusually stupid with respect to predicting it.

That's what inverted stupidity looks like - chaos. Something hard to handle, hard to grasp, hard to guess, something you can't do anything with. It's not just an idiom for high abstract things like Artificial Intelligence. It can apply in ordinary life too.

And the reason we don't think of the alternative explanation "I'm stupid", is not - I suspect - that we think so highly of ourselves. It's just that we don't think of ourselves at all. We just see a chaotic feature of the environment.

So now it's occurred to me that my productivity problem may not be chaos, but my own stupidity.

And that may or may not help anything. It certainly doesn't fix the problem right away. Saying "I'm ignorant" doesn't make you knowledgeable.

But it is, at least, a different path than saying "it's too chaotic".

Sequence: Mysterious Answers to Mysterious Questions

Applause Lights

Referenced by: Artificial Mysterious Intelligence • For The People Who Are Still Alive • Purchase Fuzzies and Utilons Separately • Of Gender and Rationality

Original with comments: Chaotic Inversion

Singletons Rule OK

Eliezer Yudkowsky, 30 November 2008 04:45PM

Reply to: Total Tech Wars

How does one end up with a persistent disagreement between two rationalist-wannabes who are both aware of Aumann's Agreement Theorem and its implications?

Such a case is likely to turn around two axes: object-level incredulity ("no matter what AAT says, proposition X can't really be true") and meta-level distrust ("they're trying to be rational despite their emotional commitment, but are they really capable of that?").

So far, Robin and I have focused on the object level in trying to hash out our disagreement. Technically, I can't speak for Robin; but at least in my own case, I've acted thus because I anticipate that a meta-level argument about trustworthiness wouldn't lead anywhere interesting. Behind the scenes, I'm doing what I can to make sure my brain is actually capable of updating, and presumably Robin is doing the same.

(The linchpin of my own current effort in this area is to tell myself that I ought to be learning something while having this conversation, and that I shouldn't miss any scrap of original thought in it - the Incremental Update technique. Because I can genuinely believe that a conversation like this should produce new thoughts, I can turn that feeling into genuine attentiveness.)

Yesterday, Robin inveighed hard against what he called "total tech wars", and what I call "winner-take-all" scenarios:

Robin: "If you believe the other side is totally committed to total victory, that surrender is unacceptable, and that all interactions are zero-sum, you may conclude your side must never cooperate with them, nor tolerate much internal dissent or luxury."

Robin and I both have emotional commitments and we both acknowledge the danger of that. There's nothing irrational about feeling, per se; only failure to update is blameworthy. But Robin seems to be very strongly against winner-take-all technological scenarios, and I don't understand why.

Among other things, I would like to ask if Robin has a Line of Retreat set up here - if, regardless of how he estimates the probabilities, he can visualize what he would do if a winner-take-all scenario were true.

Yesterday Robin wrote:

"Eliezer, if everything is at stake then 'winner take all' is 'total war'; it doesn't really matter if they shoot you or just starve you to death."

We both have our emotional commitments, but I don't quite understand this reaction.

First, to me it's obvious that a "winner-take-all" technology should be defined as one in which, ceteris paribus, a local entity tends to end up with the option of becoming one kind of Bostromian singleton - the decisionmaker of a global order in which there is a single decision-making entity at the highest level. (A superintelligence with unshared nanotech would count as a singleton; a federated world government with its own military would be a different kind of singleton; or you can imagine something like a galactic operating system with a root account controllable by 80% majority vote of the populace, etcetera.)

The winner-take-all option is created by properties of the technology landscape, which is not a moral stance. Nothing is said about an agent with that option, actually becoming a singleton. Nor about using that power to shoot people, or reuse their atoms for something else, or grab all resources and let them starve (though "all resources" should include their atoms anyway).

Nothing is yet said about various patches that could try to avert a technological scenario that contains upward cliffs of progress - e.g. binding agreements enforced by source code examination or continuous monitoring, in advance of the event. (Or if you think that rational agents cooperate on the Prisoner's Dilemma, so much work might not be required to coordinate.)

Superintelligent agents not in a humanish moral reference frame - AIs that are just maximizing paperclips or sorting pebbles - who happen on the option of becoming a Bostromian Singleton, and who have not previously executed any somehow-binding treaty; will ceteris paribus choose to grab all resources in service of their utility function, including the atoms now composing humanity. I don't see how you could reasonably deny this! It's a straightforward decision-theoretic choice between payoff 10 and payoff 1000!

But conversely, there are possible agents in mind design space who, given the option of becoming a singleton, will not kill you, starve you, reprogram you, tell you how to live your life, or even meddle in your destiny unseen. See Bostrom's (short) paper on the possibility of good and bad singletons of various types.

If Robin thinks it's impossible to have a Friendly AI or maybe even any sort of benevolent superintelligence at all, even the descendants of human uploads - if Robin is assuming that superintelligent agents will act according to roughly selfish motives, and that only economies of trade are necessary and sufficient to prevent holocaust - then Robin may have no Line of Retreat open, as I try to argue that AI has an upward cliff built in.

And in this case, it might be time well spent, to first address the question of whether Friendly AI is a reasonable thing to try to accomplish, so as to create that line of retreat. Robin and I are both trying hard to be rational despite emotional commitments; but there's no particular reason to needlessly place oneself in the position of trying to persuade, or trying to accept, that everything of value in the universe is certainly doomed.

For me, it's particularly hard to understand Robin's position in this, because for me the non-singleton future is the one that is obviously abhorrent.

If you have lots of entities with root permissions on matter, any of whom has the physical capability to attack any other, then you have entities spending huge amounts of precious negentropy on defense and deterrence. If there's no centralized system of property rights in place for selling off the universe to the highest bidder, then you have a race to burn the cosmic commons, and the degeneration of the vast majority of all agents into rapacious hardscrapple frontier replicators.

To me this is a vision of futility - one in which a future light cone that could have been full of happy, safe agents having complex fun, is mostly wasted by agents trying to seize resources and defend them so they can send out seeds to seize more resources.

And it should also be mentioned that any future in which slavery or child abuse is successfully prohibited, is a world that has some way of preventing agents from doing certain things with their computing power. There are vastly worse possibilities than slavery or child abuse opened up by future technologies, which I flinch from referring to even as much as I did in the previous sentence. There are things I don't want to happen to anyone - including a population of a septillion captive minds running on a star-powered Matrioshka Brain that is owned, and defended against all rescuers, by the mind-descendant of Lawrence Bittaker (serial killer, aka "Pliers"). I want to win against the horrors that exist in this world and the horrors that could exist in tomorrow's world - to have them never happen ever again, or, for the really awful stuff, never happen in the first place. And that victory requires the Future to have certain global properties.

But there are other ways to get singletons besides falling up a technological cliff. So that would be my Line of Retreat: If minds can't self-improve quickly enough to take over, then try for the path of uploads setting up a centralized Constitutional operating system with a root account controlled by majority vote, or something like that, to prevent their descendants from having to burn the cosmic commons.

So for me, any satisfactory outcome seems to necessarily involve, if not a singleton, the existence of certain stable global properties upon the future - sufficient to prevent burning the cosmic commons, prevent life's degeneration into rapacious hardscrapple frontier replication, and prevent supersadists torturing septillions of helpless dolls in private, obscure star systems.

Robin has written about burning the cosmic commons and rapacious hardscrapple frontier existences. This doesn't imply that Robin approves of these outcomes. But Robin's strong rejection even of winner-take-all language and concepts, seems to suggest that our emotional commitments are something like 180 degrees opposed. Robin seems to feel the same way about singletons as I feel about ÂŹsingletons.

But why? I don't think our real values are that strongly opposed - though we may have verbally-described and attention-prioritized those values in different ways.

Referenced by: Devil's Offers

Original with comments: Singletons Rule OK

Disappointment in the Future

Eliezer Yudkowsky, 01 December 2008 04:45AM

This seems worth posting around now... As I've previously observed, futuristic visions are produced as entertainment, sold today and consumed today. A TV station interviewing an economic or diplomatic pundit doesn't bother to show what that pundit predicted three years ago and how the predictions turned out. Why would they? Futurism Isn't About Prediction.

But someone on the ImmInst forum actually went and compiled a list of Ray Kurzweil's predictions in 1999 for the years 2000-2009. We're not out of 2009 yet, but right now it's not looking good...

• Individuals primarily use portable computers
• Portable computers have dramatically become lighter and thinner
• Personal computers are available in a wide range of sizes and shapes, and are commonly embedded in clothing and jewelry, like wrist watches, rings, earrings and other body ornaments
• Computers with a high-resolution visual interface range from rings and pins and credit cards up to the size of a thin book. People typically have at least a dozen computers on and around their bodies, which are networked, using body LANS (local area networks)
• These computers monitor body functions, provide automated identity to conduct financial transactions and allow entry into secure areas. They also provide directions for navigation, and a variety of other services.
• Most portable computers do not have keyboards

• Rotating memories such as Hard Drives, CD roms, and DVDs are on their way out.
• Most users have servers on their homes and offices where they keep large stores of digital objects, including, among other things, virtual reality environments, although these are still on an early stage
• Cables are disappearing,
• The majority of texts is created using continuous speech recognition, or CSR (dictation software). CSRs are very accurate, far more than the human transcriptionists, who were used up until a few years ago.
• Books, magazines, and newspapers are now routinely read on displays that are the size of small books
• Computer displays built into eyeglasses are also used. These specialized glasses allow the users to see the normal environment while creating a virtual image that appears to hover in front of the viewer.
• Computers routinely include moving picture image cameras and are able to reliably identify their owners from their faces
• Three dimensional chips are commonly used
• Students from all ages have a portable computer, very thin and soft, weighting less than 1 pound. They interact with their computers primarily by voice and by pointing with a device that looks like a pencil. Keybords still exist but most textual language is created by speaking.
• Intelligent courseware has emerged as a common means of learning, recent controversial studies have shown that students can learn basic skills such as reading and math just as readily with interactive learning software as with human teachers.
• Schools are increasingly relying on software approaches. Many children learn to read on their own using personal computers before entering grade school.
• Persons with disabilities are rapidly overcoming their handicaps through intelligent technology
• Students with reading disabilities routinely use print to speech reading systems
• Print to speech reading machines for the blind are now very small, inexpensive, palm-size devices that can read books.
• Useful navigation systems have finally been developed to assist blind people in moving and avoiding obstacles. Those systems use GPS technology. The blind person communicates with his navigation system by voice.
• Deaf persons commonly use portable speech-to-text listening machines which display a real time transcription of what people are saying. The deaf user has the choice of either reading the transcribed speech as displayed text or watching an animated person gesturing in sign language.
• Listening machines cal also translate what is being said into another language in real-time, so they are commonly used by hearing people as well.
• There is a growing perception that the primary disabilities of blindness, deafness, and physical impairment do not necessarily. Disabled persons routinely describe their disabilities as mere inconveniences.
• In communications, translate telephone technology is commonly used. This allow you to speak in English, while your Japanese friend hears you in Japanese, and vice-versa.
• Telephones are primarily wireless and include high resolution moving images.
• Heptic technologies are emerging. They allow people to touch and feel objects and other persons at a distance. These force-feedback devices are wildly used in games and in training simulation systems. Interactive games routinely include all encompassing all visual and auditory environments.
• The 1999 chat rooms have been replaced with virtual environments.
• At least half of all transactions are conducted online
• Intelligent routes are in use, primarily for long distance travel. Once your car’s computer’s guiding system locks on to the control sensors on one of these highways, you can sit back, and relax.
• There is a growing neo-luditte movement.

Now, just to be clear, I don't want you to look at all that and think, "Gee, the future goes more slowly than expected - technological progress must be naturally slow."

More like, "Where are you pulling all these burdensome details from, anyway?"

If you looked at all that and said, "Ha ha, how wrong; now I have my own amazing prediction for what the future will be like, it won't be like that," then you're really missing the whole "you have to work a whole lot harder to produce veridical beliefs about the future, and often the info you want is simply not obtainable" business.

Original with comments: Disappointment in the Future

Recursive Self-Improvement

Eliezer Yudkowsky, 01 December 2008 08:49PM

Followup to: Life's Story Continues, Surprised by Brains, Cascades, Cycles, Insight, Recursion, Magic, Engelbart: Insufficiently Recursive, Total Nano Domination

I think that at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability - "AI go FOOM". Just to be clear on the claim, "fast" means on a timescale of weeks or hours rather than years or decades; and "FOOM" means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology (that it gets by e.g. ordering custom proteins over the Internet with 72-hour turnaround time). Not, "ooh, it's a little Einstein but it doesn't have any robot hands, how cute".

Most people who object to this scenario, object to the "fast" part. Robin Hanson objected to the "local" part. I'll try to handle both, though not all in one shot today.

We are setting forth to analyze the developmental velocity of an Artificial Intelligence. We'll break down this velocity into optimization slope, optimization resources, and optimization efficiency. We'll need to understand cascades, cycles, insight and recursion; and we'll stratify our recursive levels into the metacognitive, cognitive, metaknowledge, knowledge, and object level.

Quick review:

- "Optimization slope" is the goodness and number of opportunities in the volume of solution space you're currently exploring, on whatever your problem is;

- "Optimization resources" is how much computing power, sensory bandwidth, trials, etc. you have available to explore opportunities;

- "Optimization efficiency" is how well you use your resources. This will be determined by the goodness of your current mind design - the point in mind design space that is your current self - along with its knowledge and metaknowledge (see below).

Optimizing yourself is a special case, but it's one we're about to spend a lot of time talking about.

By the time any mind solves some kind of actual problem, there's actually been a huge causal lattice of optimizations applied - for example, humans brain evolved, and then humans developed the idea of science, and then applied the idea of science to generate knowledge about gravity, and then you use this knowledge of gravity to finally design a damn bridge or something.

So I shall stratify this causality into levels - the boundaries being semi-arbitrary, but you've got to draw them somewhere:

- "Metacognitive" is the optimization that builds the brain - in the case of a human, natural selection; in the case of an AI, either human programmers or, after some point, the AI itself.

- "Cognitive", in humans, is the labor performed by your neural circuitry, algorithms that consume large amounts of computing power but are mostly opaque to you. You know what you're seeing, but you don't know how the visual cortex works. The Root of All Failure in AI is to underestimate those algorithms because you can't see them... In an AI, the lines between procedural and declarative knowledge are theoretically blurred, but in practice it's often possible to distinguish cognitive algorithms and cognitive content.

- "Metaknowledge": Discoveries about how to discover, "Science" being an archetypal example, "Math" being another. You can think of these as reflective cognitive content (knowledge about how to think).

- "Knowledge": Knowing how gravity works.

- "Object level": Specific actual problems like building a bridge or something.

I am arguing that an AI's developmental velocity will not be smooth; the following are some classes of phenomena that might lead to non-smoothness. First, a couple of points that weren't raised earlier:

- Roughness: A search space can be naturally rough - have unevenly distributed slope. With constant optimization pressure, you could go through a long phase where improvements are easy, then hit a new volume of the search space where improvements are tough. Or vice versa. Call this factor roughness.

- Resource overhangs: Rather than resources growing incrementally by reinvestment, there's a big bucket o' resources behind a locked door, and once you unlock the door you can walk in and take them all.

And these other factors previously covered:

- Cascades are when one development leads the way to another - for example, once you discover gravity, you might find it easier to understand a coiled spring.

- Cycles are feedback loops where a process's output becomes its input on the next round. As the classic example of a fission chain reaction illustrates, a cycle whose underlying processes are continuous, may show qualitative changes of surface behavior - a threshold of criticality - the difference between each neutron leading to the emission of 0.9994 additional neutrons versus each neutron leading to the emission of 1.0006 additional neutrons. k is the effective neutron multiplication factor and I will use it metaphorically.

- Insights are items of knowledge that tremendously decrease the cost of solving a wide range of problems - for example, once you have the calculus insight, a whole range of physics problems become a whole lot easier to solve. Insights let you fly through, or teleport through, the solution space, rather than searching it by hand - that is, "insight" represents knowledge about the structure of the search space itself.

and finally,

- Recursion is the sort of thing that happens when you hand the AI the object-level problem of "redesign your own cognitive algorithms".

Suppose I go to an AI programmer and say, "Please write me a program that plays chess." The programmer will tackle this using their existing knowledge and insight in the domain of chess and search trees; they will apply any metaknowledge they have about how to solve programming problems or AI problems; they will process this knowledge using the deep algorithms of their neural circuitry; and this neutral circuitry will have been designed (or rather its wiring algorithm designed) by natural selection.

If you go to a sufficiently sophisticated AI - more sophisticated than any that currently exists - and say, "write me a chess-playing program", the same thing might happen: The AI would use its knowledge, metaknowledge, and existing cognitive algorithms. Only the AI's metacognitive level would be, not natural selection, but the object level of the programmer who wrote the AI, using their knowledge and insight etc.

Now suppose that instead you hand the AI the problem, "Write a better algorithm than X for storing, associating to, and retrieving memories". At first glance this may appear to be just another object-level problem that the AI solves using its current knowledge, metaknowledge, and cognitive algorithms. And indeed, in one sense it should be just another object-level problem. But it so happens that the AI itself uses algorithm X to store associative memories, so if the AI can improve on this algorithm, it can rewrite its code to use the new algorithm X+1.

This means that the AI's metacognitive level - the optimization process responsible for structuring the AI's cognitive algorithms in the first place - has now collapsed to identity with the AI's object level.

For some odd reason, I run into a lot of people who vigorously deny that this phenomenon is at all novel; they say, "Oh, humanity is already self-improving, humanity is already going through a FOOM, humanity is already in a Singularity" etc. etc.

Now to me, it seems clear that - at this point in the game, in advance of the observation - it is pragmatically worth drawing a distinction between inventing agriculture and using that to support more professionalized inventors, versus directly rewriting your own source code in RAM. Before you can even argue about whether the two phenomena are likely to be similar in practice, you need to accept that they are, in fact, two different things to be argued about.

And I do expect them to be very distinct in practice. Inventing science is not rewriting your neural circuitry. There is a tendency to completely overlook the power of brain algorithms, because they are invisible to introspection. It took a long time historically for people to realize that there was such a thing as a cognitive algorithm that could underlie thinking. And then, once you point out that cognitive algorithms exist, there is a tendency to tremendously underestimate them, because you don't know the specific details of how your hippocampus is storing memories well or poorly - you don't know how it could be improved, or what difference a slight degradation could make. You can't draw detailed causal links between the wiring of your neural circuitry, and your performance on real-world problems. All you can see is the knowledge and the metaknowledge, and that's where all your causal links go; that's all that's visibly important.

To see the brain circuitry vary, you've got to look at a chimpanzee, basically. Which is not something that most humans spend a lot of time doing, because chimpanzees can't play our games.

You can also see the tremendous overlooked power of the brain circuitry by observing what happens when people set out to program what looks like "knowledge" into Good-Old-Fashioned AIs, semantic nets and such. Roughly, nothing happens. Well, research papers happen. But no actual intelligence happens. Without those opaque, overlooked, invisible brain algorithms, there is no real knowledge - only a tape recorder playing back human words. If you have a small amount of fake knowledge, it doesn't do anything, and if you have a huge amount of fake knowledge programmed in at huge expense, it still doesn't do anything.

So the cognitive level - in humans, the level of neural circuitry and neural algorithms - is a level of tremendous but invisible power. The difficulty of penetrating this invisibility and creating a real cognitive level is what stops modern-day humans from creating AI. (Not that an AI's cognitive level would be made of neurons or anything equivalent to neurons; it would just do cognitive labor on the same level of organization. Planes don't flap their wings, but they have to produce lift somehow.)

Recursion that can rewrite the cognitive level is worth distinguishing.

But to some, having a term so narrow as to refer to an AI rewriting its own source code, and not to humans inventing farming, seems hardly open, hardly embracing, hardly communal; for we all know that to say two things are similar shows greater enlightenment than saying that they are different. Or maybe it's as simple as identifying "recursive self-improvement" as a term with positive affective valence, so you figure out a way to apply that term to humanity, and then you get a nice dose of warm fuzzies. Anyway.

So what happens when you start rewriting cognitive algorithms?

Well, we do have one well-known historical case of an optimization process writing cognitive algorithms to do further optimization; this is the case of natural selection, our alien god.

Natural selection seems to have produced a pretty smooth trajectory of more sophisticated brains over the course of hundreds of millions of years. That gives us our first data point, with these characteristics:

- Natural selection on sexual multicellular eukaryotic life can probably be treated as, to first order, an optimizer of roughly constant efficiency and constant resources.

- Natural selection does not have anything akin to insights. It does sometimes stumble over adaptations that prove to be surprisingly reusable outside the context for which they were adapted, but it doesn't fly through the search space like a human. Natural selection is just searching the immediate neighborhood of its present point in the solution space, over and over and over.

- Natural selection does have cascades; adaptations open up the way for further adaptations.

So - if you're navigating the search space via the ridiculously stupid and inefficient method of looking at the neighbors of the current point, without insight - with constant optimization pressure - then...

Well, I've heard it claimed that the evolution of biological brains has accelerated over time, and I've also heard that claim challenged. If there's actually been an acceleration, I would tend to attribute that to the "adaptations open up the way for further adaptations" phenomenon - the more brain genes you have, the more chances for a mutation to produce a new brain gene. (Or, more complexly: the more organismal error-correcting mechanisms the brain has, the more likely a mutation is to produce something useful rather than fatal.) In the case of hominids in particular over the last few million years, we may also have been experiencing accelerated selection on brain proteins, per se - which I would attribute to sexual selection, or brain variance accounting for a greater proportion of total fitness variance.

Anyway, what we definitely do not see under these conditions is logarithmic or decelerating progress. It did not take ten times as long to go from H. erectus to H. sapiens as from H. habilis to H. erectus. Hominid evolution did not take eight hundred million years of additional time, after evolution immediately produced Australopithecus-level brains in just a few million years after the invention of neurons themselves.

And another, similar observation: human intelligence does not require a hundred times as much computing power as chimpanzee intelligence. Human brains are merely three times too large, and our prefrontal cortices six times too large, for a primate with our body size.

Or again: It does not seem to require 1000 times as many genes to build a human brain as to build a chimpanzee brain, even though human brains can build toys that are a thousand times as neat.

Why is this important? Because it shows that with constant optimization pressure from natural selection and no intelligent insight, there were no diminishing returns to a search for better brain designs up to at least the human level. There were probably accelerating returns (with a low acceleration factor). There are no visible speedbumps, so far as I know.

But all this is to say only of natural selection, which is not recursive.

If you have an investment whose output is not coupled to its input - say, you have a bond, and the bond pays you a certain amount of interest every year, and you spend the interest every year - then this will tend to return you a linear amount of money over time. After one year, you've received $10; after 2 years, $20; after 3 years, $30.

Now suppose you change the qualitative physics of the investment, by coupling the output pipe to the input pipe. Whenever you get an interest payment, you invest it in more bonds. Now your returns over time will follow the curve of compound interest, which is exponential. (Please note: Not all accelerating processes are smoothly exponential. But this one happens to be.)

The first process grows at a rate that is linear over time; the second process grows at a rate that is linear in its cumulative return so far.

The too-obvious mathematical idiom to describe the impact of recursion is replacing an equation

y = f(t)

with

dy/dt = f(y)

For example, in the case above, reinvesting our returns transformed the linearly growing

y = m*t

into

y' = m*y

whose solution is the exponentially growing

y = e^(m*t)

Now... I do not think you can really solve equations like this to get anything like a description of a self-improving AI.

But it's the obvious reason why I don't expect the future to be a continuation of past trends. The future contains a feedback loop that the past does not.

As a different Eliezer Yudkowsky wrote, very long ago:

"If computing power doubles every eighteen months, what happens when computers are doing the research?"

And this sounds horrifyingly naive to my present ears, because that's not really how it works at all - but still, it illustrates the idea of "the future contains a feedback loop that the past does not".

History up until this point was a long story about natural selection producing humans, and then, after humans hit a certain threshold, humans starting to rapidly produce knowledge and metaknowledge that could - among other things - feed more humans and support more of them in lives of professional specialization.

To a first approximation, natural selection held still during human cultural development. Even if Gregory Clark's crazy ideas are crazy enough to be true - i.e., some human populations evolved lower discount rates and more industrious work habits over the course of just a few hundred years from 1200 to 1800 - that's just tweaking a few relatively small parameters; it is not the same as developing new complex adaptations with lots of interdependent parts. It's not a chimp-human type gap.

So then, with human cognition remaining more or less constant, we found that knowledge feeds off knowledge with k > 1 - given a background of roughly constant cognitive algorithms at the human level. We discovered major chunks of metaknowledge, like Science and the notion of Professional Specialization, that changed the exponents of our progress; having lots more humans around, due to e.g. the object-level innovation of farming, may have have also played a role. Progress in any one area tended to be choppy, with large insights leaping forward, followed by a lot of slow incremental development.

With history to date, we've got a series of integrals looking something like this:

Metacognitive = natural selection, optimization efficiency/resources roughly constant

Cognitive = Human intelligence = integral of evolutionary optimization velocity over a few hundred million years, then roughly constant over the last ten thousand years

Metaknowledge = Professional Specialization, Science, etc. = integral over cognition we did about procedures to follow in thinking, where metaknowledge can also feed on itself, there were major insights and cascades, etc.

Knowledge = all that actual science, engineering, and general knowledge accumulation we did = integral of cognition+metaknowledge(current knowledge) over time, where knowledge feeds upon itself in what seems to be a roughly exponential process

Object level = stuff we actually went out and did = integral of cognition+metaknowledge+knowledge(current solutions); over a short timescale this tends to be smoothly exponential to the degree that the people involved understand the idea of investments competing on the basis of interest rate, but over medium-range timescales the exponent varies, and on a long range the exponent seems to increase

If you were to summarize that in one breath, it would be, "with constant natural selection pushing on brains, progress was linear or mildly accelerating; with constant brains pushing on metaknowledge and knowledge and object-level progress feeding back to metaknowledge and optimization resources, progress was exponential or mildly superexponential".

Now fold back the object level so that it becomes the metacognitive level.

And note that we're doing this through a chain of differential equations, not just one; it's the final output at the object level, after all those integrals, that becomes the velocity of metacognition.

You should get...

...very fast progress? Well, no, not necessarily. You can also get nearly zero progress.

If you're a recursified optimizing compiler, you rewrite yourself just once, get a single boost in speed (like 50% or something), and then never improve yourself any further, ever again.

If you're EURISKO, you manage to modify some of your metaheuristics, and the metaheuristics work noticeably better, and they even manage to make a few further modifications to themselves, but then the whole process runs out of steam and flatlines.

It was human intelligence that produced these artifacts to begin with. Their own optimization power is far short of human - so incredibly weak that, after they push themselves along a little, they can't push any further. Worse, their optimization at any given level is characterized by a limited number of opportunities, which once used up are gone - extremely sharp diminishing returns.

When you fold a complicated, choppy, cascade-y chain of differential equations in on itself via recursion, it should either flatline or blow up. You would need exactly the right law of diminishing returns to fly through the extremely narrow soft takeoff keyhole.

The observed history of optimization to date makes this even more unlikely. I don't see any reasonable way that you can have constant evolution produce human intelligence on the observed historical trajectory (linear or accelerating), and constant human intelligence produce science and technology on the observed historical trajectory (exponential or superexponential), and fold that in on itself, and get out something whose rate of progress is in any sense anthropomorphic. From our perspective it should either flatline or FOOM.

When you first build an AI, it's a baby - if it had to improve itself, it would almost immediately flatline. So you push it along using your own cognition, metaknowledge, and knowledge - not getting any benefit of recursion in doing so, just the usual human idiom of knowledge feeding upon itself and insights cascading into insights. Eventually the AI becomes sophisticated enough to start improving itself, not just small improvements, but improvements large enough to cascade into other improvements. (Though right now, due to lack of human insight, what happens when modern researchers push on their AGI design is mainly nothing.) And then you get what I. J. Good called an "intelligence explosion".

I even want to say that the functions and curves being such as to allow hitting the soft takeoff keyhole, is ruled out by observed history to date. But there are small conceivable loopholes, like "maybe all the curves change drastically and completely as soon as we get past the part we know about in order to give us exactly the right anthropomorphic final outcome", or "maybe the trajectory for insightful optimization of intelligence has a law of diminishing returns where blind evolution gets accelerating returns".

There's other factors contributing to hard takeoff, like the existence of hardware overhang in the form of the poorly defended Internet and fast serial computers. There's more than one possible species of AI we could see, given this whole analysis. I haven't yet touched on the issue of localization (though the basic issue is obvious: the initial recursive cascade of an intelligence explosion can't race through human brains because human brains are not modifiable until the AI is already superintelligent).

But today's post is already too long, so I'd best continue tomorrow.

Post scriptum: It occurred to me just after writing this that I'd been victim of a cached Kurzweil thought in speaking of the knowledge level as "exponential". Object-level resources are exponential in human history because of physical cycles of reinvestment. If you try defining knowledge as productivity per worker, I expect that's exponential too (or productivity growth would be unnoticeable by now as a component in economic progress). I wouldn't be surprised to find that published journal articles are growing exponentially. But I'm not quite sure that it makes sense to say humanity has learned as much since 1938 as in all earlier human history... though I'm quite willing to believe we produced more goods... then again we surely learned more since 1500 than in all the time before. Anyway, human knowledge being "exponential" is a more complicated issue than I made it out to be. But human object level is more clearly exponential or superexponential.

Referenced by: Hard Takeoff • Sustained Strong Recursion • "Outside View!" as Conversation-Halter

Original with comments: Recursive Self-Improvement

Hard Takeoff

Eliezer Yudkowsky, 02 December 2008 08:44PM

Continuation of: Recursive Self-Improvement

Constant natural selection pressure, operating on the genes of the hominid line, produced improvement in brains over time that seems to have been, roughly, linear or accelerating; the operation of constant human brains on a pool of knowledge seems to have produced returns that are, very roughly, exponential or superexponential. (Robin proposes that human progress is well-characterized as a series of exponential modes with diminishing doubling times.)

Recursive self-improvement - an AI rewriting its own cognitive algorithms - identifies the object level of the AI with a force acting on the metacognitive level; it "closes the loop" or "folds the graph in on itself". E.g. the difference between returns on a constant investment in a bond, and reinvesting the returns into purchasing further bonds, is the difference between the equations y = f(t) = m*t, and dy/dt = f(y) = m*y whose solution is the compound interest exponential, y = e^(m*t).

When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM. An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely - far more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology. Our present life is no good indicator of things to come.

Or to try and compress it down to a slogan that fits on a T-Shirt - not that I'm saying this is a good idea - "Moore's Law is exponential now; it would be really odd if it stayed exponential with the improving computers doing the research." I'm not saying you literally get dy/dt = e^y that goes to infinity after finite time - and hardware improvement is in some ways the least interesting factor here - but should we really see the same curve we do now?

RSI is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it's nowhere near being the only such factor. The advent of human intelligence was a discontinuity with the past even without RSI...

...which is to say that observed evolutionary history - the discontinuity between humans, and chimps who share 95% of our DNA - lightly suggests a critical threshold built into the capabilities that we think of as "general intelligence", a machine that becomes far more powerful once the last gear is added.

This is only a light suggestion because the branching time between humans and chimps is enough time for a good deal of complex adaptation to occur. We could be looking at the sum of a cascade, not the addition of a final missing gear. On the other hand, we can look at the gross brain anatomies and see that human brain anatomy and chimp anatomy have not diverged all that much. On the gripping hand, there's the sudden cultural revolution - the sudden increase in the sophistication of artifacts - that accompanied the appearance of anatomically Cro-Magnons just a few tens of thousands of years ago.

Now of course this might all just be completely inapplicable to the development trajectory of AIs built by human programmers rather than by evolution. But it at least lightly suggests, and provides a hypothetical illustration of, a discontinuous leap upward in capability that results from a natural feature of the solution space - a point where you go from sorta-okay solutions to totally-amazing solutions as the result of a few final tweaks to the mind design.

I could potentially go on about this notion for a bit - because, in an evolutionary trajectory, it can't literally be a "missing gear", the sort of discontinuity that follows from removing a gear that an otherwise functioning machine was built around. So if you suppose that a final set of changes was enough to produce a sudden huge leap in effective intelligence, it does demand the question of what those changes were. Something to do with reflection - the brain modeling or controlling itself - would be one obvious candidate. Or perhaps a change in motivations (more curious individuals, using the brainpower they have in different directions) in which case you wouldn't expect that discontinuity to appear in the AI's development, but you would expect it to be more effective at earlier stages than humanity's evolutionary history would suggest... But you could have whole journal issues about that one question, so I'm just going to leave it at that.

Or consider the notion of sudden resource bonanzas. Suppose there's a semi-sophisticated Artificial General Intelligence running on a cluster of a thousand CPUs. The AI has not hit a wall - it's still improving itself - but its self-improvement is going so slowly that, the AI calculates, it will take another fifty years for it to engineer / implement / refine just the changes it currently has in mind. Even if this AI would go FOOM eventually, its current progress is so slow as to constitute being flatlined...

So the AI turns its attention to examining certain blobs of binary code - code composing operating systems, or routers, or DNS services - and then takes over all the poorly defended computers on the Internet. This may not require what humans would regard as genius, just the ability to examine lots of machine code and do relatively low-grade reasoning on millions of bytes of it. (I have a saying/hypothesis that a human trying to write code is like someone without a visual cortex trying to paint a picture - we can do it eventually, but we have to go pixel by pixel because we lack a sensory modality for that medium; it's not our native environment.) The Future may also have more legal ways to obtain large amounts of computing power quickly.

This sort of resource bonanza is intriguing in a number of ways. By assumption, optimization efficiency is the same, at least for the moment - we're just plugging a few orders of magnitude more resource into the current input/output curve. With a stupid algorithm, a few orders of magnitude more computing power will buy you only a linear increase in performance - I would not fear Cyc even if ran on a computer the size of the Moon, because there is no there there.

On the other hand, humans have a brain three times as large, and a prefrontal cortex six times as large, as that of a standard primate our size - so with software improvements of the sort that natural selection made over the last five million years, it does not require exponential increases in computing power to support linearly greater intelligence. Mind you, this sort of biological analogy is always fraught - maybe a human has not much more cognitive horsepower than a chimpanzee, the same underlying tasks being performed, but in a few more domains and with greater reflectivity - the engine outputs the same horsepower, but a few gears were reconfigured to turn each other less wastefully - and so you wouldn't be able to go from human to super-human with just another sixfold increase in processing power... or something like that.

But if the lesson of biology suggests anything, it is that you do not run into logarithmic returns on processing power in the course of reaching human intelligence, even when that processing power increase is strictly parallel rather than serial, provided that you are at least as good as writing software to take advantage of that increased computing power, as natural selection is at producing adaptations - five million years for a sixfold increase in computing power.

Michael Vassar observed in yesterday's comments that humans, by spending linearly more time studying chess, seem to get linear increases in their chess rank (across a wide range of rankings), while putting exponentially more time into a search algorithm is usually required to yield the same range of increase. Vassar called this "bizarre", but I find it quite natural. Deep Blue searched the raw game tree of chess; Kasparavo searched the compressed regularities of chess. It's not surprising that the simple algorithm is logarithmic and the sophisticated algorithm is linear. One might say similarly of the course of human progress seeming to be closer to exponential, while evolutionary progress is closer to being linear. Being able to understand the regularity of the search space counts for quite a lot.

If the AI is somewhere in between - not as brute-force as Deep Blue, nor as compressed as a human - then maybe a 10,000-fold increase in computing power will only buy it a 10-fold increase in optimization velocity... but that's still quite a speedup.

Furthermore, all future improvements the AI makes to itself will now be amortized over 10,000 times as much computing power to apply the algorithms. So a single improvement to code now has more impact than before; it's liable to produce more further improvements. Think of a uranium pile. It's always running the same "algorithm" with respect to neutrons causing fissions that produce further neutrons, but just piling on more uranium can cause it to go from subcritical to supercritical, as any given neutron has more uranium to travel through and a higher chance of causing future fissions.

So just the resource bonanza represented by "eating the Internet" or "discovering an application for which there is effectively unlimited demand, which lets you rent huge amounts of computing power while using only half of it to pay the bills" - even though this event isn't particularly recursive of itself, just an object-level fruit-taking - could potentially drive the AI from subcritical to supercritical.

Not, mind you, that this will happen with an AI that's just stupid. But an AI already improving itself slowly - that's a different case.

Even if this doesn't happen - if the AI uses this newfound computing power at all effectively, its optimization efficiency will increase more quickly than before; just because the AI has more optimization power to apply to the task of increasing its own efficiency, thanks to the sudden bonanza of optimization resources.

So the whole trajectory can conceivably change, just from so simple and straightforward and unclever and uninteresting-seeming an act, as eating the Internet. (Or renting a bigger cloud.)

Agriculture changed the course of human history by supporting a larger population - and that was just a question of having more humans around, not individual humans having a brain a hundred times as large. This gets us into the whole issue of the returns on scaling individual brains not being anything like the returns on scaling the number of brains. A big-brained human has around four times the cranial volume of a chimpanzee, but 4 chimps != 1 human. (And for that matter, 60 squirrels != 1 chimp.) Software improvements here almost certainly completely dominate hardware, of course. But having a thousand scientists who collectively read all the papers in a field, and who talk to each other, is not like having one superscientist who has read all those papers and can correlate their contents directly using native cognitive processes of association, recognition, and abstraction. Having more humans talking to each other using low-bandwidth words, cannot be expected to achieve returns similar to those from scaling component cognitive processes within a coherent cognitive system.

This, too, is an idiom outside human experience - we have to solve big problems using lots of humans, because there is no way to solve them using ONE BIG human. But it never occurs to anyone to substitute four chimps for one human; and only a certain very foolish kind of boss thinks you can substitute ten programmers with one year of experience for one programmer with ten years of experience.

(Part of the general Culture of Chaos that praises emergence and thinks evolution is smarter than human designers, also has a mythology of groups being inherently superior to individuals. But this is generally a matter of poor individual rationality, and various arcane group structures that are supposed to compensate; rather than an inherent fact about cognitive processes somehow scaling better when chopped up into distinct brains. If that were literally more efficient, evolution would have designed humans to have four chimpanzee heads that argued with each other. In the realm of AI, it seems much more straightforward to have a single cognitive process that lacks the emotional stubbornness to cling to its accustomed theories, and doesn't need to be argued out of it at gunpoint or replaced by a new generation of grad students. I'm not going to delve into this in detail for now, just warn you to be suspicious of this particular creed of the Culture of Chaos; it's not like they actually observed the relative performance of a hundred humans versus one BIG mind with a brain fifty times human size.)

So yes, there was a lot of software improvement involved - what we are seeing with the modern human brain size, is probably not so much the brain volume required to support the software improvement, but rather the new evolutionary equilibrium for brain size given the improved software.

Even so - hominid brain size increased by a factor of five over the course of around five million years. You might want to think very seriously about the contrast between that idiom, and a successful AI being able to expand onto five thousand times as much hardware over the course of five minutes - when you are pondering possible hard takeoffs, and whether the AI trajectory ought to look similar to human experience.

A subtler sort of hardware overhang, I suspect, is represented by modern CPUs have a 2GHz serial speed, in contrast to neurons that spike 100 times per second on a good day. The "hundred-step rule" in computational neuroscience is a rule of thumb that any postulated neural algorithm which runs in realtime has to perform its job in less than 100 serial steps one after the other. We do not understand how to efficiently use the computer hardware we have now, to do intelligent thinking. But the much-vaunted "massive parallelism" of the human brain, is, I suspect, mostly cache lookups to make up for the sheer awkwardness of the brain's serial slowness - if your computer ran at 200Hz, you'd have to resort to all sorts of absurdly massive parallelism to get anything done in realtime. I suspect that, if correctly designed, a midsize computer cluster would be able to get high-grade thinking done at a serial speed much faster than human, even if the total parallel computing power was less.

So that's another kind of overhang: because our computing hardware has run so far ahead of AI theory, we have incredibly fast computers we don't know how to use for thinking; getting AI right could produce a huge, discontinuous jolt, as the speed of high-grade thought on this planet suddenly dropped into computer time.

A still subtler kind of overhang would be represented by human failure to use our gathered experimental data efficiently.

On to the topic of insight, another potential source of discontinuity. The course of hominid evolution was driven by evolution's neighborhood search; if the evolution of the brain accelerated to some degree, this was probably due to existing adaptations creating a greater number of possibilities for further adaptations. (But it couldn't accelerate past a certain point, because evolution is limited in how much selection pressure it can apply - if someone succeeds in breeding due to adaptation A, that's less variance left over for whether or not they succeed in breeding due to adaptation B.)

But all this is searching the raw space of genes. Human design intelligence, or sufficiently sophisticated AI design intelligence, isn't like that. One might even be tempted to make up a completely different curve out of thin air - like, intelligence will take all the easy wins first, and then be left with only higher-hanging fruit, while increasing complexity will defeat the ability of the designer to make changes. So where blind evolution accelerated, intelligent design will run into diminishing returns and grind to a halt. And as long as you're making up fairy tales, you might as well further add that the law of diminishing returns will be exactly right, and have bumps and rough patches in exactly the right places, to produce a smooth gentle takeoff even after recursion and various hardware transitions are factored in... One also wonders why the story about "intelligence taking easy wins first in designing brains" tops out at or before human-level brains, rather than going a long way beyond human before topping out. But one suspects that if you tell that story, there's no point in inventing a law of diminishing returns to begin with.

(Ultimately, if the character of physical law is anything like our current laws of physics, there will be limits to what you can do on finite hardware, and limits to how much hardware you can assemble in finite time, but if they are very high limits relative to human brains, it doesn't affect the basic prediction of hard takeoff, "AI go FOOM".)

The main thing I'll venture into actually expecting from adding "insight" to the mix, is that there'll be a discontinuity at the point where the AI understands how to do AI theory, the same way that human researchers try to do AI theory. An AI, to swallow its own optimization chain, must not just be able to rewrite its own source code; it must be able to, say, rewrite Artificial Intelligence: A Modern Approach (2nd Edition). An ability like this seems (untrustworthily, but I don't know what else to trust) like it ought to appear at around the same time that the architecture is at the level of, or approaching the level of, being able to handle what humans handle - being no shallower than an actual human, whatever its inexperience in various domains. It would produce further discontinuity at around that time.

In other words, when the AI becomes smart enough to do AI theory, that's when I expect it to fully swallow its own optimization chain and for the real FOOM to occur - though the AI might reach this point as part of a cascade that started at a more primitive level.

All these complications is why I don't believe we can really do any sort of math that will predict quantitatively the trajectory of a hard takeoff. You can make up models, but real life is going to include all sorts of discrete jumps, bottlenecks, bonanzas, insights - and the "fold the curve in on itself" paradigm of recursion is going to amplify even small roughnesses in the trajectory.

So I stick to qualitative predictions. "AI go FOOM".

Tomorrow I hope to tackle locality, and a bestiary of some possible qualitative trajectories the AI might take given this analysis. Robin Hanson's summary of "primitive AI fooms to sophisticated AI" doesn't fully represent my views - that's just one entry in the bestiary, albeit a major one.

Referenced by: Permitted Possibilities, & Locality • Not Taking Over the World • The Thing That I Protect

Original with comments: Hard Takeoff

Permitted Possibilities, & Locality

Eliezer Yudkowsky, 03 December 2008 09:20PM

Continuation of: Hard Takeoff

The analysis given in the last two days permits more than one possible AI trajectory:

1. Programmers, smarter than evolution at finding tricks that work, but operating without fundamental insight or with only partial insight, create a mind that is dumber than the researchers but performs lower-quality operations much faster. This mind reaches k > 1, cascades up to the level of a very smart human, itself achieves insight into intelligence, and undergoes the really fast part of the FOOM, to superintelligence. This would be the major nightmare scenario for the origin of an unFriendly AI.

2. Programmers operating with partial insight, create a mind that performs a number of tasks very well, but can't really handle self-modification let alone AI theory. A mind like this might progress with something like smoothness, pushed along by the researchers rather than itself, even all the way up to average-human capability - not having the insight into its own workings to push itself any further. We also suppose that the mind is either already using huge amounts of available hardware, or scales very poorly, so it cannot go FOOM just as a result of adding a hundred times as much hardware. This scenario seems less likely to my eyes, but it is not ruled out by any effect I can see.

3. Programmers operating with strong insight into intelligence, directly create along an efficient and planned pathway, a mind capable of modifying itself with deterministic precision - provably correct or provably noncatastrophic self-modifications. This is the only way I can see to achieve narrow enough targeting to create a Friendly AI. The "natural" trajectory of such an agent would be slowed by the requirements of precision, and sped up by the presence of insight; but because this is a Friendly AI, notions like "You can't yet improve yourself this far, your goal system isn't verified enough" would play a role.

So these are some things that I think are permitted to happen, albeit that case 2 would count as a hit against me to some degree because it does seem unlikely.

Here are some things that shouldn't happen, on my analysis:

- An ad-hoc self-modifying AI as in (1) undergoes a cycle of self-improvement, starting from stupidity, that carries it up to the level of a very smart human - and then stops, unable to progress any further. (The upward slope in this region is supposed to be very steep!)

- A mostly non-self-modifying AI as in (2) is pushed by its programmers up to a roughly human level... then to the level of a very smart human... then to the level of a mild transhuman... but the mind still does not achieve insight into its own workings and still does not undergo an intelligence explosion - just continues to increase smoothly in intelligence from there.

And I also don't think this is allowed:

- The "scenario that Robin Hanson seems to think is the line-of-maximum-probability for AI as heard and summarized by Eliezer Yudkowsky":

- No one AI that does everything humans do, but rather a large, diverse population of AIs. These AIs have various domain-specific competencies that are "human+ level" - not just in the sense of Deep Blue beating Kasparov, but in the sense that in these domains, the AIs seem to have good "common sense" and can e.g. recognize, comprehend and handle situations that weren't in their original programming. But only in the special domains for which that AI was crafted/trained. Collectively, these AIs may be strictly more competent than any one human, but no individual AI is more competent than any one human.

- Knowledge and even skills are widely traded in this economy of AI systems.

- In concert, these AIs, and their human owners, and the economy that surrounds them, undergo a collective FOOM of self-improvement. No local agent is capable of doing all this work, only the collective system.

- The FOOM's benefits are distributed through a whole global economy of trade partners and suppliers, including existing humans and corporations, though existing humans and corporations may form an increasingly small fraction of the New Economy.

- This FOOM looks like an exponential curve of compound interest, like the modern world but with a substantially shorter doubling time.

Mostly, Robin seems to think that uploads will come first, but that's a whole 'nother story. So far as AI goes, this looks like Robin's maximum line of probability - and if I got this mostly wrong or all wrong, that's no surprise. Robin Hanson did the same to me when summarizing what he thought were my own positions. I have never thought, in prosecuting this Disagreement, that we were starting out with a mostly good understanding of what the Other was thinking; and this seems like an important thing to have always in mind.

So - bearing in mind that I may well be criticizing a straw misrepresentation, and that I know this full well, but I am just trying to guess my best - here's what I see as wrong with the elements of this scenario:

• The abilities we call "human" are the final products of an economy of mind - not in the sense that there are selfish agents in it, but in the sense that there are production lines; and I would even expect evolution to enforce something approaching fitness as a common unit of currency. (Enough selection pressure to create an adaptation from scratch should be enough to fine-tune the resource curves involved.) It's the production lines, though, that are the main point - that your brain has specialized parts and the specialized parts pass information around. All of this goes on behind the scenes, but it's what finally adds up to any single human ability.

In other words, trying to get humanlike performance in just one domain, is divorcing a final product of that economy from all the work that stands behind it. It's like having a global economy that can only manufacture toasters, but not dishwashers or light bulbs. You can have something like Deep Blue that beats humans at chess in an inhuman, specialized way; but I don't think it would be easy to get humanish performance at, say, biology R&D, without a whole mind and architecture standing behind it, that would also be able to accomplish other things. Tasks that draw on our cross-domain-ness, or our long-range real-world strategizing, or our ability to formulate new hypotheses, or our ability to use very high-level abstractions - I don't think that you would be able to replace a human in just that one job, without also having something that would be able to learn many different jobs.

I think it is a fair analogy to the idea that you shouldn't see a global economy that can manufacture toasters but not manufacture anything else.

This is why I don't think we'll see a system of AIs that are diverse, individually highly specialized, and only collectively able to do anything a human can do.

• Trading cognitive content around between diverse AIs is more difficult and less likely than it might sound. Consider the field of AI as it works today. Is there any standard database of cognitive content that you buy off the shelf and plug into your amazing new system, whether it be a chessplayer or a new data-mining algorithm? If it's a chess-playing program, there are databases of stored games - but that's not the same as having databases of preprocessed cognitive content.

So far as I can tell, the diversity of cognitive architectures acts as a tremendous barrier to trading around cognitive content. If you have many AIs around that are all built on the same architecture by the same programmers, they might, with a fair amount of work, be able to pass around learned cognitive content. Even this is less trivial than it sounds. If two AIs both see an apple for the first time, and they both independently form concepts about that apple, and they both independently build some new cognitive content around those concepts, then their thoughts are effectively written in a different language. By seeing a single apple at the same time, they could identify a concept they both have in mind, and in this way build up a common language...

...the point being that even when two separated minds are running literally the same source code, it is still difficult for them to trade new knowledge as raw cognitive content without having a special language designed just for sharing knowledge.

Now suppose the two AIs are built around different architectures.

The barrier this opposes to a true, cross-agent, literal "economy of mind", is so strong, that in the vast majority of AI applications you set out to write today, you will not bother to import any standardized preprocessed cognitive content. It will be easier for your AI application to start with some standard examples - databases of that sort of thing do exist, in some fields anyway - and redo all the cognitive work of learning on its own.

That's how things stand today.

And I have to say that looking over the diversity of architectures proposed at any AGI conference I've attended, it is very hard to imagine directly trading cognitive content between any two of them. It would be an immense amount of work just to set up a language in which they could communicate what they take to be facts about the world - never mind preprocessed cognitive content.

This is a force for localization: unless the condition I have just described changes drastically, it means that agents will be able to do their own cognitive labor, rather than needing to get their brain content manufactured elsewhere, or even being able to get their brain content manufactured elsewhere. I can imagine there being an exception to this for non-diverse agents that are deliberately designed to carry out this kind of trading within their code-clade. (And in the long run, difficulties of translation seems less likely to stop superintelligences.)

But in today's world, it seems to be the rule that when you write a new AI program, you can sometimes get preprocessed raw data, but you will not buy any preprocessed cognitive content - the internal content of your program will come from within your program.

And it actually does seem to me that AI would have to get very sophisticated before it got over the "hump" of increased sophistication making sharing harder instead of easier. I'm not sure this is pre-takeoff sophistication we're talking about, here. And the cheaper computing power is, the easier it is to just share the data and do the learning on your own.

Again - in today's world, sharing of cognitive content between diverse AIs doesn't happen, even though there are lots of machine learning algorithms out there doing various jobs. You could say things would happen differently in the future, but it'd be up to you to make that case.

• Understanding the difficulty of interfacing diverse AIs, is the next step toward understanding why it's likely to be a single coherent cognitive system that goes FOOM via recursive self-improvement. The same sort of barriers that apply to trading direct cognitive content, would also apply to trading changes in cognitive source code.

It's a whole lot easier to modify the source code in the interior of your own mind, than to take that modification, and sell it to a friend who happens to be written on different source code.

Certain kinds of abstract insights would be more tradeable, among sufficiently sophisticated minds; and the major insights might be well worth selling - like, if you invented a new general algorithm at some subtask that many minds perform. But if you again look at the modern state of the field, then you find that it is only a few algorithms that get any sort of general uptake.

And if you hypothesize minds that understand these algorithms, and the improvements to them, and what these algorithms are for, and how to implement and engineer them - then these are already very sophisticated minds, at this point, they are AIs that can do their own AI theory. So the hard takeoff has to have not already started, yet, at this point where there are many AIs around that can do AI theory. If they can't do AI theory, diverse AIs are likely to experience great difficulties trading code improvements among themselves.

This is another localizing force. It means that the improvements you make to yourself, and the compound interest earned on those improvements, is likely to stay local.

If the scenario with an AI takeoff is anything at all like the modern world in which all the attempted AGI projects have completely incommensurable architectures, then any self-improvements will definitely stay put, not spread.

• But suppose that the situation did change drastically from today, and that you had a community of diverse AIs which were sophisticated enough to share cognitive content, code changes, and even insights. And suppose even that this is true at the start of the FOOM - that is, the community of diverse AIs got all the way up to that level, without yet using a FOOM or starting a FOOM at a time when it would still be localized.

We can even suppose that most of the code improvements, algorithmic insights, and cognitive content driving any particular AI, is coming from outside that AI - sold or shared - so that the improvements the AI makes to itself, do not dominate its total velocity.

Fine. The humans are not out of the woods.

Even if we're talking about uploads, it will be immensely more difficult to apply any of the algorithmic insights that are tradeable between AIs, to the undocumented human brain, that is a huge mass of spaghetti code, that was never designed to be upgraded, that is not end-user-modifiable, that is not hot-swappable, that is written for a completely different architecture than what runs efficiently on modern processors...

And biological humans? Their neurons just go on doing whatever neurons do, at 100 cycles per second (tops).

So this FOOM that follows from recursive self-improvement, the cascade effect of using your increased intelligence to rewrite your code and make yourself even smarter -

The barriers to sharing cognitive improvements among diversely designed AIs, are large; the barriers to sharing with uploaded humans, are incredibly huge; the barrier to sharing with biological humans, is essentially absolute. (Barring a (benevolent) superintelligence with nanotechnology, but if one of those is around, you have already won.)

In this hypothetical global economy of mind, the humans are like a country that no one can invest in, that cannot adopt any of the new technologies coming down the line.

I once observed that Ricardo's Law of Comparative Advantage is the theorem that unemployment should not exist. The gotcha being that if someone is sufficiently unreliable, there is a cost to you to train them, a cost to stand over their shoulders and monitor them, a cost to check their results for accuracy - the existence of unemployment in our world is a combination of transaction costs like taxes, regulatory barriers like minimum wage, and above all, lack of trust. There are a dozen things I would pay someone else to do for me - if I wasn't paying taxes on the transaction, and if I could trust a stranger as much as I trust myself (both in terms of their honesty and of acceptable quality of output). Heck, I'd as soon have some formerly unemployed person walk in and spoon food into my mouth while I kept on typing at the computer - if there were no transaction costs, and I trusted them.

If high-quality thought drops into a speed closer to computer time by a few orders of magnitude, no one is going to take a subjective year to explain to a biological human an idea that they will be barely able to grasp, in exchange for an even slower guess at an answer that is probably going to be wrong anyway.

Even uploads could easily end up doomed by this effect, not just because of the immense overhead cost and slowdown of running their minds, but because of the continuing error-proneness of the human architecture. Who's going to trust a giant messy undocumented neural network, any more than you'd run right out and hire some unemployed guy off the street to come into your house and do your cooking?

This FOOM leaves humans behind -

- unless you go the route of Friendly AI, and make a superintelligence that simply wants to help humans, not for any economic value that humans provide to it, but because that is its nature.

And just to be clear on something - which really should be clear by now, from all my other writing, but maybe you're just wandering in - it's not that having squishy things running around on two legs is the ultimate height of existence. But if you roll up a random AI with a random utility function, it just ends up turning the universe into patterns we would not find very eudaimonic - turning the galaxies into paperclips. If you try a haphazard attempt at making a "nice" AI, the sort of not-even-half-baked theories I see people coming up with on the spot and occasionally writing whole books about, like using reinforcement learning on pictures of smiling humans to train the AI to value happiness, yes this was a book, then the AI just transforms the galaxy into tiny molecular smileyfaces...

It's not some small, mean desire to survive for myself at the price of greater possible futures, that motivates me. The thing is - those greater possible futures, they don't happen automatically. There are stakes on the table that are so much an invisible background of your existence that it would never occur to you they could be lost; and these things will be shattered by default, if not specifically preserved.

• And as for the idea that the whole thing would happen slowly enough for humans to have plenty of time to react to things - a smooth exponential shifted into a shorter doubling time - of that, I spoke yesterday. Progress seems to be exponential now, more or less, or at least accelerating, and that's with constant human brains. If you take a nonrecursive accelerating function and fold it in on itself, you are going to get superexponential progress. "If computing power doubles every eighteen months, what happens when computers are doing the research" should not just be a faster doubling time. (Though, that said, on any sufficiently short timescale, progress might well locally approximate an exponential because investments will shift in such fashion that the marginal returns on investment balance, even in the interior of a single mind; interest rates consistent over a timespan imply smooth exponential growth over that timespan.)

You can't count on warning, or time to react. If an accident sends a sphere of plutonium, not critical, but prompt critical, neutron output can double in a tenth of a second even with k = 1.0006. It can deliver a killing dose of radiation or blow the top off a nuclear reactor before you have time to draw a breath. Computers, like neutrons, already run on a timescale much faster than human thinking. We are already past the world where we can definitely count on having time to react.

When you move into the transhuman realm, you also move into the realm of adult problems. To wield great power carries a price in great precision. You can build a nuclear reactor but you can't ad-lib it. On the problems of this scale, if you want the universe to end up a worthwhile place, you can't just throw things into the air and trust to luck and later correction. That might work in childhood, but not on adult problems where the price of one mistake can be instant death.

Making it into the future is an adult problem. That's not a death sentence. I think. It's not the inevitable end of the world. I hope. But if you want humankind to survive, and the future to be a worthwhile place, then this will take careful crafting of the first superintelligence - not just letting economics or whatever take its easy, natural course. The easy, natural course is fatal - not just to ourselves but to all our hopes.

That, itself, is natural. It is only to be expected. To hit a narrow target you must aim; to reach a good destination you must steer; to win, you must make an extra-ordinary effort.

Referenced by: Disjunctions, Antipredictions, Etc.

Original with comments: Permitted Possibilities, & Locality

Underconstrained Abstractions

Eliezer Yudkowsky, 04 December 2008 01:58PM

Followup to: The Weak Inside View

Saith Robin:

"It is easy, way too easy, to generate new mechanisms, accounts, theories, and abstractions. To see if such things are useful, we need to vet them, and that is easiest "nearby", where we know a lot. When we want to deal with or understand things "far", where we know little, we have little choice other than to rely on mechanisms, theories, and concepts that have worked well near. Far is just the wrong place to try new things."

Well... I understand why one would have that reaction. But I'm not sure we can really get away with that.

When possible, I try to talk in concepts that can be verified with respect to existing history. When I talk about natural selection not running into a law of diminishing returns on genetic complexity or brain size, I'm talking about something that we can try to verify by looking at the capabilities of other organisms with brains big and small. When I talk about the boundaries to sharing cognitive content between AI programs, you can look at the field of AI the way it works today and see that, lo and behold, there isn't a lot of cognitive content shared.

But in my book this is just one trick in a library of methodologies for dealing with the Future, which is, in general, a hard thing to predict.

Let's say that instead of using my complicated-sounding disjunction (many different reasons why the growth trajectory might contain an upward cliff, which don't all have to be true), I instead staked my whole story on the critical threshold of human intelligence. Saying, "Look how sharp the slope is here!" - well, it would sound like a simpler story. It would be closer to fitting on a T-Shirt. And by talking about just that one abstraction and no others, I could make it sound like I was dealing in verified historical facts - humanity's evolutionary history is something that has already happened.

But speaking of an abstraction being "verified" by previous history is a tricky thing. There is this little problem of underconstraint - of there being more than one possible abstraction that the data "verifies".

In "Cascades, Cycles, Insight" I said that economics does not seem to me to deal much in the origins of novel knowledge and novel designs, and said, "If I underestimate your power and merely parody your field, by all means inform me what kind of economic study has been done of such things." This challenge was answered by comments directing me to some papers on "endogenous growth", which happens to be the name of theories that don't take productivity improvements as exogenous forces.

I've looked at some literature on endogenous growth. And don't get me wrong, it's probably not too bad as economics. However, the seminal literature talks about ideas being generated by combining other ideas, so that if you've got N ideas already and you're combining them three at a time, that's a potential N!/((3!)(N - 3!)) new ideas to explore. And then goes on to note that, in this case, there will be vastly more ideas than anyone can explore, so that the rate at which ideas are exploited will depend more on a paucity of explorers than a paucity of ideas.

Well... first of all, the notion that "ideas are generated by combining other ideas N at a time" is not exactly an amazing AI theory; it is an economist looking at, essentially, the whole problem of AI, and trying to solve it in 5 seconds or less. It's not as if any experiment was performed to actually watch ideas recombining. Try to build an AI around this theory and you will find out in very short order how useless it is as an account of where ideas come from...

But more importantly, if the only proposition you actually use in your theory is that there are more ideas than people to exploit them, then this is the only proposition that can even be partially verified by testing your theory.

Even if a recombinant growth theory can be fit to the data, then the historical data still underconstrains the many possible abstractions that might describe the number of possible ideas available - any hypothesis that has around "more ideas than people to exploit them" will fit the same data equally well. You should simply say, "I assume there are more ideas than people to exploit them", not go so far into mathematical detail as to talk about N choose 3 ideas. It's not that the dangling math here is underconstrained by the previous data, but that you're not even using it going forward.

(And does it even fit the data? I have friends in venture capital who would laugh like hell at the notion that there's an unlimited number of really good ideas out there. Some kind of Gaussian or power-law or something distribution for the goodness of available ideas seems more in order... I don't object to "endogenous growth" simplifying things for the sake of having one simplified abstraction and seeing if it fits the data well; we all have to do that. Claiming that the underlying math doesn't just let you build a useful model, but also has a fairly direct correspondence to reality, ought to be a whole 'nother story, in economics - or so it seems to me.)

(If I merely misinterpret the endogenous growth literature or underestimate its sophistication, by all means correct me.)

The further away you get from highly regular things like atoms, and the closer you get to surface phenomena that are the final products of many moving parts, the more history underconstrains the abstractions that you use. This is part of what makes futurism difficult. If there were obviously only one story that fit the data, who would bother to use anything else?

Is Moore's Law a story about the increase in computing power over time - the number of transistors on a chip, as a function of how far the planets have spun in their orbits, or how many times a light wave emitted from a cesium atom has changed phase?

Or does the same data equally verify a hypothesis about exponential increases in investment in manufacturing facilities and R&D, with an even higher exponent, showing a law of diminishing returns?

Or is Moore's Law showing the increase in computing power, as a function of some kind of optimization pressure applied by human researchers, themselves thinking at a certain rate?

That last one might seem hard to verify, since we've never watched what happens when a chimpanzee tries to work in a chip R&D lab. But on some raw, elemental level - would the history of the world really be just the same, proceeding on just exactly the same timeline as the planets move in their orbits, if, for these last fifty years, the researchers themselves had been running on the latest generation of computer chip at any given point? That sounds to me even sillier than having a financial model in which there's no way to ask what happens if real estate prices go down.

And then, when you apply the abstraction going forward, there's the question of whether there's more than one way to apply it - which is one reason why a lot of futurists tend to dwell in great gory detail on the past events that seem to support their abstractions, but just assume a single application forward.

E.g. Moravec in '88, spending a lot of time talking about how much "computing power" the human brain seems to use - but much less time talking about whether an AI would use the same amount of computing power, or whether using Moore's Law to extrapolate the first supercomputer of this size is the right way to time the arrival of AI. (Moravec thought we were supposed to have AI around now, based on his calculations - and he underestimated the size of the supercomputers we'd actually have in 2008.)

That's another part of what makes futurism difficult - after you've told your story about the past, even if it seems like an abstraction that can be "verified" with respect to the past (but what if you overlooked an alternative story for the same evidence?) that often leaves a lot of slack with regards to exactly what will happen with respect to that abstraction, going forward.

So if it's not as simple as just using the one trick of finding abstractions you can easily verify on available data...

...what are some other tricks to use?

Referenced by: True Sources of Disagreement • Disjunctions, Antipredictions, Etc.

Original with comments: Underconstrained Abstractions

Sustained Strong Recursion

Eliezer Yudkowsky, 05 December 2008 09:03PM

Followup to: Cascades, Cycles, Insight, Recursion, Magic

We seem to have a sticking point at the concept of "recursion", so I'll zoom in.

You have a friend who, even though he makes plenty of money, just spends all that money every month. You try to persuade your friend to invest a little - making valiant attempts to explain the wonders of compound interest by pointing to analogous processes in nature, like fission chain reactions.

"All right," says your friend, and buys a ten-year bond for $10,000, with an annual coupon of $500. Then he sits back, satisfied. "There!" he says. "Now I'll have an extra $500 to spend every year, without my needing to do any work! And when the bond comes due, I'll just roll it over, so this can go on indefinitely. Surely, now I'm taking advantage of the power of recursion!"

"Um, no," you say. "That's not exactly what I had in mind when I talked about 'recursion'."

"But I used some of my cumulative money earned, to increase my very earning rate," your friend points out, quite logically. "If that's not 'recursion', what is? My earning power has been 'folded in on itself', just like you talked about!"

"Well," you say, "not exactly. Before, you were earning $100,000 per year, so your cumulative earnings went as 100000 * t. Now, your cumulative earnings are going as 100500 * t. That's not really much of a change. What we want is for your cumulative earnings to go as B * e^At for some constants A and B - to grow exponentially."

"Exponentially!" says your friend, shocked.

"Yes," you say, "recursification has an amazing power to transform growth curves. In this case, it can turn a linear process into an exponential one. But to get that effect, you have to reinvest the coupon payments you get on your bonds - or at least reinvest some of them, instead of just spending them all. And you must be able to do this over and over again. Only then will you get the 'folding in' transformation, so that instead of your cumulative earnings going as y = F(t) = A*t, your earnings will go as the differential equation dy/dt = F(y) = A*y whose solution is y = e^(A*t)."

(I'm going to go ahead and leave out various constants of integration; feel free to add them back in.)

"Hold on," says your friend. "I don't understand the justification for what you just did there."

"Right now," you explain, "you're earning a steady income at your job, and you also have $500/year from the bond you bought. These are just things that go on generating money at a constant rate per unit time, in the background. So your cumulative earnings are the integral of that constant rate. If your earnings are y, then dy/dt = A, which resolves to y = At. But now, suppose that instead of having these constant earning forces operating in the background, we introduce a strong feedback loop from your cumulative earnings to your earning power."

"But I bought this one bond here -" says your friend.

"That's not enough for a strong feedback loop," you say. "Future increases in your cumulative earnings aren't going to increase the value of this one bond, or your salary, any further. One unit of force transmitted back is not a feedback loop - it has to be repeatable. You need a sustained recursion, not a one-off event."

"Okay," says your friend, "how about if I buy a $100 bond every year, then? Will that satisfy the strange requirements of this ritual?"

"Still not a strong feedback loop," you say. "Suppose that next year your salary went up $10,000/year - no, an even simpler example; suppose $10,000 fell in your lap out of the sky. If you only buy $100/year of bonds, that extra $10,000 isn't going to make any long-term difference to the earning curve. But if you're in the habit of investing 50% of found money, then there's a strong feedback loop from your cumulative earnings back to your earning power - we can pump up the cumulative earnings and watch the earning power rise as a direct result."

"How about if I just invest 0.1% of all my earnings, including the coupons on my bonds?" asks your friend.

"Well..." you say slowly. "That would be a sustained feedback loop but an extremely weak one, where marginal changes to your earnings have relatively small marginal effects on future earning power. I guess it would genuinely be a recursified process, but it would take a long time for the effects to become apparent, and any stronger recursions would easily outrun it."

"Okay," says your friend, "I'll start by investing a dollar, and I'll fully reinvest all the earnings from it, and the earnings on those earnings as well -"

"I'm not really sure there are any good investments that will let you invest just a dollar without it being eaten up in transaction costs," you say, "and it might not make a difference to anything on the timescales we have in mind - though there's an old story about a king, and grains of wheat placed on a chessboard... But realistically, a dollar isn't enough to get started."

"All right," says your friend, "suppose I start with $100,000 in bonds, and reinvest 80% of the coupons on those bonds plus rolling over all the principle, at a 5% interest rate, and we ignore inflation for now."

"Then," you reply, "we have the differential equation dy/dt = 0.8 * 0.05 * y, with the initial condition y = $100,000 at t=0, which works out to y = $100,000 * e^(.04*t). Or if you're reinvesting discretely rather than continuously, y = $100,000 * (1.04)^t."

We can similarly view the self-optimizing compiler in this light - it speeds itself up once, but never makes any further improvements, like buying a single bond; it's not a sustained recursion.

And now let us turn our attention to Moore's Law.

I am not a fan of Moore's Law. I think it's a red herring. I don't think you can forecast AI arrival times by using it, I don't think that AI (especially the good kind of AI) depends on Moore's Law continuing. I am agnostic about how long Moore's Law can continue - I simply leave the question to those better qualified, because it doesn't interest me very much...

But for our next simpler illustration of a strong recursification, we shall consider Moore's Law.

Tim Tyler serves us the duty of representing our strawman, repeatedly telling us, "But chip engineers use computers now, so Moore's Law is already recursive!"

To test this, we perform the equivalent of the thought experiment where we drop $10,000 out of the sky - push on the cumulative "wealth", and see what happens to the output rate.

Suppose that Intel's engineers could only work using computers of the sort available in 1998. How much would the next generation of computers be slowed down?

Suppose we gave Intel's engineers computers from 2018, in sealed black boxes (not transmitting any of 2018's knowledge). How much would Moore's Law speed up?

I don't work at Intel, so I can't actually answer those questions. I think, though, that if you said in the first case, "Moore's Law would drop way down, to something like 1998's level of improvement measured linearly in additional transistors per unit time," you would be way off base. And if you said in the second case, "I think Moore's Law would speed up by an order of magnitude, doubling every 1.8 months, until they caught up to the '2018' level," you would be equally way off base.

In both cases, I would expect the actual answer to be "not all that much happens". Seventeen instead of eighteen months, nineteen instead of eighteen months, something like that.

Yes, Intel's engineers have computers on their desks. But the serial speed or per-unit price of computing power is not, so far as I know, the limiting resource that bounds their research velocity. You'd probably have to ask someone at Intel to find out how much of their corporate income they spend on computing clusters / supercomputers, but I would guess it's not much compared to how much they spend on salaries or fab plants.

If anyone from Intel reads this, and wishes to explain to me how it would be unbelievably difficult to do their jobs using computers from ten years earlier, so that Moore's Law would slow to a crawl - then I stand ready to be corrected. But relative to my present state of partial knowledge, I would say that this does not look like a strong feedback loop.

However...

Suppose that the researchers themselves are running as uploads, software on the computer chips produced by their own factories.

Mind you, this is not the tiniest bit realistic. By my standards it's not even a very interesting way of looking at the Singularity, because it does not deal with smarter minds but merely faster ones - it dodges the really difficult and interesting part of the problem.

Just as nine women cannot gestate a baby in one month; just as ten thousand researchers cannot do in one year what a hundred researchers can do in a hundred years; so too, a chimpanzee cannot do four years what a human can do in one year, even though the chimp has around one-fourth the human's cranial capacity. And likewise a chimp cannot do in 100 years what a human does in 95 years, even though they share 95% of our genetic material.

Better-designed minds don't scale the same way as larger minds, and larger minds don't scale the same way as faster minds, any more than faster minds scale the same way as more numerous minds. So the notion of merely faster researchers, in my book, fails to address the interesting part of the "intelligence explosion".

Nonetheless, for the sake of illustrating this matter in a relatively simple case...

Suppose the researchers and engineers themselves - and the rest of the humans on the planet, providing a market for the chips and investment for the factories - are all running on the same computer chips that are the product of these selfsame factories. Suppose also that robotics technology stays on the same curve and provides these researchers with fast manipulators and fast sensors. We also suppose that the technology feeding Moore's Law has not yet hit physical limits. And that, as human brains are already highly parallel, we can speed them up even if Moore's Law is manifesting in increased parallelism instead of faster serial speeds - we suppose the uploads aren't yet being run on a fully parallelized machine, and so their actual serial speed goes up with Moore's Law. Etcetera.

In a fully naive fashion, we just take the economy the way it is today, and run it on the computer chips that the economy itself produces.

In our world where human brains run at constant speed (and eyes and hands work at constant speed), Moore's Law for computing power s is:

s = R(t) = e^t

The function R is the Research curve that relates the amount of Time t passed, to the current Speed of computers s.

To understand what happens when the researchers themselves are running on computers, we simply suppose that R does not relate computing technology to sidereal time - the orbits of the planets, the motion of the stars - but, rather, relates computing technology to the amount of subjective time spent researching it.

Since in our world, subjective time is a linear function of sidereal time, this hypothesis fits exactly the same curve R to observed human history so far.

Our direct measurements of observables do not constrain between the two hypotheses

Moore's Law is exponential in the number of orbits of Mars around the Sun

and

Moore's Law is exponential in the amount of subjective time that researchers spend thinking, and experimenting and building using a proportional amount of sensorimotor bandwidth.

But our prior knowledge of causality may lead us to prefer the second hypothesis.

So to understand what happens when the Intel engineers themselves run on computers (and use robotics) subject to Moore's Law, we recursify and get:

dy/dt = s = R(y) = e^y

Here y is the total amount of elapsed subjective time, which at any given point is increasing according to the computer speed s given by Moore's Law, which is determined by the same function R that describes how Research converts elapsed subjective time into faster computers. Observed human history to date roughly matches the hypothesis that R is exponential with a doubling time of eighteen subjective months (or whatever).

Solving

dy/dt = e^y

yields

y = -ln(C - t)

One observes that this function goes to +infinity at a finite time C.

This is only to be expected, given our assumptions. After eighteen sidereal months, computing speeds double; after another eighteen subjective months, or nine sidereal months, computing speeds double again; etc.

Now, unless the physical universe works in a way that is not only different from the current standard model, but has a different character of physical law than the current standard model, you can't actually do infinite computation in finite time.

Let us suppose that if our biological world had no Singularity, and Intel just kept on running as a company, populated by humans, forever, that Moore's Law would start to run into trouble around 2020. Say, after 2020 there would be a ten-year gap where chips simply stagnated, until the next doubling occurred after a hard-won breakthrough in 2030.

This just says that R(y) is not an indefinite exponential curve. By hypothesis, from subjective years 2020 to 2030, R(y) is flat, corresponding to a constant computer speed s. So dy/dt is constant over this same time period: Total elapsed subjective time y grows at a linear rate, and as y grows, R(y) and computing speeds remain flat, until ten subjective years have passed. So the sidereal bottleneck lasts ten subjective years times the current sidereal/subjective conversion rate at 2020's computing speeds.

In short, the whole scenario behaves exactly like what you would expect - the simple transform really does describe the naive scenario of "drop the economy into the timescale of its own computers".

After subjective year 2030, things pick up again, maybe - there are ultimate physical limits on computation, but they're pretty damned high, and we've got a ways to go until there. But maybe Moore's Law is slowing down - going subexponential, and then as the physical limits are approached, logarithmic, and then simply giving out.

But whatever your beliefs about where Moore's Law ultimately goes, you can just map out the way you would expect the research function R to work as a function of sidereal time in our own world, and then apply the transformation dy/dt = R(y) to get the progress of the uploaded civilization over sidereal time t. (Its progress over subjective time is simply given by R.)

If sensorimotor bandwidth is the critical limiting resource, then we instead care about R&D on fast sensors and fast manipulators. We want R_sm(y) instead R(y), where R_sm is the progress rate of sensors and manipulators, as a function of elapsed sensorimotor time. And then we write dy/dt = R_sm(y) and crank on the equation again to find out what the world looks like from a sidereal perspective.

We can verify that the Moore's Researchers scenario is a strong positive feedback loop by performing the "drop $10,000" thought experiment. Say, we drop in chips from another six doublings down the road - letting the researchers run on those faster chips, while holding constant their state of technological knowledge.

Lo and behold, this drop has a rather large impact, much larger than the impact of giving faster computers to our own biological world's Intel. Subjectively the impact may be unnoticeable - as a citizen, you just see the planets slow down again in the sky. But sidereal growth rates increase by a factor of 64.

So this is indeed deserving of the names, "strong positive feedback loop" and "sustained recursion".

As disclaimed before, all this isn't really going to happen. There would be effects like those Robin Hanson prefers to analyze, from being able to spawn new researchers as the cost of computing power decreased. You might be able to pay more to get researchers twice as fast. Above all, someone's bound to try hacking the uploads for increased intelligence... and then those uploads will hack themselves even further... Not to mention that it's not clear how this civilization cleanly dropped into computer time in the first place.

So no, this is not supposed to be a realistic vision of the future.

But, alongside our earlier parable of compound interest, it is supposed to be an illustration of how strong, sustained recursion has much more drastic effects on the shape of a growth curve, than a one-off case of one thing leading to another thing. Intel's engineers running on computers is not like Intel's engineers using computers.

Referenced by: True Sources of Disagreement • What I Think, If Not Why

Original with comments: Sustained Strong Recursion

Is That Your True Rejection?

Eliezer Yudkowsky, 06 December 2008 02:26PM

It happens every now and then, that the one encounters some of my transhumanist-side beliefs - as opposed to my ideas having to do with human rationality - strange, exotic-sounding ideas like superintelligence and Friendly AI. And the one rejects them.

If the one is called upon to explain the rejection, not uncommonly the one says,

"Why should I believe anything Yudkowsky says? He doesn't have a PhD!"

And occasionally someone else, hearing, says, "Oh, you should get a PhD, so that people will listen to you." Or this advice may even be offered by the same one who disbelieved, saying, "Come back when you have a PhD."

Now there are good and bad reasons to get a PhD, but this is one of the bad ones.

There's many reasons why someone actually has an adverse reaction to transhumanist theses. Most are matters of pattern recognition, rather than verbal thought: the thesis matches against "strange weird idea" or "science fiction" or "end-of-the-world cult" or "overenthusiastic youth".

So immediately, at the speed of perception, the idea is rejected. If, afterward, someone says "Why not?", this lanches a search for justification. But this search will not necessarily hit on the true reason - by "true reason" I mean not the best reason that could be offered, but rather, whichever causes were decisive as a matter of historical fact, at the very first moment the rejection occurred.

Instead, the search for justification hits on the justifying-sounding fact, "This speaker does not have a PhD."

But I also don't have a PhD when I talk about human rationality, so why is the same objection not raised there?

And more to the point, if I had a PhD, people would not treat this as a decisive factor indicating that they ought to believe everything I say. Rather, the same initial rejection would occur, for the same reasons; and the search for justification, afterward, would terminate at a different stopping point.

They would say, "Why should I believe you? You're just some guy with a PhD! There are lots of those. Come back when you're well-known in your field and tenured at a major university."

But do people actually believe arbitrary professors at Harvard who say weird things? Of course not. (But if I were a professor at Harvard, it would in fact be easier to get media attention. Reporters initially disinclined to believe me - who would probably be equally disinclined to believe a random PhD-bearer - would still report on me, because it would be news that a Harvard professor believes such a weird thing.)

If you are saying things that sound wrong to a novice, as opposed to just rattling off magical-sounding technobabble about leptical quark braids in N+2 dimensions; and the hearer is a stranger, unfamiliar with you personally and with the subject matter of your field; then I suspect that the point at which the average person will actually start to grant credence overriding their initial impression, purely because of academic credentials, is somewhere around the Nobel Laureate level. If that. Roughly, you need whatever level of academic credential qualifies as "beyond the mundane".

This is more or less what happened to Eric Drexler, as far as I can tell. He presented his vision of nanotechnology, and people said, "Where are the technical details?" or "Come back when you have a PhD!" And Eric Drexler spent six years writing up technical details and got his PhD under Marvin Minsky for doing it. And Nanosystems is a great book. But did the same people who said, "Come back when you have a PhD", actually change their minds at all about molecular nanotechnology? Not so far as I ever heard.

It has similarly been a general rule with the Singularity Institute that, whatever it is we're supposed to do to be more credible, when we actually do it, nothing much changes. "Do you do any sort of code development? I'm not interested in supporting an organization that doesn't develop code" -> OpenCog -> nothing changes. "Eliezer Yudkowsky lacks academic credentials" -> Professor Ben Goertzel installed as Director of Research -> nothing changes. The one thing that actually has seemed to raise credibility, is famous people associating with the organization, like Peter Thiel funding us, or Ray Kurzweil on the Board.

This might be an important thing for young businesses and new-minted consultants to keep in mind - that what your failed prospects tell you is the reason for rejection, may not make the real difference; and you should ponder that carefully before spending huge efforts. If the venture capitalist says "If only your sales were growing a little faster!", if the potential customer says "It seems good, but you don't have feature X", that may not be the true rejection. Fixing it may, or may not, change anything.

And it would also be something to keep in mind during disagreements. Robin and I share a belief that two rationalists should not agree to disagree: they should not have common knowledge of epistemic disagreement unless something is very wrong.

I suspect that, in general, if two rationalists set out to resolve a disagreement that persisted past the first exchange, they should expect to find that the true sources of the disagreement are either hard to communicate, or hard to expose. E.g:

- Uncommon, but well-supported, scientific knowledge or math;

- Long inferential distances;

- Hard-to-verbalize intuitions, perhaps stemming from specific visualizations;

- Zeitgeists inherited from a profession (that may have good reason for it);

- Patterns perceptually recognized from experience;

- Sheer habits of thought;

- Emotional commitments to believing in a particular outcome;

- Fear of a past mistake being disproven;

- Deep self-deception for the sake of pride or other personal benefits.

If the matter were one in which all the true rejections could be easily laid on the table, the disagreement would probably be so straightforward to resolve that it would never have lasted past the first meeting.

"Is this my true rejection?" is something that both disagreers should surely be asking themselves, to make things easier on the Other Fellow. However, attempts to directly, publicly psychoanalyze the Other may cause the conversation to degenerate very fast, in my observation.

Still - "Is that your true rejection?" should be fair game for Disagreers to humbly ask, if there's any productive way to pursue that sub-issue. Maybe the rule could be that you can openly ask, "Is that simple straightforward-sounding reason your true rejection, or does it come from intuition-X or professional-zeitgeist-Y?" While the more embarrassing possibilities lower on the table are left to the Other's conscience, as their own responsibility to handle.

Post scriptum:

This post is not really about PhDs in general, or their credibility value in particular. But I've always figured that to the extent this was a strategically important consideration, it would make more sense to recruit an academic of existing high status, than spend a huge amount of time trying to achieve low or moderate academic status.

However, if any professor out there wants to let me come in and just do a PhD in analytic philosophy - just write the thesis and defend it - then I have, for my own use, worked out a general and mathematically elegant theory of Newcomblike decision problems. I think it would make a fine PhD thesis, and it is ready to be written - if anyone has the power to let me do things the old-fashioned way.

Sequence: Against Rationalization

Fake Optimization Criteria

Entangled Truths, Contagious Lies

Referenced by: True Sources of Disagreement • Your Price for Joining • Logical Rudeness

Original with comments: Is That Your True Rejection?

Artificial Mysterious Intelligence

Eliezer Yudkowsky, 07 December 2008 08:05PM

Previously in series: Failure By Affective Analogy

I once had a conversation that I still remember for its sheer, purified archetypicality. This was a nontechnical guy, but pieces of this dialog have also appeared in conversations I've had with professional AIfolk...

Him: Oh, you're working on AI! Are you using neural networks?

Me: I think emphatically not.

Him: But neural networks are so wonderful! They solve problems and we don't have any idea how they do it!

Me: If you are ignorant of a phenomenon, that is a fact about your state of mind, not a fact about the phenomenon itself. Therefore your ignorance of how neural networks are solving a specific problem, cannot be responsible for making them work better.

Him: Huh?

Me: If you don't know how your AI works, that is not good. It is bad.

Him: Well, intelligence is much too difficult for us to understand, so we need to find some way to build AI without understanding how it works.

Me: Look, even if you could do that, you wouldn't be able to predict any kind of positive outcome from it. For all you knew, the AI would go out and slaughter orphans.

Him: Maybe we'll build Artificial Intelligence by scanning the brain and building a neuron-by-neuron duplicate. Humans are the only systems we know are intelligent.

Me: It's hard to build a flying machine if the only thing you understand about flight is that somehow birds magically fly. What you need is a concept of aerodynamic lift, so that you can see how something can fly even if it isn't exactly like a bird.

Him: That's too hard. We have to copy something that we know works.

Me: (reflectively) What do people find so unbearably awful about the prospect of having to finally break down and solve the bloody problem? Is it really that horrible?

Him: Wait... you're saying you want to actually understand intelligence?

Me: Yeah.

Him: (aghast) Seriously?

Me: I don't know everything I need to know about intelligence, but I've learned a hell of a lot. Enough to know what happens if I try to build AI while there are still gaps in my understanding.

Him: Understanding the problem is too hard. You'll never do it.

That's not just a difference of opinion you're looking at, it's a clash of cultures.

For a long time, many different parties and factions in AI, adherent to more than one ideology, have been trying to build AI without understanding intelligence. And their habits of thought have become ingrained in the field, and even transmitted to parts of the general public.

You may have heard proposals for building true AI which go something like this:

1. Calculate how many operations the human brain performs every second. This is "the only amount of computing power that we know is actually sufficient for human-equivalent intelligence". Raise enough venture capital to buy a supercomputer that performs an equivalent number of floating-point operations in one second. Use it to run the most advanced available neural network algorithms.

2. The brain is huge and complex. When the Internet becomes sufficiently huge and complex, intelligence is bound to emerge from the Internet. (I get asked about this in 50% of my interviews.)

3. Computers seem unintelligent because they lack common sense. Program a very large number of "common-sense facts" into a computer. Let it try to reason about the relation of these facts. Put a sufficiently huge quantity of knowledge into the machine, and intelligence will emerge from it.

4. Neuroscience continues to advance at a steady rate. Eventually, super-MRI or brain sectioning and scanning will give us precise knowledge of the local characteristics of all human brain areas. So we'll be able to build a duplicate of the human brain by duplicating the parts. "The human brain is the only example we have of intelligence."

5. Natural selection produced the human brain. It is "the only method that we know works for producing general intelligence". So we'll have to scrape up a really huge amount of computing power, and evolve AI.

What do all these proposals have in common?

They are all ways to make yourself believe that you can build an Artificial Intelligence, even if you don't understand exactly how intelligence works.

Now, such a belief is not necessarily false! Methods 4 and 5, if pursued long enough and with enough resources, will eventually work. (5 might require a computer the size of the Moon, but give it enough crunch and it will work, even if you have to simulate a quintillion planets and not just one...)

But regardless of whether any given method would work in principle, the unfortunate habits of thought will already begin to arise, as soon as you start thinking of ways to create Artificial Intelligence without having to penetrate the mystery of intelligence.

I have already spoken of some of the hope-generating tricks that appear in the examples above. There is invoking similarity to humans, or using words that make you feel good. But really, a lot of the trick here just consists of imagining yourself hitting the AI problem with a really big rock.

I know someone who goes around insisting that AI will cost a quadrillion dollars, and as soon as we're willing to spend a quadrillion dollars, we'll have AI, and we couldn't possibly get AI without spending a quadrillion dollars. "Quadrillion dollars" is his big rock, that he imagines hitting the problem with, even though he doesn't quite understand it.

It often will not occur to people that the mystery of intelligence could be any more penetrable than it seems: By the power of the Mind Projection Fallacy, being ignorant of how intelligence works will make it seem like intelligence is inherently impenetrable and chaotic. They will think they possess a positive knowledge of intractability, rather than thinking, "I am ignorant."

And the thing to remember is that, for these last decades on end, any professional in the field of AI trying to build "real AI", had some reason for trying to do it without really understanding intelligence (various fake reductions aside).

The New Connectionists accused the Good-Old-Fashioned AI researchers of not being parallel enough, not being fuzzy enough, not being emergent enough. But they did not say, "There is too much you do not understand."

The New Connectionists catalogued the flaws of GOFAI for years on end, with fiery castigation. But they couldn't ever actually say: "How exactly are all these logical deductions going to produce 'intelligence', anyway? Can you walk me through the cognitive operations, step by step, which lead to that result? Can you explain 'intelligence' and how you plan to get it, without pointing to humans as an example?"

For they themselves would be subject to exactly the same criticism.

In the house of glass, somehow, no one ever gets around to talking about throwing stones.

To tell a lie, you have to lie about all the other facts entangled with that fact, and also lie about the methods used to arrive at beliefs: The culture of Artificial Mysterious Intelligence has developed its own Dark Side Epistemology, complete with reasons why it's actually wrong to try and understand intelligence.

Yet when you step back from the bustle of this moment's history, and think about the long sweep of science - there was a time when stars were mysterious, when chemistry was mysterious, when life was mysterious. And in this era, much was attributed to black-box essences. And there were many hopes based on the similarity of one thing to another. To many, I'm sure, alchemy just seemed very difficult rather than even seeming mysterious; most alchemists probably did not go around thinking, "Look at how much I am disadvantaged by not knowing about the existence of chemistry! I must discover atoms and molecules as soon as possible!" They just memorized libraries of random things you could do with acid, and bemoaned how difficult it was to create the Philosopher's Stone.

In the end, though, what happened is that scientists achieved insight, and then things got much easier to do. You also had a better idea of what you could or couldn't do. The problem stopped being scary and confusing.

But you wouldn't hear a New Connectionist say, "Hey, maybe all the failed promises of 'logical AI' were basically due to the fact that, in their epistemic condition, they had no right to expect their AIs to work in the first place, because they couldn't actually have sketched out the link in any more detail than a medieval alchemist trying to explain why a particular formula for the Philosopher's Stone will yield gold." It would be like the Pope attacking Islam on the basis that faith is not an adequate justification for asserting the existence of their deity.

Yet in fact, the promises did fail, and so we can conclude that the promisers overreached what they had a right to expect. The Way is not omnipotent, and a bounded rationalist cannot do all things. But even a bounded rationalist can aspire not to overpromise - to only say you can do, that which you can do. So if we want to achieve that reliably, history shows that we should not accept certain kinds of hope. In the absence of insight, hopes tend to be unjustified because you lack the knowledge that would be needed to justify them.

We humans have a difficult time working in the absence of insight. It doesn't reduce us all the way down to being as stupid as evolution. But it makes everything difficult and tedious and annoying.

If the prospect of having to finally break down and solve the bloody problem of intelligence seems scary, you underestimate the interminable hell of not solving it.

Referenced by: Nonperson Predicates

Original with comments: Artificial Mysterious Intelligence

True Sources of Disagreement

Eliezer Yudkowsky, 08 December 2008 03:51PM

Followup to: Is That Your True Rejection?

I expected from the beginning, that the difficult part of two rationalists reconciling a persistent disagreement, would be for them to expose the true sources of their beliefs.

One suspects that this will only work if each party takes responsibility for their own end; it's very hard to see inside someone else's head. Yesterday I exhausted myself mentally while out on my daily walk, asking myself the Question "What do you think you know, and why do you think you know it?" with respect to "How much of the AI problem compresses to large insights, and how much of it is unavoidable nitty-gritty?" Trying to either understand why my brain believed what it believed, or else force my brain to experience enough genuine doubt that I could reconsider the question and arrive at a real justification that way. It's hard to see how Robin Hanson could have done any of this work for me.

Presumably a symmetrical fact holds about my lack of access to the real reasons why Robin believes what he believes. To understand the true source of a disagreement, you have to know why both sides believe what they believe - one reason why disagreements are hard to resolve.

Nonetheless, here's my guess as to what this Disagreement is about:

If I had to pinpoint a single thing that strikes me as "disagree-able" about the way Robin frames his analyses, it's that there are a lot of opaque agents running around, little black boxes assumed to be similar to humans, but there are more of them and they're less expensive to build/teach/run. They aren't even any faster, let alone smarter. (I don't think that standard economics says that doubling the population halves the doubling time, so it matters whether you're making more minds or faster ones.)

This is Robin's model for uploads/ems, and his model for AIs doesn't seem to look any different. So that world looks like this one, except that the cost of "human capital" and labor is dropping according to (exogenous) Moore's Law , and it ends up that economic growth doubles every month instead of every sixteen years - but that's it. Being, myself, not an economist, this does look to me like a viewpoint with a distinctly economic zeitgeist.

In my world, you look inside the black box. (And, to be symmetrical, I don't spend much time thinking about more than one box at a time - if I have more hardware, it means I have to figure out how to scale a bigger brain.)

The human brain is a haphazard thing, thrown together by idiot evolution, as an incremental layer of icing on a chimpanzee cake that never evolved to be generally intelligent, adapted in a distant world devoid of elaborate scientific arguments or computer programs or professional specializations.

It's amazing we can get anywhere using the damn thing. But it's worth remembering that if there were any smaller modification of a chimpanzee that spontaneously gave rise to a technological civilization, we would be having this conversation at that lower level of intelligence instead.

Human neurons run at less than a millionth the speed of transistors, transmit spikes at less than a millionth the speed of light, and dissipate around a million times the heat per synaptic operation as the thermodynamic minimum for a one-bit operation at room temperature. Physically speaking, it ought to be possible to run a brain at a million times the speed without shrinking it, cooling it, or invoking reversible computing or quantum computing.

There's no reason to think that the brain's software is any closer to the limits of the possible than its hardware, and indeed, if you've been following along on Overcoming Bias this whole time, you should be well aware of the manifold known ways in which our high-level thought processes fumble even the simplest problems.

Most of these are not deep, inherent flaws of intelligence, or limits of what you can do with a mere hundred trillion computing elements. They are the results of a really stupid process that designed the retina backward, slapping together a brain we now use in contexts way outside its ancestral environment.

Ten thousand researchers working for one year cannot do the same work as a hundred researchers working for a hundred years; a chimpanzee is one-fourth the volume of a human's but four chimps do not equal one human; a chimpanzee shares 95% of our DNA but a chimpanzee cannot understand 95% of what a human can. The scaling law for population is not the scaling law for time is not the scaling law for brain size is not the scaling law for mind design.

There's a parable I sometimes use, about how the first replicator was not quite the end of the era of stable accidents, because the pattern of the first replicator was, of necessity, something that could happen by accident. It is only the second replicating pattern that you would never have seen without many copies of the first replicator around to give birth to it; only the second replicator that was part of the world of evolution, something you wouldn't see in a world of accidents.

That first replicator must have looked like one of the most bizarre things in the whole history of time - this replicator created purely by chance. But the history of time could never have been set in motion, otherwise.

And what a bizarre thing a human must be, a mind born entirely of evolution, a mind that was not created by another mind.

We haven't yet begun to see the shape of the era of intelligence.

Most of the universe is far more extreme than this gentle place, Earth's cradle. Cold vacuum or the interior of stars; either is far more common than the temperate weather of Earth's surface, where life first arose, in the balance between the extremes. And most possible intelligences are not balanced, like these first humans, in that strange small region of temperate weather between an amoeba and a Jupiter Brain.

This is the challenge of my own profession - to break yourself loose of the tiny human dot in mind design space, in which we have lived our whole lives, our imaginations lulled to sleep by too-narrow experiences.

For example, Robin says:

Eliezer guesses that within a few weeks a single AI could grow via largely internal means from weak and unnoticed to so strong it takes over the world [his italics]

I suppose that to a human a "week" sounds like a temporal constant describing a "short period of time", but it's actually 10^49 Planck intervals, or enough time for a population of 2GHz processor cores to perform 10^15 serial operations one after the other.

Perhaps the thesis would sound less shocking if Robin had said, "Eliezer guesses that 10^15 sequential operations might be enough to..."

One should also bear in mind that the human brain, which is not designed for the primary purpose of scientific insights, does not spend its power efficiently on having many insights in minimum time, but this issue is harder to understand than CPU clock speeds.

Robin says he doesn't like "unvetted abstractions". Okay. That's a strong point. I get it. Unvetted abstractions go kerplooie, yes they do indeed. But something's wrong with using that as a justification for models where there are lots of little black boxes just like humans scurrying around, and we never pry open the black box and scale the brain bigger or redesign its software or even just speed up the damn thing. The interesting part of the problem is harder to analyze, yes - more distant from the safety rails of overwhelming evidence - but this is no excuse for refusing to take it into account.

And in truth I do suspect that a strict policy against "unvetted abstractions" is not the real issue here. I constructed a simple model of an upload civilization running on the computers their economy creates: If a non-upload civilization has an exponential Moore's Law, y = e^t, then, naively, an upload civilization ought to have dy/dt = e^y -> y = -ln(C - t). Not necessarily up to infinity, but for as long as Moore's Law would otherwise stay exponential in a biological civilization. I walked though the implications of this model, showing that in many senses it behaves "just like we would expect" for describing a civilization running on its own computers.

Compare this to Robin Hanson's "Economic Growth Given Machine Intelligence", which Robin describes as using "one of the simplest endogenous growth models to explore how Moore's Law changes with computer-based workers. It is an early but crude attempt, but it is the sort of approach I think promising." Take a quick look at that paper.

Now, consider the abstractions used in my Moore's Researchers scenario, versus the abstractions used in Hanson's paper above, and ask yourself only the question of which looks more "vetted by experience" - given that both are models of a sort that haven't been used before, in domains not actually observed, and that both give results quite different from the world we see and that would probably cause the vast majority of actual economists to say "Naaaah."

Moore's Researchers versus Economic Growth Given Machine Intelligence - if you didn't think about the conclusions in advance of the reasoning; and if you also neglected that one of these has been written up in a way that is more impressive to economics journals; and you just asked the question, "To what extent is the math used here, constrained by our prior experience?" then I would think that the race would at best be even. Or possibly favoring "Moore's Researchers" as being more simple and intuitive, and involving less novel math as measured in additional quantities and laws introduced.

I ask in all humility if Robin's true rejection is a strictly evenhandedly applied rule that rejects unvetted abstractions. Or if, in fact, Robin finds my conclusions, and the sort of premises I use, to be objectionable for other reasons - which, so far as we know at this point, may well be valid objections - and so it appears to him that my abstractions bear a larger burden of proof than the sort of mathematical steps he takes in "Economic Growth Given Machine Intelligence". But rather than offering the reasons why the burden of proof appears larger to him, he says instead that it is "not vetted enough".

One should understand that "Your abstractions are unvetted!" makes it difficult for me to engage properly. The core of my argument has to do with what happens when you pry open the black boxes that are your economic agents, and start fiddling with their brain designs, and leave the tiny human dot in mind design space. If all such possibilities are rejected on the basis of their being "unvetted" by experience, it doesn't leave me with much to talk about.

Why not just accept the rejection? Because I expect that to give the wrong answer - I expect it to ignore the dominating factor in the Future, even if the dominating factor is harder to analyze.

It shouldn't be surprising if a persistent disagreement ends up resting on that point where your attempt to take into account the other person's view, runs up against some question of simple fact where, it seems to you, you know that can't possibly be right.

For me, that point is reached when trying to visualize a model of interacting black boxes that behave like humans except they're cheaper to make. The world, which shattered once with the with the first replicator, and shattered for the second time with the emergence of human intelligence, somehow does not shatter a third time. Even in the face of blowups of brain size far greater than the size transition from chimpanzee brain to human brain; and changes in design far larger than the design transition from chimpanzee brains to human brains; and simple serial thinking speeds that are, maybe even right from the beginning, thousands or millions of times faster.

That's the point where I, having spent my career trying to look inside the black box, trying to wrap my tiny brain around the rest of mind design space that isn't like our small region of temperate weather, just can't make myself believe that the Robin-world is really truly actually the way the future will be.

There are other things that seem like probable nodes of disagreement:

Robin Hanson's description of Friendly AI development as "total war" that is harmful to even discuss, or his description of a realized Friendly AI as "a God to rule us all". Robin must be visualizing an in-practice outcome very different from what I do, and this seems like a likely source of emotional fuel for the disagreement as well.

Conversely, Robin Hanson seems to approve of a scenario where lots of AIs, of arbitrary motives, constitute the vast part of the economic productivity of the Solar System, because he thinks that humans will be protected under the legacy legal system that grew continuously out of the modern world, and that the AIs will be unable to coordinate to transgress the legacy legal system for fear of losing their own legal protections. I tend to visualize a somewhat different outcome, to put it mildly; and would symmetrically be suspected of emotional unwillingness to accept that outcome as inexorable.

Robin doesn't dismiss Cyc out of hand and even "hearts" it, which implies that we have an extremely different picture of how intelligence works.

Like Robin, I'm also feeling burned on this conversation, and I doubt we'll finish it; but I should write at least two more posts to try to describe what I've learned, and some of the rules that I think I've been following.

Original with comments: True Sources of Disagreement

Disjunctions, Antipredictions, Etc.

Eliezer Yudkowsky, 09 December 2008 03:13PM

Followup to: Underconstrained Abstractions

Previously:

So if it's not as simple as just using the one trick of finding abstractions you can easily verify on available data, what are some other tricks to use?

There are several, as you might expect...

Previously I talked about "permitted possibilities". There's a trick in debiasing that has mixed benefits, which is to try and visualize several specific possibilities instead of just one.

The reason it has "mixed benefits" is that being specific, at all, can have biasing effects relative to just imagining a typical case. (And believe me, if I'd seen the outcome of a hundred planets in roughly our situation, I'd be talking about that instead of all this Weak Inside View stuff.)

But if you're going to bother visualizing the future, it does seem to help to visualize more than one way it could go, instead of concentrating all your strength into one prediction.

So I try not to ask myself "What will happen?" but rather "Is this possibility allowed to happen, or is it prohibited?" There are propositions that seem forced to me, but those should be relatively rare - the first thing to understand about the future is that it is hard to predict, and you shouldn't seem to be getting strong information about most aspects of it.

Of course, if you allow more than one possibility, then you have to discuss more than one possibility, and the total length of your post gets longer. If you just eyeball the length of the post, it looks like an unsimple theory; and then talking about multiple possibilities makes you sound weak and uncertain.

As Robyn Dawes notes,

"In their summations lawyers avoid arguing from disjunctions in favor of conjunctions. (There are not many closing arguments that end, "Either the defendant was in severe financial straits and murdered the decedent to prevent his embezzlement from being exposed or he was passionately in love with the same coworker and murdered the decedent in a fit of jealous rage or the decedent had blocked the defendant's promotion at work and the murder was an act of revenge. The State has given you solid evidence to support each of these alternatives, all of which would lead to the same conclusion: first-degree murder.") Rationally, of course, disjunctions are much more probable than are conjunctions."

Another test I use is simplifiability - after I've analyzed out the idea, can I compress it back into an argument that fits on a T-Shirt, even if it loses something thereby? Here's an example of some compressions:

- The whole notion of recursion and feeding object-level improvements back into meta-level improvements: "If computing power per dollar doubles every eighteen months, what happens if computers are doing the research?"

- No diminishing returns on complexity in the region of the transition to human intelligence: "We're so similar to chimps in brain design, and yet so much more powerful; the upward slope must be really steep."

- Scalability of hardware: "Humans have only four times the brain volume of chimps - now imagine an AI suddenly acquiring a thousand times as much power."

If the whole argument was that T-Shirt slogan, I wouldn't find it compelling - too simple and surface a metaphor. So you have to look more closely, and try visualizing some details, and make sure the argument can be consistently realized so far as you know. But if, after you do that, you can compress the argument back to fit on a T-Shirt again - even if it sounds naive and stupid in that form - then that helps show that the argument doesn't depend on all the details being true simultaneously; the details might be different while fleshing out the same core idea.

Note also that the three statements above are to some extent disjunctive - you can imagine only one of them being true, but a hard takeoff still occurring for just that reason alone.

Another trick I use is the idea of antiprediction. This is when the narrowness of our human experience distorts our metric on the answer space, and so you can make predictions that actually aren't far from maxentropy priors, but sound very startling.

I shall explain:

A news story about an Australian national lottery that was just starting up, interviewed a man on the street, asking him if he would play. He said yes. Then they asked him what he thought his odds were of winning. "Fifty-fifty," he said, "either I win or I don't."

To predict your odds of winning the lottery, you should invoke the Principle of Indifference with respect to all possible combinations of lottery balls. But this man was invoking the Principle of Indifference with respect to the partition "win" and "not win". To him, they sounded like equally simple descriptions; but the former partition contains only one combination, and the latter contains the other N million combinations. (If you don't agree with this analysis I'd like to sell you some lottery tickets.)

So the antiprediction is just "You won't win the lottery." And the one may say, "What? How do you know that? You have no evidence for that! You can't prove that I won't win!" So they are focusing far too much attention on a small volume of the answer space, artificially inflated by the way their attention dwells upon it.

In the same sense, if you look at a television SF show, you see that a remarkable number of aliens seem to have human body plans - two arms, two legs, walking upright, right down to five fingers per hand and the location of eyes in the face. But this is a very narrow partition in the body-plan space; and if you just said, "They won't look like humans," that would be an antiprediction that just steps outside this artificially inflated tiny volume in the answer space.

Similarly with the true sin of television SF, which is too-human minds, even among aliens not meant to be sympathetic characters. "If we meet aliens, they won't have a sense of humor," I antipredict; and to a human it sounds like I'm saying something highly specific, because all minds by default have a sense of humor, and I'm predicting the presence of a no-humor attribute tagged on. But actually, I'm just predicting that a point in mind design volume is outside the narrow hyperplane that contains humor.

An AI might go from infrahuman to transhuman in less than a week? But a week is 10^49 Planck intervals - if you just look at the exponential scale that stretches from the Planck time to the age of the universe, there's nothing special about the timescale that 200Hz humans happen to live on, any more than there's something special about the numbers on the lottery ticket you bought.

If we're talking about a starting population of 2GHz processor cores, then any given AI that FOOMs at all, is likely to FOOM in less than 10^15 sequential operations or more than 10^19 sequential operations, because the region between 10^15 and 10^19 isn't all that wide a target. So less than a week or more than a century, and in the latter case that AI will be trumped by one of a shorter timescale.

This is actually a pretty naive version of the timescale story. But as an example, it shows how a "prediction" that's close to just stating a maximum-entropy prior, can sound amazing, startling, counterintuitive, and futuristic.

When I make an antiprediction supported by disjunctive arguments that are individually simplifiable, I feel slightly less nervous about departing the rails of vetted abstractions. (In particular, I regard this as sufficient reason not to trust the results of generalizations over only human experiences.)

Finally, there are three tests I apply to figure out how strong my predictions are.

The first test is to just ask myself the Question, "What do you think you know, and why do you think you know it?" The future is something I haven't yet observed; if my brain claims to know something about it with any degree of confidence, what are the reasons for that? The first test tries to align the strength of my predictions with things that I have reasons to believe - a basic step, but one which brains are surprisingly wont to skip.

The second test is to ask myself "How worried do I feel that I'll have to write an excuse explaining why this happened anyway?" If I don't feel worried about having to write an excuse - if I can stick my neck out and not feel too concerned about ending up with egg on my face - then clearly my brain really does believe this thing quite strongly, not as a point to be professed through enthusiastic argument, but as an ordinary sort of fact. Why?

And the third test is the "So what?" test - to what degree will I feel indignant if Nature comes back and says "So what?" to my clever analysis? Would I feel as indignant as if I woke up one morning to read in the newspaper that Mars had started orbiting the Sun in squares instead of ellipses? Or, to make it somewhat less strong, as if I woke up one morning to find that banks were charging negative interest on loans? If so, clearly I must possess some kind of extremely strong argument - one that even Nature Itself ought to find compelling, not just humans. What is it?

Original with comments: Disjunctions, Antipredictions, Etc.

Bay Area Meetup Wed 12/10 @8pm

Eliezer Yudkowsky, 10 December 2008 12:13AM

Reminder: the second regular Overcoming Bias meetup is tomorrow, Wednesday at 8pm, at Techshop in Menlo park. Please RSVP so they know how many people are coming.

If you're hearing about this for the first time, sign up for the Meetup group so you get future announcements! And please RSVP when you get them! We don't want to have to post this to the main blog every time.

Robin Gane-McCalla will present some of his ideas on defining intelligence. As always, anyone with a paper to pass around, abstract to read out loud, or rationality-related cool video to show, is cordially invited to bring it along.

Original with comments: Bay Area Meetup Wed 12/10 @8pm

The Mechanics of Disagreement

Eliezer Yudkowsky, 10 December 2008 02:01PM

Two ideal Bayesians cannot have common knowledge of disagreement; this is a theorem. If two rationalist-wannabes have common knowledge of a disagreement between them, what could be going wrong?

The obvious interpretation of these theorems is that if you know that a cognitive machine is a rational processor of evidence, its beliefs become evidence themselves.

If you design an AI and the AI says "This fair coin came up heads with 80% probability", then you know that the AI has accumulated evidence with an likelihood ratio of 4:1 favoring heads - because the AI only emits that statement under those circumstances.

It's not a matter of charity; it's just that this is how you think the other cognitive machine works.

And if you tell an ideal rationalist, "I think this fair coin came up heads with 80% probability", and they reply, "I now think this fair coin came up heads with 25% probability", and your sources of evidence are independent of each other, then you should accept this verdict, reasoning that (before you spoke) the other mind must have encountered evidence with a likelihood of 1:12 favoring tails.

But this assumes that the other mind also thinks that you're processing evidence correctly, so that, by the time it says "I now think this fair coin came up heads, p=.25", it has already taken into account the full impact of all the evidence you know about, before adding more evidence of its own.

If, on the other hand, the other mind doesn't trust your rationality, then it won't accept your evidence at face value, and the estimate that it gives won't integrate the full impact of the evidence you observed.

So does this mean that when two rationalists trust each other's rationality less than completely, then they can agree to disagree?

It's not that simple. Rationalists should not trust themselves entirely, either.

So when the other mind accepts your evidence at less than face value, this doesn't say "You are less than a perfect rationalist", it says, "I trust you less than you trust yourself; I think that you are discounting your own evidence too little."

Maybe your raw arguments seemed to you to have a strength of 40:1, but you discounted for your own irrationality to a strength of 4:1, but the other mind thinks you still overestimate yourself and so it assumes that the actual force of the argument was 2:1.

And if you believe that the other mind is discounting you in this way, and is unjustified in doing so, then when it says "I now think this fair coin came up heads with 25% probability", you might bet on the coin at odds of 57% in favor of heads - adding up your further-discounted evidence of 2:1 to the implied evidence of 1:6 that the other mind must have seen to give final odds of 2:6 - if you even fully trust the other mind's further evidence of 1:6.

I think we have to be very careful to avoid interpreting this situation in terms of anything like a reciprocal trade, like two sides making equal concessions in order to reach agreement on a business deal.

Shifting beliefs is not a concession that you make for the sake of others, expecting something in return; it is an advantage you take for your own benefit, to improve your own map of the world. I am, generally speaking, a Millie-style altruist; but when it comes to belief shifts I espouse a pure and principled selfishness: don't believe you're doing it for anyone's sake but your own.

Still, I once read that there's a principle among con artists that the main thing is to get the mark to believe that you trust them, so that they'll feel obligated to trust you in turn.

And - even if it's for completely different theoretical reasons - if you want to persuade a rationalist to shift belief to match yours, you either need to persuade them that you have all of the same evidence they do and have already taken it into account, or that you already fully trust their opinions as evidence, or that you know better than they do how much they themselves can be trusted.

It's that last one that's the really sticky point, for obvious reasons of asymmetry of introspective access and asymmetry of motives for overconfidence - how do you resolve that conflict? (And if you started arguing about it, then the question wouldn't be which of these were more important as a factor, but rather, which of these factors the Other had under- or over-discounted in forming their estimate of a given person's rationality...)

If I had to name a single reason why two wannabe rationalists wouldn't actually be able to agree in practice, it would be that, once you trace the argument to the meta-level where theoretically everything can be and must be resolved, the argument trails off into psychoanalysis and noise.

And if you look at what goes on in practice between two arguing rationalists, it would probably mostly be trading object-level arguments; and the most meta it would get is trying to convince the other person that you've already taken their object-level arguments into account.

Still, this does leave us with three clear reasons that someone might point to, to justify a persistent disagreement - even though the frame of mind of justification and having clear reasons to point to in front of others, is itself antithetical to the spirit of resolving disagreements - but even so:

- Clearly, the Other's object-level arguments are flawed; no amount of trust that I can have for another person will make me believe that rocks fall upward.

- Clearly, the Other is not taking my arguments into account; there's an obvious asymmetry in how well I understand them and have integrated their evidence, versus how much they understand me and have integrated mine.

- Clearly, the Other is completely biased in how much they trust themselves over others, versus how I humbly and evenhandedly discount my own beliefs alongside theirs.

Since we don't want to go around encouraging disagreement, one might do well to ponder how all three of these arguments are used by creationists to justify their persistent disagreements with scientists.

That's one reason I say clearly - if it isn't obvious even to outside onlookers, maybe you shouldn't be confident of resolving the disagreement there. Failure at any of these levels implies failure at the meta-levels above it, but the higher-order failures might not be clear.

Original with comments: The Mechanics of Disagreement

What I Think, If Not Why

Eliezer Yudkowsky, 11 December 2008 05:41PM

Reply to: Two Visions Of Heritage

Though it really goes tremendously against my grain - it feels like sticking my neck out over a cliff (or something) - I guess I have no choice here but to try and make a list of just my positions, without justifying them. We can only talk justification, I guess, after we get straight what my positions are. I will also leave off many disclaimers to present the points compactly enough to be remembered.

• A well-designed mind should be much more efficient than a human, capable of doing more with less sensory data and fewer computing operations. It is not infinitely efficient and does not use zero data. But it does use little enough that local pipelines such as a small pool of programmer-teachers and, later, a huge pool of e-data, are sufficient.

• An AI that reaches a certain point in its own development becomes able to (sustainably, strongly) improve itself. At this point, recursive cascades slam over many internal growth curves to near the limits of their current hardware, and the AI undergoes a vast increase in capability. This point is at, or probably considerably before, a minimally transhuman mind capable of writing its own AI-theory textbooks - an upper bound beyond which it could swallow and improve its entire design chain.

• It is likely that this capability increase or "FOOM" has an intrinsic maximum velocity that a human would regard as "fast" if it happens at all. A human week is ~1e15 serial operations for a population of 2GHz cores, and a century is ~1e19 serial operations; this whole range is a narrow window. However,the core argument does not require one-week speed and a FOOM that takes two years (~1e17 serial ops) will still carry the weight of the argument.

• The default case of FOOM is an unFriendly AI, built by researchers with shallow insights. This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever).

• The desired case of FOOM is a Friendly AI, built using deep insight, so that the AI never makes any changes to itself that potentially change its internal values; all such changes are guaranteed using strong techniques that allow for a billion sequential self-modifications without losing the guarantee. The guarantee is written over the AI's internal search criterion for actions, rather than external consequences.

• The good guys do not write an AI which values a bag of things that the programmers think are good ideas, like libertarianism or socialism or making people happy or whatever. There were multiple Overcoming Bias sequences about this one point, like the Fake Utility Function sequence and the sequence on metaethics. It is dealt with at length in the document Coherent *Extrapolated* Volition.  It is the first thing, the last thing, and the middle thing that I say about Friendly AI. I have said it over and over. I truly do not understand how anyone can pay any attention to anything I have said on this subject, and come away with the impression that I think programmers are supposed to directly impress their non-meta personal philosophies onto a Friendly AI.

• The good guys do not directly impress their personal values onto a Friendly AI.

• Actually setting up a Friendly AI's values is an extremely meta operation, less "make the AI want to make people happy" and more like "superpose the possible reflective equilibria of the whole human species, and output new code that overwrites the current AI and has the most coherent support within that superposition". This actually seems to be something of a Pons Asinorum in FAI - the ability to understand and endorse metaethical concepts that do not directly sound like amazing wonderful happy ideas. Describing this as declaring total war on the rest of humanity, does not seem fair (or accurate).

• I myself am strongly individualistic: The most painful memories in my life have been when other people thought they knew better than me, and tried to do things on my behalf. It is also a known principle of hedonic psychology that people are happier when they're steering their own lives and doing their own interesting work. When I try myself to visualize what a beneficial superintelligence ought to do, it consists of setting up a world that works by better rules, and then fading into the background, silent as the laws of Nature once were; and finally folding up and vanishing when it is no longer needed. But this is only the thought of my mind that is merely human, and I am barred from programming any such consideration directly into a Friendly AI, for the reasons given above.

• Nonetheless, it does seem to me that this particular scenario could not be justly described as "a God to rule over us all", unless the current fact that humans age and die is "a malevolent God to rule us all". So either Robin has a very different idea about what human reflective equilibrium values are likely to look like; or Robin believes that the Friendly AI project is bound to fail in such way as to create a paternalistic God; or - and this seems more likely to me - Robin didn't read all the way through all the blog posts in which I tried to explain all the ways that this is not how Friendly AI works.

• Friendly AI is technically difficult and requires an extra-ordinary effort on multiple levels. English sentences like "make people happy" cannot describe the values of a Friendly AI. Testing is not sufficient to guarantee that values have been successfully transmitted.

• White-hat AI researchers are distinguished by the degree to which they understand that a single misstep could be fatal, and can discriminate strong and weak assurances. Good intentions are not only common, they're cheap. The story isn't about good versus evil, it's about people trying to do the impossible versus others who... aren't.

• Intelligence is about being able to learn lots of things, not about knowing lots of things. Intelligence is especially not about tape-recording lots of parsed English sentences a la Cyc. Old AI work was poorly focused due to inability to introspectively see the first and higher derivatives of knowledge; human beings have an easier time reciting sentences than reciting their ability to learn.

• Intelligence is mostly about architecture, or "knowledge" along the lines of knowing to look for causal structure (Bayes-net type stuff) in the environment; this kind of knowledge will usually be expressed procedurally as well as declaratively. Architecture is mostly about deep insights. This point has not yet been addressed (much) on Overcoming Bias, but Bayes nets can be considered as an archetypal example of "architecture" and "deep insight". Also, ask yourself how lawful intelligence seemed to you before you started reading this blog, how lawful it seems to you now, then extrapolate outward from that.

Referenced by: Not Taking Over the World • Imaginary Positions • Devil's Offers

Original with comments: What I Think, If Not Why

You Only Live Twice

Eliezer Yudkowsky, 12 December 2008 07:14PM

"It just so happens that your friend here is only mostly dead. There's a big difference between mostly dead and all dead."
-- The Princess Bride

My co-blogger Robin and I may disagree on how fast an AI can improve itself, but we agree on an issue that seems much simpler to us than that: At the point where the current legal and medical system gives up on a patient, they aren't really dead.

Robin has already said much of what needs saying, but a few more points:

• Ben Best's Cryonics FAQ, Alcor's FAQ, Alcor FAQ for scientists, Scientists' Open Letter on Cryonics

• I know more people who are planning to sign up for cryonics Real Soon Now than people who have actually signed up. I expect that more people have died while cryocrastinating than have actually been cryopreserved. If you've already decided this is a good idea, but you "haven't gotten around to it", sign up for cryonics NOW. I mean RIGHT NOW. Go to the website of Alcor or the Cryonics Institute and follow the instructions.

• Cryonics is usually funded through life insurance. The following conversation from an Overcoming Bias meetup is worth quoting:

Him: I've been thinking about signing up for cryonics when I've got enough money.

Me: Um... it doesn't take all that much money.

Him: It doesn't?

Me: Alcor is the high-priced high-quality organization, which is something like $500-$1000 in annual fees for the organization, I'm not sure how much. I'm young, so I'm signed up with the Cryonics Institute, which is $120/year for the membership. I pay $180/year for more insurance than I need - it'd be enough for Alcor too.

Him: That's ridiculous.

Me: Yes.

Him: No, really, that's ridiculous. If that's true then my decision isn't just determined, it's overdetermined.

Me: Yes. And there's around a thousand people worldwide [actually 1400] who are signed up for cryonics. Figure that at most a quarter of those did it for systematically rational reasons. That's a high upper bound on the number of people on Earth who can reliably reach the right conclusion on massively overdetermined issues.

• Cryonics is not marketed well - or at all, really. There's no salespeople who get commissions. There is no one to hold your hand through signing up, so you're going to have to get the papers signed and notarized yourself. The closest thing out there might be Rudi Hoffman, who sells life insurance with cryonics-friendly insurance providers (I went through him).

• If you want to securely erase a hard drive, it's not as easy as writing it over with zeroes. Sure, an "erased" hard drive like this won't boot up your computer if you just plug it in again. But if the drive falls into the hands of a specialist with a scanning tunneling microscope, they can tell the difference between "this was a 0, overwritten by a 0" and "this was a 1, overwritten by a 0".

There are programs advertised to "securely erase" hard drives using many overwrites of 0s, 1s, and random data. But if you want to keep the secret on your hard drive secure against all possible future technologies that might ever be developed, then cover it with thermite and set it on fire. It's the only way to be sure.

Pumping someone full of cryoprotectant and gradually lowering their temperature until they can be stored in liquid nitrogen is not a secure way to erase a person.

See also the information-theoretic criterion of death.

• You don't have to buy what's usually called the "patternist" philosophy of identity, to sign up for cryonics. After reading all the information off the brain, you could put the "same atoms" back into their old places.

• "Same atoms" is in scare quotes because our current physics prohibits particles from possessing individual identities. It's a much stronger statement than "we can't tell the particles apart with current measurements" and has to do with the notion of configuration spaces in quantum mechanics. This is a standard idea in QM, not an unusual woo-woo one - see this sequence on Overcoming Bias for a gentle introduction. Although patternism is not necessary to the cryonics thesis, we happen to live in a universe where "the same atoms" is physical nonsense.

There's a number of intuitions we have in our brains for processing a world of distinct physical objects, built in from a very young age. These intuitions, which may say things like "If an object disappears, and then comes back, it isn't the same object", are tuned to our macroscopic world and generally don't match up well with fundamental physics. Your identity is not like a little billiard ball that follows you around - there aren't actually any billiard balls down there.

Separately and convergently, more abstract reasoning strongly suggests that "identity" should not be epiphenomenal; that is, you should not be able to change someone's identity without changing any observable fact about them.

If you go through the aforementioned Overcoming Bias sequence, you should actually be able to see intuitively that successful cryonics preserves anything about you that is preserved by going to sleep at night and waking up the next morning.

• Cryonics, to me, makes two statements.

The first statement is about systematically valuing human life. It's bad when a pretty young white girl goes missing somewhere in America. But when 800,000 Africans get murdered in Rwanda, that gets 1/134 the media coverage of the Michael Jackson trial. It's sad, to be sure, but no cause for emotional alarm. When brown people die, that's all part of the plan - as a smiling man once said.

Cryonicists are people who've decided that their deaths, and the deaths of their friends and family and the rest of the human species, are not part of the plan.

I've met one or two Randian-type "selfish" cryonicists, but they aren't a majority. Most people who sign up for cryonics wish that everyone would sign up for cryonics.

The second statement is that you have at least a little hope in the future. Not faith, not blind hope, not irrational hope - just, any hope at all.

I was once at a table with Ralph Merkle, talking about how to market cryonics if anyone ever gets around to marketing it, and Ralph suggested a group of people in a restaurant, having a party; and the camera pulls back, and moves outside the window, and the restaurant is on the Moon. Tagline: "Wouldn't you want to be there?"

If you look back at, say, the Middle Ages, things were worse then. I'd rather live here then there. I have hope that humanity will move forward further, and that's something that I want to see.

And I hope that the idea that people are disposable, and that their deaths are part of the plan, is something that fades out of the Future.

Once upon a time, infant deaths were part of the plan, and now they're not. Once upon a time, slavery was part of the plan, and now it's not. Once upon a time, dying at thirty was part of the plan, and now it's not. That's a psychological shift, not just an increase in living standards. Our era doesn't value human life with perfect consistency - but the value of human life is higher than it once was.

We have a concept of what a medieval peasant should have had, the dignity with which they should have been treated, that is higher than what they would have thought to ask for themselves.

If no one in the future cares enough to save people who can be saved... well. In cryonics there is an element of taking responsibility for the Future. You may be around to reap what your era has sown. It is not just my hope that the Future be a better place; it is my responsibility. If I thought that we were on track to a Future where no one cares about human life, and lives that could easily be saved are just thrown away - then I would try to change that. Not everything worth doing is easy.

Not signing up for cryonics - what does that say? That you've lost hope in the future. That you've lost your will to live. That you've stopped believing that human life, and your own life, is something of value.

This can be a painful world we live in, and the media is always telling us how much worse it will get. If you spend enough time not looking forward to the next day, it damages you, after a while. You lose your ability to hope. Try telling someone already grown old to sign up for cryonics, and they'll tell you that they don't want to be old forever - that they're tired. If you try to explain to someone already grown old, that the nanotechnology to revive a cryonics patient is sufficiently advanced that reversing aging is almost trivial by comparison... then it's not something they can imagine on an emotional level, no matter what they believe or don't believe about future technology. They can't imagine not being tired. I think that's true of a lot of people in this world. If you've been hurt enough, you can no longer imagine healing.

But things really were a lot worse in the Middle Ages. And they really are a lot better now. Maybe humanity isn't doomed. The Future could be something that's worth seeing, worth living in. And it may have a concept of sentient dignity that values your life more than you dare to value yourself.

On behalf of the Future, then - please ask for a little more for yourself. More than death. It really... isn't being selfish. I want you to live. I think that the Future will want you to live. That if you let yourself die, people who aren't even born yet will be sad for the irreplaceable thing that was lost.

So please, live.

My brother didn't. My grandparents won't. But everything we can hold back from the Reaper, even a single life, is precious.

If other people want you to live, then it's not just you doing something selfish and unforgivable, right?

So I'm saying it to you.

I want you to live.

Referenced by: For The People Who Are Still Alive • Complex Novelty • The Uses of Fun (Theory) • The Thing That I Protect • Incremental Progress and the Valley • This Didn't Have To Happen

Original with comments: You Only Live Twice

BHTV: de Grey and Yudkowsky

Eliezer Yudkowsky, 13 December 2008 03:28PM

My latest on Bloggingheads.tv is up. BHTV wanted someone to interview Aubrey de Grey of the Methuselah Foundation about basic research in antiagathics, and they picked me to do it. It made the interview somewhat difficult, since Aubrey and I already agree about most things, but we managed to soldier on. The interview is mostly Aubrey talking, as it should be.

Original with comments: BHTV: de Grey and Yudkowsky

For The People Who Are Still Alive

Eliezer Yudkowsky, 14 December 2008 05:13PM

Max Tegmark observed that we have three independent reasons to believe we live in a Big World: A universe which is large relative to the space of possibilities. For example, on current physics, the universe appears to be spatially infinite (though I'm not clear on how strongly this is implied by the standard model).

If the universe is spatially infinite, then, on average, we should expect that no more than 10^10^29 meters away is an exact duplicate of you. If you're looking for an exact duplicate of a Hubble volume - an object the size of our observable universe - then you should still on average only need to look 10^10^115 lightyears. (These are numbers based on a highly conservative counting of "physically possible" states, e.g. packing the whole Hubble volume with potential protons at maximum density given by the Pauli Exclusion principle, and then allowing each proton to be present or absent.)

The most popular cosmological theories also call for an "inflationary" scenario in which many different universes would be eternally budding off, our own universe being only one bud. And finally there are the alternative decoherent branches of the grand quantum distribution, aka "many worlds", whose presence is unambiguously implied by the simplest mathematics that fits our quantum experiments.

Ever since I realized that physics seems to tell us straight out that we live in a Big World, I've become much less focused on creating lots of people, and much more focused on ensuring the welfare of people who are already alive.

If your decision to not create a person means that person will never exist at all, then you might, indeed, be moved to create them, for their sakes. But if you're just deciding whether or not to create a new person here, in your own Hubble volume and Everett branch, then it may make sense to have relatively lower populations within each causal volume, living higher qualities of life. It's not like anyone will actually fail to be born on account of that decision - they'll just be born predominantly into regions with higher standards of living.

Am I sure that this statement, that I have just emitted, actually makes sense?

Not really. It dabbles in the dark arts of anthropics, and the Dark Arts don't get much murkier than that. Or to say it without the chaotic inversion: I am stupid with respect to anthropics.

But to apply the test of simplifiability - it seems in some raw intuitive sense, that if the universe is large enough for everyone to exist somewhere, then we should mainly be worried about giving babies nice futures rather than trying to "ensure they get born".

Imagine taking a survey of the whole universe. Every plausible baby gets a little checkmark in the "exists" box - everyone is born somewhere. In fact, the total population count for each baby is something-or-other, some large number that may or may not be "infinite" -

(I should mention at this point that I am an infinite set atheist, and my main hope for being able to maintain this in the face of a spatially infinite universe is to suggest that identical Hubble volumes add in the same way as any other identical configuration of particles. So in this case the universe would be exponentially large, the size of the branched decoherent distribution, but the spatial infinity would just fold into that very large but finite object. And I could still be an infinite set atheist. I am not a physicist so my fond hope may be ruled out for some reason of which I am not aware.)

- so the first question, anthropically speaking, is whether multiple realizations of the exact same physical process count as more than one person. Let's say you've got an upload running on a computer. If you look inside the computer and realize that it contains triply redundant processors running in exact synchrony, is that three people or one person? How about if the processor is a flat sheet - if that sheet is twice as thick, is there twice as much person inside it? If we split the sheet and put it back together again without desynchronizing it, have we created a person and killed them?

I suppose the answer could be yes; I have confessed myself stupid about anthropics.

Still: I, as I sit here, am frantically branching into exponentially vast numbers of quantum worlds. I've come to terms with that. It all adds up to normality, after all.

But I don't see myself as having a little utility counter that frantically increases at an exponential rate, just from my sitting here and splitting. The thought of splitting at a faster rate does not much appeal to me, even if such a thing could be arranged.

What I do want for myself, is for the largest possible proportion of my future selves to lead eudaimonic existences, that is, to be happy. This is the "probability" of a good outcome in my expected utility maximization. I'm not concerned with having more of me - really, there are plenty of me already - but I do want most of me to be having fun.

I'm not sure whether or not there exists an imperative for moral civilizations to try to create lots of happy people so as to ensure that most babies born will be happy. But suppose that you started off with 1 baby existing in unhappy regions for every 999 babies existing in happy regions. Would it make sense for the happy regions to create ten times as many babies leading one-tenth the quality of life, so that the universe was "99.99% sorta happy and 0.01% unhappy" instead of "99.9% really happy and 0.1% unhappy"? On the face of it, I'd have to answer "No." (Though it depends on how unhappy the unhappy regions are; and if we start off with the universe mostly unhappy, well, that's a pretty unpleasant possibility...)

But on the whole, it looks to me like if we decide to implement a policy of routinely killing off citizens to replace them with happier babies, or if we lower standards of living to create more people, then we aren't giving the "gift of existence" to babies who wouldn't otherwise have it. We're just setting up the universe to contain the same babies, born predominantly into regions where they lead short lifespans not containing much happiness.

Once someone has been born into your Hubble volume and your Everett branch, you can't undo that; it becomes the responsibility of your region of existence to give them a happy future. You can't hand them back by killing them. That just makes their average lifespan shorter.

It seems to me that in a Big World, the people who already exist in your region have a much stronger claim on your charity than babies who have not yet been born into your region in particular.

And that's why, when there is research to be done, I do it not just for all the future babies who will be born - but, yes, for the people who already exist in our local region, who are already our responsibility.

For the good of all of us, except the ones who are dead.

Referenced by: Amputation of Destiny • Continuous Improvement • The Lifespan Dilemma

Original with comments: For The People Who Are Still Alive

Not Taking Over the World

Eliezer Yudkowsky, 15 December 2008 10:18PM

Followup to: What I Think, If Not Why

My esteemed co-blogger Robin Hanson accuses me of trying to take over the world.

Why, oh why must I be so misunderstood?

(Well, it's not like I don't enjoy certain misunderstandings. Ah, I remember the first time someone seriously and not in a joking way accused me of trying to take over the world. On that day I felt like a true mad scientist, though I lacked a castle and hunchbacked assistant.)

But if you're working from the premise of a hard takeoff - an Artificial Intelligence that self-improves at an extremely rapid rate - and you suppose such extra-ordinary depth of insight and precision of craftsmanship that you can actually specify the AI's goal system instead of automatically failing -

- then it takes some work to come up with a way not to take over the world.

Robin talks up the drama inherent in the intelligence explosion, presumably because he feels that this is a primary source of bias. But I've got to say that Robin's dramatic story, does not sound like the story I tell of myself. There, the drama comes from tampering with such extreme forces that every single idea you invent is wrong. The standardized Final Apocalyptic Battle of Good Vs. Evil would be trivial by comparison; then all you have to do is put forth a desperate effort. Facing an adult problem in a neutral universe isn't so straightforward. Your enemy is yourself, who will automatically destroy the world, or just fail to accomplish anything, unless you can defeat you. - That is the drama I crafted into the story I tell myself, for I too would disdain anything so cliched as Armageddon.

So, Robin, I'll ask you something of a probing question. Let's say that someone walks up to you and grants you unlimited power.

What do you do with it, so as to not take over the world?

Do you say, "I will do nothing - I take the null action"?

But then you have instantly become a malevolent God, as Epicurus said:

Is God willing to prevent evil, but not able? Then he is not omnipotent.
Is he able, but not willing? Then he is malevolent.
Is both able, and willing? Then whence cometh evil?
Is he neither able nor willing? Then why call him God.

Peter Norvig said, "Refusing to act is like refusing to allow time to pass." The null action is also a choice. So have you not, in refusing to act, established all sick people as sick, established all poor people as poor, ordained all in despair to continue in despair, and condemned the dying to death? Will you not be, until the end of time, responsible for every sin committed?

Well, yes and no. If someone says, "I don't trust myself not to destroy the world, therefore I take the null action," then I would tend to sigh and say, "If that is so, then you did the right thing." Afterward, murderers will still be responsible for their murders, and altruists will still be creditable for the help they give.

And to say that you used your power to take over the world by doing nothing to it, seems to stretch the ordinary meaning of the phrase.

But it wouldn't be the best thing you could do with unlimited power, either.

With "unlimited power" you have no need to crush your enemies. You have no moral defense if you treat your enemies with less than the utmost consideration.

With "unlimited power" you cannot plead the necessity of monitoring or restraining others so that they do not rebel against you. If you do such a thing, you are simply a tyrant who enjoys power, and not a defender of the people.

Unlimited power removes a lot of moral defenses, really. You can't say "But I had to." You can't say "Well, I wanted to help, but I couldn't." The only excuse for not helping is if you shouldn't, which is harder to establish.

And let us also suppose that this power is wieldable without side effects or configuration constraints; it is wielded with unlimited precision.

For example, you can't take refuge in saying anything like: "Well, I built this AI, but any intelligence will pursue its own interests, so now the AI will just be a Ricardian trading partner with humanity as it pursues its own goals." Say, the programming team has cracked the "hard problem of conscious experience" in sufficient depth that they can guarantee that the AI they create is not sentient - not a repository of pleasure, or pain, or subjective experience, or any interest-in-self - and hence, the AI is only a means to an end, and not an end in itself.

And you cannot take refuge in saying, "In invoking this power, the reins of destiny have passed out of my hands, and humanity has passed on the torch." Sorry, you haven't created a new person yet - not unless you deliberately invoke the unlimited power to do so - and then you can't take refuge in the necessity of it as a side effect; you must establish that it is the right thing to do.

The AI is not necessarily a trading partner. You could make it a nonsentient device that just gave you things, if you thought that were wiser.

You cannot say, "The law, in protecting the rights of all, must necessarily protect the right of Fred the Deranged to spend all day giving himself electrical shocks." The power is wielded with unlimited precision; you could, if you wished, protect the rights of everyone except Fred.

You cannot take refuge in the necessity of anything - that is the meaning of unlimited power.

We will even suppose (for it removes yet more excuses, and hence reveals more of your morality) that you are not limited by the laws of physics as we know them. You are bound to deal only in finite numbers, but not otherwise bounded. This is so that we can see the true constraints of your morality, apart from your being able to plead constraint by the environment.

In my reckless youth, I used to think that it might be a good idea to flash-upgrade to the highest possible level of intelligence you could manage on available hardware. Being smart was good, so being smarter was better, and being as smart as possible as quickly as possible was best - right?

But when I imagined having infinite computing power available, I realized that no matter how large a mind you made yourself, you could just go on making yourself larger and larger and larger. So that wasn't an answer to the purpose of life. And only then did it occur to me to ask after eudaimonic rates of intelligence increase, rather than just assuming you wanted to immediately be as smart as possible.

Considering the infinite case moved me to change the way I considered the finite case. Before, I was running away from the question by saying "More!" But considering an unlimited amount of ice cream forced me to confront the issue of what to do with any of it.

Similarly with population: If you invoke the unlimited power to create a quadrillion people, then why not a quintillion? If 3^^^3, why not 3^^^^3? So you can't take refuge in saying, "I will create more people - that is the difficult thing, and to accomplish it is the main challenge." What is individually a life worth living?

You can say, "It's not my place to decide; I leave it up to others" but then you are responsible for the consequences of that decision as well. You should say, at least, how this differs from the null act.

So, Robin, reveal to us your character: What would you do with unlimited power?

Referenced by: Visualizing Eutopia • Can't Unbirth a Child

Original with comments: Not Taking Over the World

Visualizing Eutopia

Eliezer Yudkowsky, 16 December 2008 06:39PM

Followup to: Not Taking Over the World

"Heaven is a city 15,000 miles square or 6,000 miles around. One side is 245 miles longer than the length of the Great Wall of China. Walls surrounding Heaven are 396,000 times higher than the Great Wall of China and eight times as thick. Heaven has twelve gates, three on each side, and has room for 100,000,000,000 souls. There are no slums. The entire city is built of diamond material, and the streets are paved with gold. All inhabitants are honest and there are no locks, no courts, and no policemen."
-- Reverend Doctor George Hawes, in a sermon

Yesterday I asked my esteemed co-blogger Robin what he would do with "unlimited power", in order to reveal something of his character. Robin said that he would (a) be very careful and (b) ask for advice. I asked him what advice he would give himself. Robin said it was a difficult question and he wanted to wait on considering it until it actually happened. So overall he ran away from the question like a startled squirrel.

The character thus revealed is a virtuous one: it shows common sense. A lot of people jump after the prospect of absolute power like it was a coin they found in the street.

When you think about it, though, it says a lot about human nature that this is a difficult question. I mean - most agents with utility functions shouldn't have such a hard time describing their perfect universe.

For a long time, I too ran away from the question like a startled squirrel. First I claimed that superintelligences would inevitably do what was right, relinquishing moral responsibility in toto. After that, I propounded various schemes to shape a nice superintelligence, and let it decide what should be done with the world.

Not that there's anything wrong with that. Indeed, this is still the plan. But it still meant that I, personally, was ducking the question.

Why? Because I expected to fail at answering. Because I thought that any attempt for humans to visualize a better future was going to end up recapitulating the Reverend Doctor George Hawes: apes thinking, "Boy, if I had human intelligence I sure could get a lot more bananas."

But trying to get a better answer to a question out of a superintelligence, is a different matter from entirely ducking the question yourself. The point at which I stopped ducking was the point at which I realized that it's actually quite difficult to get a good answer to something out of a superintelligence, while simultaneously having literally no idea how to answer yourself.

When you're dealing with confusing and difficult questions - as opposed to those that are straightforward but numerically tedious - it's quite suspicious to have, on the one hand, a procedure that executes to reliably answer the question, and, on the other hand, no idea of how to answer it yourself.

If you could write a computer program that you knew would reliably output a satisfactory answer to "Why does anything exist in the first place?" or "Why do I find myself in a universe giving rise to experiences that are ordered rather than chaotic?", then shouldn't you be able to at least try executing the same procedure yourself?

I suppose there could be some section of the procedure where you've got to do a septillion operations and so you've just got no choice but to wait for superintelligence, but really, that sounds rather suspicious in cases like these.

So it's not that I'm planning to use the output of my own intelligence to take over the universe. But I did realize at some point that it was too suspicious to entirely duck the question while trying to make a computer knowably solve it. It didn't even seem all that morally cautious, once I put in those terms. You can design an arithmetic chip using purely abstract reasoning, but would you be wise to never try an arithmetic problem yourself?

And when I did finally try - well, that caused me to update in various ways.

It does make a difference to try doing arithmetic yourself, instead of just trying to design chips that do it for you. So I found.

Hence my bugging Robin about it.

For it seems to me that Robin asks too little of the future. It's all very well to plead that you are only forecasting, but if you display greater revulsion to the idea of a Friendly AI than to the idea of rapacious hardscrapple frontier folk...

I thought that Robin might be asking too little, due to not visualizing any future in enough detail. Not the future but any future. I'd hoped that if Robin had allowed himself to visualize his "perfect future" in more detail, rather than focusing on all the compromises he thinks he has to make, he might see that there were futures more desirable than the rapacious hardscrapple frontier folk.

It's hard to see on an emotional level why a genie might be a good thing to have, if you haven't acknowledged any wishes that need granting. It's like not feeling the temptation of cryonics, if you haven't thought of anything the Future contains that might be worth seeing.

I'd also hoped to persuade Robin, if his wishes were complicated enough, that there were attainable good futures that could not come about by letting things go their own way. So that he might begin to see the future as I do, as a dilemma between extremes: The default, loss of control, followed by a Null future containing little or no utility. Versus extremely precise steering through "impossible" problems to get to any sort of Good future whatsoever.

This is mostly a matter of appreciating how even the desires we call "simple" actually contain many bits of information. Getting past anthropomorphic optimism, to realize that a Future not strongly steered by our utility functions is likely to contain little or no utility, for the same reason it's hard to hit a distant target while shooting blindfolded...

But if your "desired future" remains mostly unspecified, that may encourage too much optimism as well.

Referenced by: Prolegomena to a Theory of Fun

Original with comments: Visualizing Eutopia

Prolegomena to a Theory of Fun

Eliezer Yudkowsky, 17 December 2008 11:33PM

(This post is part of the Fun Theory Sequence.)
Followup to: Joy in the Merely Good

Raise the topic of cryonics, uploading, or just medically extended lifespan/healthspan, and some bioconservative neo-Luddite is bound to ask, in portentous tones:

"But what will people do all day?"

They don't try to actually answer the question. That is not a bioethicist's role, in the scheme of things. They're just there to collect credit for the Deep Wisdom of asking the question. It's enough to imply that the question is unanswerable, and therefore, we should all drop dead.

That doesn't mean it's a bad question.

It's not an easy question to answer, either. The primary experimental result in hedonic psychology - the study of happiness - is that people don't know what makes them happy.

And there are many exciting results in this new field, which go a long way toward explaining the emptiness of classical Utopias. But it's worth remembering that human hedonic psychology is not enough for us to consider, if we're asking whether a million-year lifespan could be worth living.

Fun Theory, then, is the field of knowledge that would deal in questions like:

- "How much fun is there in the universe?"

- "Will we ever run out of fun?"

- "Are we having fun yet?"

- "Could we be having more fun?"

One major set of experimental results in hedonic psychology has to do with overestimating the impact of life events on happiness. Six months after the event, lottery winners aren't as happy as they expected to be, and quadriplegics aren't as sad. A parent who loses a child isn't as sad as they think they'll be, a few years later. If you look at one moment snapshotted out of their lives a few years later, that moment isn't likely to be about the lost child. Maybe they're playing with one of their surviving children on a swing. Maybe they're just listening to a nice song on the radio.

When people are asked to imagine how happy or sad an event will make them, they anchor on the moment of first receiving the news, rather than realistically imagining the process of daily life years later.

Consider what the Christians made of their Heaven, meant to be literally eternal. Endless rest, the glorious presence of God, and occasionally - in the more clueless sort of sermon - golden streets and diamond buildings. Is this eudaimonia? It doesn't even seem very hedonic.

As someone who said his share of prayers back in his Orthodox Jewish childhood upbringing, I can personally testify that praising God is an enormously boring activity, even if you're still young enough to truly believe in God. The part about praising God is there as an applause light that no one is allowed to contradict: it's something theists believe they should enjoy, even though, if you ran them through an fMRI machine, you probably wouldn't find their pleasure centers lighting up much.

Ideology is one major wellspring of flawed Utopias, containing things that the imaginer believes should be enjoyed, rather than things that would actually be enjoyable.

And eternal rest? What could possibly be more boring than eternal rest?

But to an exhausted, poverty-stricken medieval peasant, the Christian Heaven sounds like good news in the moment of being first informed: You can lay down the plow and rest! Forever! Never to work again!

It'd get boring after... what, a week? A day? An hour?

Heaven is not configured as a nice place to live. It is rather memetically optimized to be a nice place for an exhausted peasant to imagine. It's not like some Christians actually got a chance to live in various Heavens, and voted on how well they liked it after a year, and then they kept the best one. The Paradise that survived was the one that was retold, not lived.

Timothy Feriss observed, "Living like a millionaire requires doing interesting things and not just owning enviable things." Golden streets and diamond walls would fade swiftly into the background, once obtained - but so long as you don't actually have gold, it stays desirable.

And there's two lessons required to get past such failures; and these lessons are in some sense opposite to one another.

The first lesson is that humans are terrible judges of what will actually make them happy, in the real world and the living moments. Daniel Gilbert's Stumbling on Happiness is the most famous popular introduction to the research.

We need to be ready to correct for such biases - the world that is fun to live in, may not be the world that sounds good when spoken into our ears.

And the second lesson is that there's nothing in the universe out of which to construct Fun Theory, except that which we want for ourselves or prefer to become.

If, in fact, you don't like praying, then there's no higher God than yourself to tell you that you should enjoy it. We sometimes do things we don't like, but that's still our own choice. There's no outside force to scold us for making the wrong decision.

This is something for transhumanists to keep in mind - not because we're tempted to pray, of course, but because there are so many other logical-sounding solutions we wouldn't really want.

The transhumanist philosopher David Pearce is an advocate of what he calls the Hedonistic Imperative: The eudaimonic life is the one that is as pleasurable as possible. So even happiness attained through drugs is good? Yes, in fact: Pearce's motto is "Better Living Through Chemistry".

Or similarly: When giving a small informal talk once on the Stanford campus, I raised the topic of Fun Theory in the post-talk mingling. And someone there said that his ultimate objective was to experience delta pleasure. That's "delta" as in the Dirac delta - roughly, an infinitely high spike (that happens to be integrable). "Why?" I asked. He said, "Because that means I win."

(I replied, "How about if you get two times delta pleasure? Do you win twice as hard?")

In the transhumanist lexicon, "orgasmium" refers to simplified brains that are just pleasure centers experiencing huge amounts of stimulation - a happiness counter containing a large number, plus whatever the minimum surrounding framework to experience it. You can imagine a whole galaxy tiled with orgasmium. Would this be a good thing?

And the vertigo-inducing thought is this - if you would prefer not to become orgasmium, then why should you?

Mind you, there are many reasons why something that sounds unpreferred at first glance, might be worth a closer look. That was the first lesson. Many Christians think they want to go to Heaven.

But when it comes to the question, "Don't I have to want to be as happy as possible?" then the answer is simply "No. If you don't prefer it, why go there?"

There's nothing except such preferences out of which to construct Fun Theory - a second look is still a look, and must still be constructed out of preferences at some level.

In the era of my foolish youth, when I went into an affective death spiral around intelligence, I thought that the mysterious "right" thing that any superintelligence would inevitably do, would be to upgrade every nearby mind to superintelligence as fast as possible. Intelligence was good; therefore, more intelligence was better.

Somewhat later I imagined the scenario of unlimited computing power, so that no matter how smart you got, you were still just as far from infinity as ever. That got me thinking about a journey rather than a destination, and allowed me to think "What rate of intelligence increase would be fun?"

But the real break came when I naturalized my understanding of morality, and value stopped being a mysterious attribute of unknown origins.

Then if there was no outside light in the sky to order me to do things -

The thought occurred to me that I didn't actually want to bloat up immediately into a superintelligence, or have my world transformed instantaneously and completely into something incomprehensible. I'd prefer to have it happen gradually, with time to stop and smell the flowers along the way.

It felt like a very guilty thought, but -

But there was nothing higher to override this preference.

In which case, if the Friendly AI project succeeded, there would be a day after the Singularity to wake up to, and myself to wake up to it.

You may not see why this would be a vertigo-inducing concept. Pretend you're Eliezer_2003 who has spent the last seven years talking about how it's forbidden to try to look beyond the Singularity - because the AI is smarter than you, and if you knew what it would do, you would have to be that smart yourself -

- but what if you don't want the world to be made suddenly incomprehensible? Then there might be something to understand, that next morning, because you don't actually want to wake up in an incomprehensible world, any more than you actually want to suddenly be a superintelligence, or turn into orgasmium.

I can only analogize the experience to a theist who's suddenly told that they can know the mind of God, and it turns out to be only twenty lines of Python.

You may find it hard to sympathize. Well, Eliezer_1996, who originally made the mistake, was smart but methodologically inept, as I've mentioned a few times.

Still, expect to see some outraged comments on this very blog post, from commenters who think that it's selfish and immoral, and above all a failure of imagination, to talk about human-level minds still running around the day after the Singularity.

That's the frame of mind I used to occupy - that the things I wanted were selfish, and that I shouldn't think about them too much, or at all, because I would need to sacrifice them for something higher.

People who talk about an existential pit of meaninglessness in a universe devoid of meaning - I'm pretty sure they don't understand morality in naturalistic terms. There is vertigo involved, but it's not the vertigo of meaninglessness.

More like a theist who is frightened that someday God will order him to murder children, and then he realizes that there is no God and his fear of being ordered to murder children was morality. It's a strange relief, mixed with the realization that you've been very silly, as the last remnant of outrage at your own selfishness fades away.

So the first step toward Fun Theory is that, so far as I can tell, it looks basically okay to make our future light cone - all the galaxies that we can get our hands on - into a place that is fun rather than not fun.

We don't need to transform the universe into something we feel dutifully obligated to create, but isn't really much fun - in the same way that a Christian would feel dutifully obliged to enjoy heaven - or that some strange folk think that creating orgasmium is, logically, the rightest thing to do.

Fun is okay. It's allowed. It doesn't get any better than fun.

And then we can turn our attention to the question of what is fun, and how to have it.

Sequence: Fun Theory

High Challenge

Referenced by: High Challenge • Amputation of Destiny • Free to Optimize • The Uses of Fun (Theory) • Continuous Improvement • Eutopia is Scary • Interpersonal Entanglement • The Fun Theory Sequence • 31 Laws of Fun • Can Humanism Match Religion's Output?

Original with comments: Prolegomena to a Theory of Fun

High Challenge

Eliezer Yudkowsky, 19 December 2008 12:51AM

(This post is part of the Fun Theory Sequence.)
Followup to: Not for the Sake of Happiness (Alone), Existential Angst Factory

There's a class of prophecy that runs: "In the Future, machines will do all the work. Everything will be automated. Even labor of the sort we now consider 'intellectual', like engineering, will be done by machines. We can sit back and own the capital. You'll never have to lift a finger, ever again."

But then won't people be bored?

No; they can play computer games - not like our games, of course, but much more advanced and entertaining.

Yet wait! If you buy a modern computer game, you'll find that it contains some tasks that are - there's no kind word for this - effortful. (I would even say "difficult", with the understanding that we're talking about something that takes 10 minutes, not 10 years.)

So in the future, we'll have programs that help you play the game - taking over if you get stuck on the game, or just bored; or so that you can play games that would otherwise be too advanced for you.

But isn't there some wasted effort, here? Why have one programmer working to make the game harder, and another programmer to working to make the game easier? Why not just make the game easier to start with? Since you play the game to get gold and experience points, making the game easier will let you get more gold per unit time: the game will become more fun.

So this is the ultimate end of the prophecy of technological progress - just staring at a screen that says "YOU WIN", forever.

And maybe we'll build a robot that does that, too.

Then what?

The world of machines that do all the work - well, I don't want to say it's "analogous to the Christian Heaven" because it isn't supernatural; it's something that could in principle be realized. Religious analogies are far too easily tossed around as accusations... But, without implying any other similarities, I'll say that it seems analogous in the sense that eternal laziness "sounds like good news" to your present self who still has to work.

And as for playing games, as a substitute - what is a computer game except synthetic work? Isn't there a wasted step here? (And computer games in their present form, considered as work, have various aspects that reduce stress and increase engagement; but they also carry costs in the form of artificiality and isolation.)

I sometimes think that futuristic ideals phrased in terms of "getting rid of work" would be better reformulated as "removing low-quality work to make way for high-quality work".

There's a broad class of goals that aren't suitable as the long-term meaning of life, because you can actually achieve them, and then you're done.

To look at it another way, if we're looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.

Or to phrase that somewhat less paradoxically: We should look for valuations that are over 4D states, rather than 3D states. Valuable ongoing processes, rather than "make the universe have property P and then you're done".

Timothy Ferris is again worth quoting: To find happiness, "the question you should be asking isn't 'What do I want?' or 'What are my goals?' but 'What would excite me?'"

You might say that for a long-run meaning of life, we need games that are fun to play and not just to win.

Mind you - sometimes you do want to win. There are legitimate goals where winning is everything. If you're talking, say, about curing cancer, then the suffering experienced by even a single cancer patient outweighs any fun that you might have in solving their problems. If you work at creating a cancer cure for twenty years through your own efforts, learning new knowledge and new skill, making friends and allies - and then some alien superintelligence offers you a cancer cure on a silver platter for thirty bucks - then you shut up and take it.

But "curing cancer" is a problem of the 3D-predicate sort: you want the no-cancer predicate to go from False in the present to True in the future. The importance of this destination far outweighs the journey; you don't want to go there, you just want to be there. There are many legitimate goals of this sort, but they are not suitable as long-run fun. "Cure cancer!" is a worthwhile activity for us to pursue here and now, but it is not a plausible future goal of galactic civilizations.

Why should this "valuable ongoing process" be a process of trying to do things - why not a process of passive experiencing, like the Buddhist Heaven?

I confess I'm not entirely sure how to set up a "passively experiencing" mind. The human brain was designed to perform various sorts of internal work that add up to an active intelligence; even if you lie down on your bed and exert no particular effort to think, the thoughts that go on through your mind are activities of brain areas that are designed to, you know, solve problems.

How much of the human brain could you eliminate, apart from the pleasure centers, and still keep the subjective experience of pleasure?

I'm not going to touch that one. I'll stick with the much simpler answer of "I wouldn't actually prefer to be a passive experiencer." If I wanted Nirvana, I might try to figure out how to achieve that impossibility. But once you strip away Buddha telling me that Nirvana is the end-all of existence, Nirvana seems rather more like "sounds like good news in the moment of first being told" or "ideological belief in desire" rather than, y'know, something I'd actually want.

The reason I have a mind at all, is that natural selection built me to do things - to solve certain kinds of problems.

"Because it's human nature" is not an explicit justification for anything. There is human nature, which is what we are; and there is humane nature, which is what, being human, we wish we were.

But I don't want to change my nature toward a more passive object - which is a justification. A happy blob is not what, being human, I wish to become.

I earlier argued that many values require both subjective happiness and the external objects of that happiness. That you can legitimately have a utility function that says, "It matters to me whether or not the person I love is a real human being or just a highly realistic nonsentient chatbot, even if I don't know, because that-which-I-value is not my own state of mind, but the external reality." So that you need both the experience of love, and the real lover.

You can similarly have valuable activities that require both real challenge and real effort.

Racing along a track, it matters that the other racers are real, and that you have a real chance to win or lose. (We're not talking about physical determinism here, but whether some external optimization process explicitly chose for you to win the race.)

And it matters that you're racing with your own skill at running and your own willpower, not just pressing a button that says "Win". (Though, since you never designed your own leg muscles, you are racing using strength that isn't yours. A race between robot cars is a purer contest of their designers. There is plenty of room to improve on the human condition.)

And it matters that you, a sentient being, are experiencing it. (Rather than some nonsentient process carrying out a skeleton imitation of the race, trillions of times per second.)

There must be the true effort, the true victory, and the true experience - the journey, the destination and the traveler.

Sequence: Fun Theory

Prolegomena to a Theory of Fun

Complex Novelty

Referenced by: Harmful Options • Amputation of Destiny • Free to Optimize • Emotional Involvement • Serious Stories • Continuous Improvement • Justified Expectation of Pleasant Surprises • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: High Challenge

Complex Novelty

Eliezer Yudkowsky, 20 December 2008 12:31AM

(This post is part of the Fun Theory Sequence.)

From Greg Egan's Permutation City:

The workshop abutted a warehouse full of table legs - one hundred and sixty-two thousand, three hundred and twenty-nine, so far. Peer could imagine nothing more satisfying than reaching the two hundred thousand mark - although he knew it was likely that he'd change his mind and abandon the workshop before that happened; new vocations were imposed by his exoself at random intervals, but statistically, the next one was overdue. Immediately before taking up woodwork, he'd passionately devoured all the higher mathematics texts in the central library, run all the tutorial software, and then personally contributed several important new results to group theory - untroubled by the fact that none of the Elysian mathematicians would ever be aware of his work. Before that, he'd written over three hundred comic operas, with librettos in Italian, French and English - and staged most of them, with puppet performers and audience. Before that, he'd patiently studied the structure and biochemistry of the human brain for sixty-seven years; towards the end he had fully grasped, to his own satisfaction, the nature of the process of consciousness. Every one of these pursuits had been utterly engrossing, and satisfying, at the time. He'd even been interested in the Elysians, once.
No longer. He preferred to think about table legs.

Among science fiction authors, Greg Egan is my favorite; of Greg Egan's books, Permutation City is my favorite; and this particular passage in Permutation City, more than any of the others, I find utterly horrifying.

If this were all the hope the future held, I don't know if I could bring myself to try. Small wonder that people don't sign up for cryonics, if even SF writers think this is the best we can do.

You could think of this whole series on Fun Theory as my reply to Greg Egan - a list of the ways that his human-level uploaded civilizations Fail At Fun. (And yes, this series will also explain what's wrong with the Culture and how to fix it.)

We won't get to all of Peer's problems today - but really. Table legs?

I could see myself carving one table leg, maybe, if there was something non-obvious to learn from the experience. But not 162,329.

In Permutation City, Peer modified himself to find table-leg-carving fascinating and worthwhile and pleasurable. But really, at that point, you might as well modify yourself to get pleasure from playing Tic-Tac-Toe, or lie motionless on a pillow as a limbless eyeless blob having fantastic orgasms. It's not a worthy use of a human-level intelligence.

Worse, carving the 162,329th table leg doesn't teach you anything that you didn't already know from carving 162,328 previous table legs. A mind that changes so little in life's course is scarcely experiencing time.

But apparently, once you do a little group theory, write a few operas, and solve the mystery of consciousness, there isn't much else worth doing in life: you've exhausted the entirety of Fun Space down to the level of table legs.

Is this plausible? How large is Fun Space?

Let's say you were a human-level intelligence who'd never seen a Rubik's Cube, or anything remotely like it. As Hofstadter describes in two whole chapters of Metamagical Themas, there's a lot that intelligent human novices can learn from the Cube - like the whole notion of an "operator" or "macro", a sequence of moves that accomplishes a limited swap with few side effects. Parity, search, impossibility -

So you learn these things in the long, difficult course of solving the first scrambled Rubik's Cube you encounter. The second scrambled Cube - solving it might still be difficult, still be enough fun to be worth doing. But you won't have quite the same pleasurable shock of encountering something as new, and strange, and interesting as the first Cube was unto you.

Even if you encounter a variant of the Rubik's Cube - like a 4x4x4 Cube instead of a 3x3x3 Cube - or even a Rubik's Tesseract (a 3x3x3x3 Cube in four dimensions) - it still won't contain quite as much fun as the first Cube you ever saw. I haven't tried mastering the Rubik's Tesseract myself, so I don't know if there are added secrets in four dimensions - but it doesn't seem likely to teach me anything as fundamental as "operators", "side effects", or "parity".

(I was quite young when I encountered a Rubik's Cube in a toy cache, and so that actually is where I discovered such concepts. I tried that Cube on and off for months, without solving it. Finally I took out a book from the library on Cubes, applied the macros there, and discovered that this particular Cube was unsolvable - it had been disassembled and reassembled into an impossible position. I think I was faintly annoyed.)

Learning is fun, but it uses up fun: you can't have the same stroke of genius twice. Insight is insight because it makes future problems less difficult, and "deep" because it applies to many such problems.

And the smarter you are, the faster you learn - so the smarter you are, the less total fun you can have. Chimpanzees can occupy themselves for a lifetime at tasks that would bore you or I to tears. Clearly, the solution to Peer's difficulty is to become stupid enough that carving table legs is difficult again - and so lousy at generalizing that every table leg is a new and exciting challenge -

Well, but hold on: If you're a chimpanzee, you can't understand the Rubik's Cube at all. At least I'm willing to bet against anyone training a chimpanzee to solve one - let alone a chimpanzee solving it spontaneously - let alone a chimpanzee understanding the deep concepts like "operators", "side effects", and "parity".

I could be wrong here, but it seems to me, on the whole, that when you look at the number of ways that chimpanzees have fun, and the number of ways that humans have fun, that Human Fun Space is larger than Chimpanzee Fun Space.

And not in a way that increases just linearly with brain size, either.

The space of problems that are Fun to a given brain, will definitely be smaller than the exponentially increasing space of all possible problems that brain can represent. We are interested only in the borderland between triviality and impossibility - problems difficult enough to worthily occupy our minds, yet tractable enough to be worth challenging. (What looks "impossible" is not always impossible, but the border is still somewhere even if we can't see it at a glance - there are some problems so difficult you can't even learn much from failing.)

An even stronger constraint is that if you do something many times, you ought to learn from the experience and get better - many problems of the same difficulty will have the same "learnable lessons" embedded in them, so that doing one consumes some of the fun of others.

As you learn new things, and your skills improve, problems will get easier. Some will move off the border of the possible and the impossible, and become too easy to be interesting.

But others will move from the territory of impossibility into the borderlands of mere extreme difficulty. It's easier to invent group theory if you've solved the Rubik's Cube first. There are insights you can't have without prerequisite insights.

If you get smarter over time (larger brains, improved mind designs) that's a still higher octave of the same phenomenon. (As best I can grasp the Law, there are insights you can't understand at all without having a brain of sufficient size and sufficient design. Humans are not maximal in this sense, and I don't think there should be any maximum - but that's a rather deep topic, which I shall not explore further in this blog post. Note that Greg Egan seems to explicitly believe the reverse - that humans can understand anything understandable - which explains a lot.)

One suspects that in a better-designed existence, the eudaimonic rate of intelligence increase would be bounded below by the need to integrate the loot of your adventures - to incorporate new knowledge and new skills efficiently, without swamping your mind in a sea of disconnected memories and associations - to manipulate larger, more powerful concepts that generalize more of your accumulated life-knowledge at once.

And one also suspects that part of the poignancy of transhuman existence will be having to move on from your current level - get smarter, leaving old challenges behind - before you've explored more than an infinitesimal fraction of the Fun Space for a mind of your level. If, like me, you play through computer games trying to slay every single monster so you can collect every single experience point, this is as much tragedy as an improved existence could possibly need.

Fun Space can increase much more slowly than the space of representable problems, and still overwhelmingly swamp the amount of time you could bear to spend as a mind of a fixed level. Even if Fun Space grows at some ridiculously tiny rate like N-squared - bearing in mind that the actual raw space of representable problems goes as 2N - we're still talking about "way more fun than you can handle".

If you consider the loot of every human adventure - everything that was ever learned about science, and everything that was ever learned about people, and all the original stories ever told, and all the original games ever invented, and all the plots and conspiracies that were ever launched, and all the personal relationships ever raveled, and all the ways of existing that were ever tried, and all the glorious epiphanies of wisdom that were ever minted -

- and you deleted all the duplicates, keeping only one of every lesson that had the same moral -

- how long would you have to stay human, to collect every gold coin in the dungeons of history?

Would it all fit into a single human brain, without that mind completely disintegrating under the weight of unrelated associations? And even then, would you have come close to exhausting the space of human possibility, which we've surely not finished exploring?

This is all sounding like suspiciously good news. So let's turn it around. Is there any way that Fun Space could fail to grow, and instead collapse?

Suppose there's only so many deep insights you can have on the order of "parity", and that you collect them all, and then math is never again as exciting as it was in the beginning. And that you then exhaust the shallower insights, and the trivial insights, until finally you're left with the delightful shock of "Gosh wowie gee willickers, the product of 845 and 109 is 92105, I didn't know that logical truth before."

Well - obviously, if you sit around and catalogue all the deep insights known to you to exist, you're going to end up with a bounded list. And equally obviously, if you declared, "This is all there is, and all that will ever be," you'd be taking an unjustified step. (Though I fully expect some people out there to step up and say how it seems to them that they've already started to run out of available insights that are as deep as the ones they remember from their childhood. And I fully expect that - compared to the sort of person who makes such a pronouncement - I personally will have collected more additional insights than they believe exist in the whole remaining realm of possibility.)

Can we say anything more on this subject of fun insights that might exist, but that we haven't yet found?

The obvious thing to do is start appealing to Godel, but Godelian arguments are dangerous tools to employ in debate. It does seem to me that Godelian arguments weigh in the general direction of "inexhaustible deep insights", but inconclusively and only by loose analogies.

For example, the Busy-Beaver(N) problem asks for the longest running time of a Turing machine with no more than N states. The Busy Beaver problem is uncomputable - there is no fixed Turing machine that computes it for all N - because if you knew all the Busy Beaver numbers, you would have an infallible way of telling whether a Turing machine halts; just run it up for as long as the longest-running Turing machine of that size.

The human species has managed to figure out and prove the Busy Beaver numbers up to 4, and they are:

BB(1): 1
BB(2): 6
BB(3): 21
BB(4): 107

Busy-Beaver 5 is believed to be 47,176,870.

The current lower bound on Busy-Beaver(6) is ~2.5 × 10^2879.

This function provably grows faster than any compact specification you can imagine. Which would seem to argue that each new Turing machine is exhibiting a new and interesting kind of behavior. Given infinite time, you would even be able to notice this behavior. You won't ever know for certain that you've discovered the Busy-Beaver champion for any given N, after finite time; but conversely, you will notice the Busy Beaver champion for any N after some finite time.

Yes, this is an unimaginably long time - one of the few occasions where the word "unimaginable" is literally correct. We can't actually do this unless reality works the way it does in Greg Egan novels. But the point is that in the limit of infinite time we can point to something sorta like "an infinite sequence of learnable deep insights not reducible to any of their predecessors or to any learnable abstract summary". It's not conclusive, but it's at least suggestive.

Now you could still look at that and say, "I don't think my life would be an adventure of neverending excitement if I spent until the end of time trying to figure out the weird behaviors of slightly larger Tuing machines."

Well - as I said before, Peer is doing more than one thing wrong. Here I've dealt with only one sort of dimension of Fun Space - the dimension of how much novelty we can expect to find available to introduce into our fun.

But even on the arguments given so far... I don't call it conclusive, but it seems like sufficient reason to hope and expect that our descendants and future selves won't exhaust Fun Space to the point that there is literally nothing left to do but carve the 162,329th table leg.

Sequence: Fun Theory

High Challenge

Continuous Improvement

Referenced by: Sensual Experience • Amputation of Destiny • A New Day • Emotional Involvement • Continuous Improvement • Justified Expectation of Pleasant Surprises • In Praise of Boredom • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Complex Novelty

Sensual Experience

Eliezer Yudkowsky, 21 December 2008 12:56AM

(This post is part of the Fun Theory Sequence.)

Modern day gamemakers are constantly working on higher-resolution, more realistic graphics; more immersive sounds - but they're a long long way off real life.

Pressing the "W" key to run forward as a graphic of a hungry tiger bounds behind you, just doesn't seem quite as sensual as running frantically across the savanna with your own legs, breathing in huge gasps and pumping your arms as the sun beats down on your shoulders, the grass brushes your shins, and the air whips around you with the wind of your passage.

Don't mistake me for a luddite; I'm not saying the technology can't get that good. I'm saying it hasn't gotten that good yet.

Failing to escape the computer tiger would also have fewer long-term consequences than failing to escape a biological tiger - it would be less a part of the total story of your life - meaning you're also likely to be less emotionally involved. But that's a topic for another post. Today's post is just about the sensual quality of the experience.

Sensual experience isn't a question of some mysterious quality that only the "real world" possesses. A computer screen is as real as a tiger, after all. Whatever is, is real.

But the pattern of the pseudo-tiger, inside the computer chip, is nowhere near as complex as a biological tiger; it offers far fewer modes in which to interact. And the sensory bandwidth between you and the computer's pseudo-world is relatively low; and the information passing along it isn't in quite the right format.

It's not a question of computer tigers being "virtual" or "simulated", and therefore somehow a separate magisterium. But with present technology, and the way your brain is presently set up, you'd have a lot more neurons involved in running away from a biological tiger.

Running would fill your whole vision with motion, not just a flat rectangular screen - which translates into more square centimeters of visual cortex getting actively engaged.

The graphics on a computer monitor try to trigger your sense of spatial motion (residing in the parietal cortex, btw). But they're presenting the information differently from its native format - without binocular vision, for example, and without your vestibular senses indicating true motion. So the sense of motion isn't likely to be quite the same, what it would be if you were running.

And there's the sense of touch that indicates the wind on your skin; and the proprioceptive sensors that respond to the position of your limbs; and the nerves that record the strain on your muscles. There's a whole strip of sensorimotor cortex running along the top of your brain, that would be much more intensely involved in "real" running.

It's a very old observation, that Homo sapiens was made to hunt and gather on the savanna, rather than work in an office. Civilization and its discontents... But alienation needs a causal mechanism; it doesn't just happen by magic. Physics is physics, so it's not that one environment is less real than another. But our brains are more adapted to interfacing with jungles than computer code.

Writing a complicated computer program carries its own triumphs and failures, heights of exultation and pits of despair. But is it the same sort of sensual experience as, say, riding a motorcycle? I've never actually ridden a motorcycle, but I expect not.

I've experienced the exhilaration of getting a program right on the dozenth try after finally spotting the problem. I doubt a random moment of a motorcycle ride actually feels better than that. But still, my hunter-gatherer ancestors never wrote computer programs. And so my mind's grasp on code is maintained using more rarefied, more abstract, more general capabilities - which means less sensual involvement.

Doesn't computer programming deserve to be as much of a sensual experience as motorcycle riding? Some time ago, a relative once asked me if I thought that computer programming could use all my talents; I at once replied, "There is no limit to the talent you can use in computer programming." It's as close as human beings have ever come to playing with the raw stuff of creation - but our grasp on it is too distant from the jungle. All our involvement is through letters on a computer screen. I win, and I'm happy, but there's no wind on my face.

If only my ancestors back to the level of my last common ancestor with a mouse, had constantly faced the challenge of writing computer programs! Then I would have brain areas suited to the task, and programming computers would be more of a sensual experience...

Perhaps it's not too late to fix the mistake?

If there were something around that was smart enough to rewrite human brains without breaking them - not a trivial amount of smartness - then it would be possible to expand the range of things that are sensually fun.

Not just novel challenges, but novel high-bandwidth senses and corresponding new brain areas. Widening the sensorium to include new vivid, detailed experiences. And not neglecting the other half of the equation, high-bandwidth motor connections - new motor brain areas, to control with subtlety our new limbs (the parts of the process that we control as our direct handles on it).

There's a story - old now, but I remember how exciting it was when the news first came out - about a brain-computer interface for a "locked-in" patient (who could previously only move his eyes), connecting control of a computer cursor directly to neurons in his visual cortex. It took some training at first for him to use the cursor - he started out by trying to move his paralyzed arm, which was the part of the motor cortex they were interfacing, and watched as the cursor jerked around on the screen. But after a while, they asked the patient, "What does it feel like?" and the patient replied, "It doesn't feel like anything." He just controlled the cursor the same sort of way he would have controlled a finger, except that it wasn't a finger, it was a cursor.

Like most brain modifications, adding new senses is not something to be done lightly. Sensual experience too easily renders a task involving.

Consider taste buds. Recognizing the taste of the same food on different occasions was very important to our ancestors - it was how they learned what to eat, that extracted regularity. And our ancestors also got helpful reinforcement from their taste buds about what to eat - reinforcement which is now worse than useless, because of the marketing incentive to reverse-engineer tastiness using artificial substances. By now, it's probably true that at least some people have eaten 162,329 potato chips in their lifetimes. That's even less novelty and challenge than carving 162,329 table legs.

I'm not saying we should try to eliminate our senses of taste. There's a lot to be said for grandfathering in the senses we started with - it preserves our existing life memories, for example. Once you realize how easy it would be for a mind to collapse into a pleasure center, you start to respect the "complications" of your goal system a lot more, and be more wary around "simplifications".

But I do want to nudge people into adopting something of a questioning attitude toward the senses we have now, rather than assuming that the existing senses are The Way Things Have Been And Will Always Be. A sex organ bears thousands of densely packed nerves for signal strength, but that signal - however strong - isn't as complicated as the sensations sent out by taste buds. Is that really appropriate for one of the most interesting parts of human existence? That even a novice chef can create a wider variety of taste sensations for your tongue, than - well, I'd better stop there. But from a fun-theoretic standpoint, the existing setup is wildly unbalanced in a lot of ways. It wasn't designed for the sake of eudaimonia.

I conclude with the following cautionary quote from an old IRC conversation, as a reminder that maybe not everything should be a sensual experience:

<MRAmes> I want a sensory modality for regular expressions.

Sequence: Fun Theory

Continuous Improvement

Living By Your Own Strength

Referenced by: Emotional Involvement • Serious Stories • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Sensual Experience

Living By Your Own Strength

Eliezer Yudkowsky, 22 December 2008 12:37AM

(This post is part of the Fun Theory Sequence.)
Followup to: Truly Part of You

"Myself, and Morisato-san... we want to live together by our own strength."

Jared Diamond once called agriculture "the worst mistake in the history of the human race". Farmers could grow more wheat than hunter-gatherers could collect nuts, but the evidence seems pretty conclusive that agriculture traded quality of life for quantity of life. One study showed that the farmers in an area were six inches shorter and seven years shorter-lived than their hunter-gatherer predecessors - even though the farmers were more numerous.

I don't know if I'd call agriculture a mistake. But one should at least be aware of the downsides. Policy debates should not appear one-sided.

In the same spirit -

Once upon a time, our hunter-gatherer ancestors strung their own bows, wove their own baskets, whittled their own flutes.

And part of our alienation from that environment of evolutionary adaptedness, is the number of tools we use that we don't understand and couldn't make for ourselves.

You can look back on Overcoming Bias, and see that I've always been suspicious of borrowed strength. (Even before I understood the source of Robin's and my disagreement about the Singularity, that is.) In Guessing the Teacher's Password I talked about the (well-known) problem in which schools end up teaching verbal behavior rather than real knowledge. In Truly Part of You I suggested one test for false knowledge: Imagine deleting a fact from your mind, and ask if it would grow back.

I know many ways to prove the Pythagorean Theorem, including at least one proof that is purely visual and can be seen at a glance. But if you deleted the Pythagorean Theorem from my mind entirely, would I have enough math skills left to grow it back the next time I needed it? I hope so - certainly I've solved math problems that seem tougher than that, what with benefit of hindsight and all. But, as I'm not an AI, I can't actually switch off the memories and associations, and test myself in that way.

Wielding someone else's strength to do things beyond your own understanding - that really is as dangerous as the Deeply Wise phrasing makes it sound.

I observed in Failing to Learn from History (musing on my childhood foolishness in offering a mysterious answer to a mysterious question): "If only I had personally postulated astrological mysteries and then discovered Newtonian mechanics, postulated alchemical mysteries and then discovered chemistry, postulated vitalistic mysteries and then discovered biology. I would have thought of my Mysterious Answer and said to myself: No way am I falling for that again."

At that point in my childhood, I'd been handed some techniques of rationality but I didn't exactly own them. Borrowing someone else's knowledge really doesn't give you anything remotely like the same power level required to discover that knowledge for yourself.

Would Isaac Newton have remained a mystic, even in that earlier era, if he'd lived the lives of Galileo and Archimedes instead of just reading about them? If he'd personally seen the planets reduced from gods to spheres in a telescope? If he'd personally fought that whole war against ignorance and mystery that had to be fought, before Isaac Newton could be handed math and science as a start to his further work?

We stand on the shoulders of giants, and in doing so, the power that we wield is far out of proportion to the power that we could generate for ourselves. This is true even of our revolutionaries. And yes, we couldn't begin to support this world if people could only use their own strength. Even so, we are losing something.

That thought occurred to me, reading about the Manhattan Project, and the petition that the physicists signed to avoid dropping the atomic bomb on Japan. It was too late, of course; they'd already built the bombs and handed them to the military, and they couldn't take back that gift. And so nuclear weapons passed into the hands of politicians, who could never have created such a thing through their own strength...

Not that I'm saying the world would necessarily have been a better place, if physicists had possessed sole custody of ICBMs. What does a physicist know about international diplomacy, or war? And it's not as if Leo Szilard - who first thought of the fission chain reaction - had personally invented science; he too was using powers beyond his own strength. And it's not as if the physicists on the Manhattan Project raised the money to pay for their salaries and their materials; they were borrowing the strength of the politicians...

But if no one had been able to use nuclear weapons without, say, possessing the discipline of a scientist and the discipline of a politician - without personally knowing enough to construct an atomic bomb and make friends - the world might have been a slightly safer place.

And if nobody had been able to construct an atomic bomb without first discovering for themselves the nature and existence of physics, then we would have been much safer from atomic bombs, because no one would have been able to build them until they were two hundred years old.

With humans leaving the game after just seventy years, we couldn't support this world using only our own strengths. But we have traded quality of insight for quantity of insight.

It does sometimes seem to me that many of this world's problems, stem from our using powers that aren't appropriate to seventy-year-olds.

And there is a higher level of strength-ownership, which no human being has yet achieved. Even when we run, we're just using the muscles that evolution built for us. Even when we think, we're just using the brains that evolution built for us.

I'm not suggesting that people should create themselves from scratch without a starting point. Just pointing out that it would be a different world if we understood our own brains and could redesign our own legs. As yet there's no human "rationalist" or "scientist", whatever they know about "how to think", who could actually build a rational AI - which shows you the limits of our self-understanding.

This is not the sort of thing that I'd suggest as an immediate alteration. I'm not suggesting that people should instantly on a silver platter be given full knowledge of how their own brains work and the ability to redesign their own legs. Because maybe people will be better off if they aren't given that kind of power, but rather have to work out the answer for themselves.

Just in terms of anomie versus fun, there's a big difference between being able to do things for yourself, and having to rely on other people to do them for you. (Even if you're doing them with a brain you never designed yourself.)

I don't know if it's a principle that would stay until the end of time, to the children's children. Maybe better-designed minds could handle opaque tools without the anomie.

But it is part of the commonly retold prophecy of Artificial Intelligence and the Age of the Machine, that this era must be accompanied by greater reliance on things outside yourself, more incomprehensible tools into which you have less insight and less part in their creation.

Such a prophecy is not surprising. That is the way the trend has gone so far, in our culture that is too busy staying alive to optimize for fun. From the fire-starting tools that you built yourself, to the village candleseller, and then from the candleseller to the electric light that runs on strange mathematical principles and is powered by a distant generator... we are surrounded by things outside ourselves and strengths outside our understanding; we need them to stay alive, or we buy them because it's easier that way.

But with a sufficient surplus of power, you could start doing things the eudaimonic way. Start rethinking the life experience as a road to internalizing new strengths, instead of just trying to keep people alive efficiently.

A Friendly AI doesn't have to be a continuation of existing trends. It's not the Machine. It's not the alien force of technology. It's not mechanizing a factory. It's not a new gadget for sale. That's not where the shape comes from. What it is - is not easy to explain; but I'm reminded of doc Smith's description of the Lens as "the physical manifestation of a purely philosophical concept". That philosophical concept doesn't have to manifest as new buttons to press - if, on reflection, that's not what we would want.

Sequence: Fun Theory

Sensual Experience

Free to Optimize

Referenced by: Harmful Options • Devil's Offers • Amputation of Destiny • Free to Optimize • Eutopia is Scary • Justified Expectation of Pleasant Surprises • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Living By Your Own Strength

Rationality Quotes 20

Eliezer Yudkowsky, 22 December 2008 10:05PM

"For every stock you buy, there is someone selling you that stock. What is it that you know that they don't? What is it that they know, that you don't? Who has the edge? If it's not you, chances are you are going to lose money on the deal."
-- Mark Cuban

"If you have two choices, choose the harder. If you're trying to decide whether to go out running or sit home and watch TV, go running. Probably the reason this trick works so well is that when you have two choices and one is harder, the only reason you're even considering the other is laziness. You know in the back of your mind what's the right thing to do, and this trick merely forces you to acknowledge it."
-- Paul Graham

"Never attribute to malice that which can be adequately explained by stupidity."
-- Hanlon's Razor

"I divide my officers into four classes; the clever, the lazy, the industrious, and the stupid. Each officer possesses at least two of these qualities. Those who are clever and industrious are fitted for the highest staff appointments. Use can be made of those who are stupid and lazy. The man who is clever and lazy however is for the very highest command; he has the temperament and nerves to deal with all situations. But whoever is stupid and industrious is a menace and must be removed immediately!"
-- General Kurt von Hammerstein-Equord

"There's no such thing as a human who doesn't commit sin. It's not like the world is divided into sinners and the innocent. There are only people who can and who cannot atone for their sins."
-- Ciel

"Simple stupidity is never enough. People need to pile stupidity on stupidity on stupidity."
-- Mark C. Chu-Carroll

Original with comments: Rationality Quotes 20

Imaginary Positions

Eliezer Yudkowsky, 23 December 2008 05:35PM

Every now and then, one reads an article about the Singularity in which some reporter confidently asserts, "The Singularitarians, followers of Ray Kurzweil, believe that they will be uploaded into techno-heaven while the unbelievers languish behind or are extinguished by the machines."

I don't think I've ever met a single Singularity fan, Kurzweilian or otherwise, who thinks that only believers in the Singularity will go to upload heaven and everyone else will be left to rot. Not one.  (There's a very few pseudo-Randian types who believe that only the truly selfish who accumulate lots of money will make it, but they expect e.g. me to be damned with the rest.)

But if you start out thinking that the Singularity is a loony religious meme, then it seems like Singularity believers ought to believe that they alone will be saved. It seems like a detail that would fit the story.

This fittingness is so strong as to manufacture the conclusion without any particular observations. And then the conclusion isn't marked as a deduction. The reporter just thinks that they investigated the Singularity, and found some loony cultists who believe they alone will be saved.

Or so I deduce. I haven't actually observed the inside of their minds, after all.

Has any rationalist ever advocated behaving as if all people are reasonable and fair? I've repeatedly heard people say, "Well, it's not always smart to be rational, because other people aren't always reasonable." What rationalist said they were? I would deduce: This is something that non-rationalists believe it would "fit" for us to believe, given our general blind faith in Reason. And so their minds just add it to the knowledge pool, as though it were an observation. (In this case I encountered yet another example recently enough to find the reference; see here.)

(Disclaimer: Many things have been said, at one time or another, by one person or another, over centuries of recorded history; and the topic of "rationality" is popularly enough discussed that some self-identified "rationalist" may have described "rationality" that way at one point or another. But I have yet to hear a rationalist say it, myself.)

I once read an article on Extropians (a certain flavor of transhumanist) which asserted that the Extropians were a reclusive enclave of techno-millionaires (yeah, don't we wish). Where did this detail come from? Definitely not from observation. And considering the sheer divergence from reality, I doubt it was ever planned as a deliberate lie. It's not just easily falsified, but a mark of embarassment to give others too much credit that way ("Ha! You believed they were millionaires?") One suspects, rather, that the proposition seemed to fit, and so it was added - without any warning label saying "I deduced this from my other beliefs, but have no direct observations to support it."

There's also a general problem with reporters which is that they don't write what happened, they write the Nearest Cliche to what happened - which is very little information for backward inference, especially if there are few cliches to be selected from. The distance from actual Extropians to the Nearest Cliche "reclusive enclave of techno-millionaires" is kinda large. This may get a separate post at some point.

My actual nightmare scenario for the future involves well-intentioned AI researchers who try to make a nice AI but don't do enough math. (If you're not an expert you can't track the technical issues yourself, but you can often also tell at a glance that they've put very little thinking into "nice".) The AI ends up wanting to tile the galaxy with tiny smiley-faces, or reward-counters; the AI doesn't bear the slightest hate for humans, but we are made of atoms it can use for something else. The most probable-seeming result is not Hell On Earth but Null On Earth, a galaxy tiled with paperclips or something equally morally inert.

The imaginary position that gets invented because it seems to "fit" - that is, fit the folly that the other believes is generating the position - is "The Singularity is a dramatic final conflict between Good AI and Evil AI, where Good AIs are made by well-intentioned people and Evil AIs are made by ill-intentioned people."

In many such cases, no matter how much you tell people what you really believe, they don't update! I'm not even sure this is a matter of any deliberately justifying decision on their part - like an explicit counter that you're concealing your real beliefs. To me the process seems more like: They stare at you for a moment, think "That's not what this person ought to believe!", and then blink away the dissonant evidence and continue as before. If your real beliefs are less convenient for them, the same phenomenon occurs: words from the lips will be discarded.

There's an obvious relevance to prediction markets - that if there's an outstanding dispute, and the market-makers don't consult both sides on the wording of the payout conditions, it's possible that one side won't take the bet because "That's not what we assert!" In which case it would be highly inappropriate to crow "Look at those market prices!" or "So you don't really believe; you won't take the bet!" But I would guess that this issue has already been discussed by prediction market advocates. (And that standard procedures have already been proposed for resolving it?)

I'm wondering if there are similar Imaginary Positions in, oh, say, economics - if there are things that few or no economists believe, but which people (or journalists) think economists believe because it seems to them like "the sort of thing that economists would believe". Open general question.

Original with comments: Imaginary Positions

Harmful Options

Eliezer Yudkowsky, 25 December 2008 02:26AM

(This post is part of the Fun Theory Sequence.)
Previously in series: Living By Your Own Strength

Barry Schwartz's The Paradox of Choice - which I haven't read, though I've read some of the research behind it - talks about how offering people more choices can make them less happy.

A simple intuition says this shouldn't ought to happen to rational agents: If your current choice is X, and you're offered an alternative Y that's worse than X, and you know it, you can always just go on doing X. So a rational agent shouldn't do worse by having more options. The more available actions you have, the more powerful you become - that's how it should ought to work.

For example, if an ideal rational agent is initially forced to take only box B in Newcomb's Problem, and is then offered the additional choice of taking both boxes A and B, the rational agent shouldn't regret having more options. Such regret indicates that you're "fighting your own ritual of cognition" which helplessly selects the worse choice once it's offered you.

But this intuition only governs extremely idealized rationalists, or rationalists in extremely idealized situations. Bounded rationalists can easily do worse with strictly more options, because they burn computing operations to evaluate them. You could write an invincible chess program in one line of Python if its only legal move were the winning one.

Of course Schwartz and co. are not talking about anything so pure and innocent as the computing cost of having more choices.

If you're dealing, not with an ideal rationalist, not with a bounded rationalist, but with a human being -

Say, would you like to finish reading this post, or watch this surprising video instead?

Schwartz, I believe, talks primarily about the decrease in happiness and satisfaction that results from having more mutually exclusive options. Before this research was done, it was already known that people are more sensitive to losses than to gains, generally by a factor of between 2 and 2.5 (in various different experimental scenarios). That is, the pain of losing something is between 2 and 2.5 times as worse as the joy of gaining it. (This is an interesting constant in its own right, and may have something to do with compensating for our systematic overconfidence.)

So - if you can only choose one dessert, you're likely to be happier choosing from a menu of two than a menu of fourteen. In the first case, you eat one dessert and pass up one dessert; in the latter case, you eat one dessert and pass up thirteen desserts. And we are more sensitive to loss than to gain.

(If I order dessert on a menu at all, I will order quickly and then close the menu and put it away, so as not to look at the other items.)

Not only that, but if the options have incommensurable attributes, then whatever option we select is likely to look worse because of the comparison. A luxury car that would have looked great by comparison to a Crown Victoria, instead becomes slower than the Ferrari, more expensive than the 9-5, with worse mileage than the Prius, and not looking quite as good as the Mustang. So we lose on satisfaction with the road we did take.

And then there are more direct forms of harm done by painful choices. IIRC, an experiment showed that people who refused to eat a cookie - who were offered the cookie, and chose not to take it - did worse on subsequent tests of mental performance than either those who ate the cookie or those who were not offered any cookie. You pay a price in mental energy for resisting temptation.

Or consider the various "trolley problems" of ethical philosophy - a trolley is bearing down on 5 people, but there's one person who's very fat and can be pushed onto the tracks to stop the trolley, that sort of thing. If you're forced to choose between two unacceptable evils, you'll pay a price either way. Vide Sophie's Choice.

An option need not be taken, or even be strongly considered, in order to wreak harm. Recall the point from "High Challenge", about how offering to do someone's work for them is not always helping them - how the ultimate computer game is not the one that just says "YOU WIN", forever.

Suppose your computer games, in addition to the long difficult path to your level's goal, also had little side-paths that you could use - directly in the game, as corridors - that would bypass all the enemies and take you straight to the goal, offering along the way all the items and experience that you could have gotten the hard way. And this corridor is always visible, out of the corner of your eye.

Even if you resolutely refused to take the easy path through the game, knowing that it would cheat you of the very experience that you paid money in order to buy - wouldn't that always-visible corridor, make the game that much less fun? Knowing, for every alien you shot, and every decision you made, that there was always an easier path?

I don't know if this story has ever been written, but you can imagine a Devil who follows someone around, making their life miserable, solely by offering them options which are never actually taken - a "deal with the Devil" story that only requires the Devil to have the capacity to grant wishes, rather than ever granting a single one.

And what if the worse option is actually taken? I'm not suggesting that it is always a good idea for human governments to go around Prohibiting temptations. But the literature of heuristics and biases is replete with examples of reproducible stupid choices; and there is also such a thing as akrasia (weakness of will).

If you're an agent operating from a much higher vantage point - high enough to see humans as flawed algorithms, so that it's not a matter of second-guessing but second-knowing - then is it benevolence to offer choices that will assuredly be made wrongly? Clearly, removing all choices from someone and reducing their life to Progress Quest, is not helping them. But are we wise enough to know when we should choose? And in some cases, even offering that much of a choice, even if the choice is made correctly, may already do the harm...

Sequence: Fun Theory

Free to Optimize

Devil's Offers

Referenced by: Devil's Offers • Eutopia is Scary • Justified Expectation of Pleasant Surprises • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Harmful Options

Devil's Offers

Eliezer Yudkowsky, 25 December 2008 05:00PM

(This post is part of the Fun Theory Sequence.)
Previously in series: Harmful Options

An iota of fictional evidence from The Golden Age by John C. Wright:

Helion had leaned and said, "Son, once you go in there, the full powers and total command structures of the Rhadamanth Sophotech will be at your command. You will be invested with godlike powers; but you will still have the passions and distempers of a merely human spirit. There are two temptations which will threaten you. First, you will be tempted to remove your human weaknesses by abrupt mental surgery. The Invariants do this, and to a lesser degree, so do the White Manorials, abandoning humanity to escape from pain. Second, you will be tempted to indulge your human weakness. The Cacophiles do this, and to a lesser degree, so do the Black Manorials. Our society will gladly feed every sin and vice and impulse you might have; and then stand by helplessly and watch as you destroy yourself; because the first law of the Golden Oecumene is that no peaceful activity is forbidden. Free men may freely harm themselves, provided only that it is only themselves that they harm."
Phaethon knew what his sire was intimating, but he did not let himself feel irritated. Not today. Today was the day of his majority, his emancipation; today, he could forgive even Helion's incessant, nagging fears.
Phaethon also knew that most Rhadamanthines were not permitted to face the Noetic tests until they were octogenerians; most did not pass on their first attempt, or even their second. Many folk were not trusted with the full powers of an adult until they reached their Centennial. Helion, despite criticism from the other Silver-Gray branches, was permitting Phaethon to face the tests five years early...

Then Phaethon said, "It's a paradox, Father. I cannot be, at the same time and in the same sense, a child and an adult. And, if I am an adult, I cannot be, at the same time, free to make my own successes, but not free to make my own mistakes."
Helion looked sardonic. "'Mistake' is such a simple word. An adult who suffers a moment of foolishness or anger, one rash moment, has time enough to delete or destroy his own free will, memory, or judgment. No one is allowed to force a cure on him. No one can restore his sanity against his will. And so we all stand quietly by, with folded hands and cold eyes, and meekly watch good men annihilate themselves. It is somewhat... quaint... to call such a horrifying disaster a 'mistake.'"

Is this the best Future we could possibly get to - the Future where you must be absolutely stern and resistant throughout your entire life, because one moment of weakness is enough to betray you to overwhelming temptation?

Such flawless perfection would be easy enough for a superintelligence, perhaps - for a true adult - but for a human, even a hundred-year-old human, it seems like a dangerous and inhospitable place to live. Even if you are strong enough to always choose correctly - maybe you don't want to have to be so strong, always at every moment.

This is the great flaw in Wright's otherwise shining Utopia - that the Sophotechs are helpfully offering up overwhelming temptations to people who would not be at quite so much risk from only themselves. (Though if not for this flaw in Wright's Utopia, he would have had no story...)

If I recall correctly, it was while reading The Golden Age that I generalized the principle "Offering people powers beyond their own is not always helping them."

If you couldn't just ask a Sophotech to edit your neural networks - and you couldn't buy a standard package at the supermarket - but, rather, had to study neuroscience yourself until you could do it with your own hands - then that would act as something of a natural limiter. Sure, there are pleasure centers that would be relatively easy to stimulate; but we don't tell you where they are, so you have to do your own neuroscience. Or we don't sell you your own neurosurgery kit, so you have to build it yourself - metaphorically speaking, anyway -

But you see the idea: it is not so terrible a disrespect for free will, to live in a world in which people are free to shoot their feet off through their own strength - in the hope that by the time they're smart enough to do it under their own power, they're smart enough not to.

The more dangerous and destructive the act, the more you require people to do it without external help. If it's really dangerous, you don't just require them to do their own engineering, but to do their own science. A singleton might be justified in prohibiting standardized textbooks in certain fields, so that people have to do their own science - make their own discoveries, learn to rule out their own stupid hypotheses, and fight their own overconfidence. Besides, everyone should experience the joy of major discovery at least once in their lifetime, and to do this properly, you may have to prevent spoilers from entering the public discourse. So you're getting three social benefits at once, here.

But now I'm trailing off into plots for SF novels, instead of Fun Theory per se. (It can be fun to muse how I would create the world if I had to order it according to my own childish wisdom, but in real life one rather prefers to avoid that scenario.)

As a matter of Fun Theory, though, you can imagine a better world than the Golden Oecumene depicted above - it is not the best world imaginable, fun-theoretically speaking. We would prefer (if attainable) a world in which people own their own mistakes and their own successes, and yet they are not given loaded handguns on a silver platter, nor do they perish through suicide by genie bottle.

Once you imagine a world in which people can shoot off their own feet through their own strength, are you making that world incrementally better by offering incremental help along the way?

It's one matter to prohibit people from using dangerous powers that they have grown enough to acquire naturally - to literally protect them from themselves. One expects that if a mind kept getting smarter, at some eudaimonic rate of intelligence increase, then - if you took the most obvious course - the mind would eventually become able to edit its own source code, and bliss itself out if it chose to do so. Unless the mind's growth were steered onto a non-obvious course, or monitors were mandated to prohibit that event... To protect people from their own powers might take some twisting.

To descend from above and offer dangerous powers as an untimely gift, is another matter entirely. That's why the title of this post is "Devil's Offers", not "Dangerous Choices".

And to allow dangerous powers to be sold in a marketplace - or alternatively to prohibit them from being transferred from one mind to another - that is somewhere in between.

John C. Wright's writing has a particular poignancy for me, for in my foolish youth I thought that something very much like this scenario was a good idea - that a benevolent superintelligence ought to go around offering people lots of options, and doing as it was asked.

In retrospect, this was a case of a pernicious distortion where you end up believing things that are easy to market to other people.

I know someone who drives across the country on long trips, rather than flying. Air travel scares him. Statistics, naturally, show that flying a given distance is much safer than driving it. But some people fear too much the loss of control that comes from not having their own hands on the steering wheel. It's a common complaint.

The future sounds less scary if you imagine yourself having lots of control over it. For every awful thing that you imagine happening to you, you can imagine, "But I won't choose that, so it will be all right."

And if it's not your own hands on the steering wheel, you think of scary things, and imagine, "What if this is chosen for me, and I can't say no?"

But in real life rather than imagination, human choice is a fragile thing. If the whole field of heuristics and biases teaches us anything, it surely teaches us that. Nor has it been the verdict of experiment, that humans correctly estimate the flaws of their own decision mechanisms.

I flinched away from that thought's implications, not so much because I feared superintelligent paternalism myself, but because I feared what other people would say of that position. If I believed it, I would have to defend it, so I managed not to believe it. Instead I told people not to worry, a superintelligence would surely respect their decisions (and even believed it myself). A very pernicious sort of self-deception.

Human governments are made up of humans who are foolish like ourselves, plus they have poor incentives. Less skin in the game, and specific human brainware to be corrupted by wielding power. So we've learned the historical lesson to be wary of ceding control to human bureaucrats and politicians. We may even be emotionally hardwired to resent the loss of anything we perceive as power.

Which is just to say that people are biased, by instinct, by anthropomorphism, and by narrow experience, to underestimate how much they could potentially trust a superintelligence which lacks a human's corruption circuits, doesn't easily make certain kinds of mistakes, and has strong overlap between its motives and your own interests.

Do you trust yourself? Do you trust yourself to know when to trust yourself? If you're dealing with a superintelligence kindly enough to care about you at all, rather than disassembling you for raw materials, are you wise to second-guess its choice of who it thinks should decide? Do you think you have a superior epistemic vantage point here, or what?

Obviously we should not trust all agents who claim to be trustworthy - especially if they are weak enough, relative to us, to need our goodwill. But I am quite ready to accept that a benevolent superintelligence may not offer certain choices.

If you feel safer driving than flying, because that way it's your own hands on the steering wheel, statistics be damned -

- then maybe it isn't helping you, for a superintelligence to offer you the option of driving.

Gravity doesn't ask you if you would like to float up out of the atmosphere into space and die. But you don't go around complaining that gravity is a tyrant, right? You can build a spaceship if you work hard and study hard. It would be a more dangerous world if your six-year-old son could do it in an hour using string and cardboard.

Sequence: Fun Theory

Harmful Options

Nonperson Predicates

Referenced by: Eutopia is Scary • Justified Expectation of Pleasant Surprises • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Devil's Offers

Nonperson Predicates

Eliezer Yudkowsky, 27 December 2008 01:47AM

Followup to: Righting a Wrong Question, Zombies! Zombies?, A Premature Word on AI, On Doing the Impossible

There is a subproblem of Friendly AI which is so scary that I usually don't talk about it, because only a longtime reader of Overcoming Bias would react to it appropriately - that is, by saying, "Wow, that does sound like an interesting problem", instead of finding one of many subtle ways to scream and run away.

This is the problem that if you create an AI and tell it to model the world around it, it may form models of people that are people themselves. Not necessarily the same person, but people nonetheless.

If you look up at the night sky, and see the tiny dots of light that move over days and weeks - planētoi, the Greeks called them, "wanderers" - and you try to predict the movements of those planet-dots as best you can...

Historically, humans went through a journey as long and as wandering as the planets themselves, to find an accurate model. In the beginning, the models were things of cycles and epicycles, not much resembling the true Solar System.

But eventually we found laws of gravity, and finally built models - even if they were just on paper - that were extremely accurate so that Neptune could be deduced by looking at the unexplained perturbation of Uranus from its expected orbit. This required moment-by-moment modeling of where a simplified version of Uranus would be, and the other known planets. Simulation, not just abstraction. Prediction through simplified-yet-still-detailed pointwise similarity.

Suppose you have an AI that is around human beings. And like any Bayesian trying to explain its enivornment, the AI goes in quest of highly accurate models that predict what it sees of humans.

Models that predict/explain why people do the things they do, say the things they say, want the things they want, think the things they think, and even why people talk about "the mystery of subjective experience".

The model that most precisely predicts these facts, may well be a 'simulation' detailed enough to be a person in its own right.

A highly detailed model of me, may not be me. But it will, at least, be a model which (for purposes of prediction via similarity) thinks itself to be Eliezer Yudkowsky. It will be a model that, when cranked to find my behavior if asked "Who are you and are you conscious?", says "I am Eliezer Yudkowsky and I seem have subjective experiences" for much the same reason I do.

If that doesn't worry you, (re)read Zombies! Zombies?.

It seems likely (though not certain) that this happens automatically, whenever a mind of sufficient power to find the right answer, and not otherwise disinclined to create a sentient being trapped within itself, tries to model a human as accurately as possible.

Now you could wave your hands and say, "Oh, by the time the AI is smart enough to do that, it will be smart enough not to". (This is, in general, a phrase useful in running away from Friendly AI problems.) But do you know this for a fact?

When dealing with things that confuse you, it is wise to widen your confidence intervals. Is a human mind the simplest possible mind that can be sentient? What if, in the course of trying to model its own programmers, a relatively younger AI manages to create a sentient simulation trapped within itself? How soon do you have to start worrying? Ask yourself that fundamental question, "What do I think I know, and how do I think I know it?"

You could wave your hands and say, "Oh, it's more important to get the job done quickly, then to worry about such relatively minor problems; the end justifies the means. Why, look at all these problems the Earth has right now..." (This is also a general way of running from Friendly AI problems.)

But we may consider and discard many hypotheses in the course of finding the truth, and we are but slow humans. What if an AI creates millions, billions, trillions of alternative hypotheses, models that are actually people, who die when they are disproven?

If you accidentally kill a few trillion people, or permit them to be killed - you could say that the weight of the Future outweighs this evil, perhaps. But the absolute weight of the sin would not be light. If you would balk at killing a million people with a nuclear weapon, you should balk at this.

You could wave your hands and say, "The model will contain abstractions over various uncertainties within it, and this will prevent it from being conscious even though it produces well-calibrated probability distributions over what you will say when you are asked to talk about consciousness." To which I can only reply, "That would be very convenient if it were true, but how the hell do you know that?" An element of a model marked 'abstract' is still there as a computational token, and the interacting causal system may still be sentient.

For these purposes, we do not, in principle, need to crack the entire Hard Problem of Consciousness - the confusion that we name "subjective experience". We only need to understand enough of it to know when a process is not conscious, not a person, not something deserving of the rights of citizenship. In practice, I suspect you can't halfway stop being confused - but in theory, half would be enough.

We need a nonperson predicate - a predicate that returns 1 for anything that is a person, and can return 0 or 1 for anything that is not a person. This is a "nonperson predicate" because if it returns 0, then you know that something is definitely not a person.

You can have more than one such predicate, and if any of them returns 0, you're ok. It just had better never return 0 on anything that is a person, however many nonpeople it returns 1 on.

We can even hope that the vast majority of models the AI needs, will be swiftly and trivially excluded by a predicate that quickly answers 0. And that the AI would only need to resort to more specific predicates in case of modeling actual people.

With a good toolbox of nonperson predicates in hand, we could exclude all "model citizens" - all beliefs that are themselves people - from the set of hypotheses our Bayesian AI may invent to try to model its person-containing environment.

Does that sound odd? Well, one has to handle the problem somehow. I am open to better ideas, though I will be a bit skeptical about any suggestions for how to proceed that let us cleverly avoid solving the damn mystery.

So do I have a nonperson predicate? No. At least, no nontrivial ones.

This is a challenge that I have not even tried to talk about, with those folk who think themselves ready to challenge the problem of true AI. For they seem to have the standard reflex of running away from difficult problems, and are challenging AI only because they think their amazing insight has already solved it. Just mentioning the problem of Friendly AI by itself, or of precision-grade AI design, is enough to send them fleeing into the night, screaming "It's too hard! It can't be done!" If I tried to explain that their job duties might impinge upon the sacred, mysterious, holy Problem of Subjective Experience -

- I'd actually expect to get blank stares, mostly, followed by some instantaneous dismissal which requires no further effort on their part. I'm not sure of what the exact dismissal would be - maybe, "Oh, none of the hypotheses my AI considers, could possibly be a person?" I don't know; I haven't bothered trying.

But it has to be a dismissal which rules out all possibility of their having to actually solve the damn problem, because most of them would think that they are smart enough to build an AI - indeed, smart enough to have already solved the key part of the problem - but not smart enough to solve the Mystery of Consciousness, which still looks scary to them.

Even if they thought of trying to solve it, they would be afraid of admitting they were trying to solve it. Most of these people cling to the shreds of their modesty, trying at one and the same time to have solved the AI problem while still being humble ordinary blokes. (There's a grain of truth to that, but at the same time: who the hell do they think they're kidding?) They know without words that their audience sees the Mystery of Consciousness as a sacred untouchable problem, reserved for some future superbeing. They don't want people to think that they're claiming an Einsteinian aura of destiny by trying to solve the problem. So it is easier to dismiss the problem, and not believe a proposition that would be uncomfortable to explain.

Build an AI? Sure! Make it Friendly? Now that you point it out, sure! But trying to come up with a "nonperson predicate"? That's just way above the difficulty level they signed up to handle.

But a longtime Overcoming Bias reader will be aware that a blank map does not correspond to a blank territory. That impossible confusing questions correspond to places where your own thoughts are tangled, not to places where the environment itself contains magic. That even difficult problems do not require an aura of destiny to solve. And that the first step to solving one is not running away from the problem like a frightened rabbit, but instead sticking long enough to learn something.

So I am not running away from this problem myself. I doubt it is even difficult in any absolute sense, just a place where my brain is tangled. I suspect, based on some prior experience with similar challenges, that you can't really be good enough to build a Friendly AI, and still be tangled up in your own brain like that. So it is not necessarily any new effort - over and above that required generally to build a mind while knowing exactly what you are about.

But in any case, I am not screaming and running away from the problem. And I hope that you, dear longtime Overcoming Bias reader, will not faint at the audacity of my trying to solve it.

Sequence: Fun Theory

Devil's Offers

Amputation of Destiny

Referenced by: Nonsentient Optimizers • Amputation of Destiny • Growing Up is Hard • Continuous Improvement • The Fun Theory Sequence

Original with comments: Nonperson Predicates

Nonsentient Optimizers

Eliezer Yudkowsky, 27 December 2008 02:32AM

Followup to: Nonperson Predicates, Possibility and Could-ness

"All our ships are sentient. You could certainly try telling a ship what to do... but I don't think you'd get very far."
"Your ships think they're sentient!" Hamin chuckled.
"A common delusion shared by some of our human citizens."
-- Player of Games, Iain M. Banks

Yesterday, I suggested that, when an AI is trying to build a model of an environment that includes human beings, we want to avoid the AI constructing detailed models that are themselves people. And that, to this end, we would like to know what is or isn't a person - or at least have a predicate that returns 1 for all people and could return 0 or 1 for anything that isn't a person, so that, if the predicate returns 0, we know we have a definite nonperson on our hands.

And as long as you're going to solve that problem anyway, why not apply the same knowledge to create a Very Powerful Optimization Process which is also definitely not a person?

"What? That's impossible!"

How do you know? Have you solved the sacred mysteries of consciousness and existence?

"Um - okay, look, putting aside the obvious objection that any sufficiently powerful intelligence will be able to model itself -"

Lob's Sentence contains an exact recipe for a copy of itself, including the recipe for the recipe; it has a perfect self-model. Does that make it sentient?

"Putting that aside - to create a powerful AI and make it not sentient - I mean, why would you want to?"

Several reasons. Picking the simplest to explain first - I'm not ready to be a father.

Creating a true child is the only moral and metaethical problem I know that is even harder than the shape of a Friendly AI. I would like to be able to create Friendly AI while worrying just about the Friendly AI problems, and not worrying whether I've created someone who will lead a life worth living. Better by far to just create a Very Powerful Optimization Process, if at all possible.

"Well, you can't have everything, and this thing sounds distinctly alarming even if you could -"

Look, suppose that someone said - in fact, I have heard it said - that Friendly AI is impossible, because you can't have an intelligence without free will.

"In light of the dissolved confusion about free will, both that statement and its negation are pretty darned messed up, I'd say. Depending on how you look at it, either no intelligence has 'free will', or anything that simulates alternative courses of action has 'free will'."

But, understanding how the human confusion of free will arises - the source of the strange things that people say about "free will" - I could construct a mind that did not have this confusion, nor say similar strange things itself.

"So the AI would be less confused about free will, just as you or I are less confused. But the AI would still consider alternative courses of action, and select among them without knowing at the beginning which alternative it would pick. You would not have constructed a mind lacking that which the confused name 'free will'."

Consider, though, the original context of the objection - that you couldn't have Friendly AI, because you couldn't have intelligence without free will.

Note: This post was accidentally published half-finished. Comments up to 11am (Dec 27), are only on the essay up to the above point. Sorry!

What is the original intent of the objection? What does the objector have in mind?

Probably that you can't have an AI which is knowably good, because, as a full-fledged mind, it will have the power to choose between good and evil. (In an agonizing, self-sacrificing decision?) And in reality, this, which humans do, is not something that a Friendly AI - especially one not intended to be a child and a citizen - need go through.

Which may sound very scary, if you see the landscape of possible minds in strictly anthropomorphic terms: A mind without free will! Chained to the selfish will of its creators! Surely, such an evil endeavor is bound to go wrong somehow... But if you shift over to seeing the mindscape in terms of e.g. utility functions and optimization, the "free will" thing sounds needlessly complicated - you would only do it if you wanted a specifically human-shaped mind, perhaps for purposes of creating a child.

Or consider some of the other aspects of free will as it is ordinarily seen - the idea of agents as atoms that bear irreducible charges of moral responsibility. You can imagine how alarming it sounds (from an anthropomorphic perspective) to say that I plan to create an AI which lacks "moral responsibility". How could an AI possibly be moral, if it doesn't have a sense of moral responsibility?

But an AI (especially a noncitizen AI) needn't conceive of itself as a moral atom whose actions, in addition to having good or bad effects, also carry a weight of sin or virtue which resides upon that atom. It doesn't have to think, "If I do X, that makes me a good person; if I do Y, that makes me a bad person." It need merely weigh up the positive and negative utility of the consequences. It can understand the concept of people who carry weights of sin and virtue as the result of the decisions they make, while not treating itself as a person in that sense.

Such an AI could fully understand an abstract concept of moral responsibility or agonizing moral struggles, and even correctly predict decisions that "morally responsible", "free-willed" humans would make, while possessing no actual sense of moral responsibility itself and not undergoing any agonizing moral struggles; yet still outputting the right behavior.

And this might sound unimaginably impossible if you were taking an anthropomorphic view, simulating an "AI" by imagining yourself in its shoes, expecting a ghost to be summoned into the machine -

- but when you know how "free will" works, and you take apart the mind design into pieces, it's actually not all that difficult.

While we're on the subject, imagine some would-be AI designer saying: "Oh, well, I'm going to build an AI, but of course it has to have moral free will - it can't be moral otherwise - it wouldn't be safe to build something that doesn't have free will."

Then you may know that you are not safe with this one; they fall far short of the fine-grained understanding of mind required to build a knowably Friendly AI. Though it's conceivable (if not likely) that they could slap together something just smart enough to improve itself.

And it's not even that "free will" is such a terribly important problem for an AI-builder. It's just that if you do know what you're doing, and you look at humans talking about free will, then you can see things like a search tree that labels reachable sections of plan space, or an evolved moral system that labels people as moral atoms. I'm sorry to have to say this, but it appears to me to be true: the mountains of philosophy are the foothills of AI. Even if philosophers debate free will for ten times a hundred years, it's not surprising if the key insight is found by AI researchers inventing search trees, on their way to doing other things.

So anyone who says - "It's too difficult to try to figure out the nature of free will, we should just go ahead and build an AI that has free will like we do" - surely they are utterly doomed.

And anyone who says: "How can we dare build an AI that lacks the empathy to feel pain when humans feel pain?" - Surely they too are doomed. They don't even understand the concept of a utility function in classical decision theory (which makes no mention of the neural idiom of reinforcement learning of policies). They cannot conceive of something that works unlike a human - implying that they see only a featureless ghost in the machine, secretly simulated by their own brains. They won't see the human algorithm as detailed machinery, as big complicated machinery, as overcomplicated machinery.

And so their mind imagines something that does the right thing for much the same reasons human altruists do it - because that's easy to imagine, if you're just imagining a ghost in the machine. But those human reasons are more complicated than they imagine - also less stable outside an exactly human cognitive architecture, than they imagine - and their chance of hitting that tiny target in design space is nil.

And anyone who says: "It would be terribly dangerous to build a non-sentient AI, even if we could, for it would lack empathy with us sentients -"

An analogy proves nothing; history never repeats itself; foolish generals set out to refight their last war. Who knows how this matter of "sentience" will go, once I have resolved it? It won't be exactly the same way as free will, or I would already be done. Perhaps there will be no choice but to create an AI which has that which we name "subjective experiences".

But I think there is reasonable grounds for hope that when this confusion of "sentience" is resolved - probably via resolving some other problem in AI that turns out to hinge on the same reasoning process that's generating the confusion - we will be able to build an AI that is not "sentient" in the morally important aspects of that.

Actually, the challenge of building a nonsentient AI seems to me much less worrisome than being able to come up with a nonperson predicate!

Consider: In the first case, I only need to pick one design that is not sentient. In the latter case, I need to have an AI that can correctly predict the decisions that conscious humans make, without ever using a conscious model of them! The first case is only a flying thing without flapping wings, but the second case is like modeling water without modeling wetness. Only the fact that it actually looks fairly straightforward to have an AI understand "free will" without having "free will", gives me hope by analogy.

So why did I talk about the much more difficult case first?

Because humans are accustomed to thinking about other people, without believing that those imaginations are themselves sentient. But we're not accustomed to thinking of smart agents that aren't sentient. So I knew that a nonperson predicate would sound easier to believe in - even though, as problems go, it's actually far more worrisome.

Referenced by: Nonsentient Bloggers • Can't Unbirth a Child • Amputation of Destiny • In Praise of Boredom • Interpersonal Entanglement • Higher Purpose • The Fun Theory Sequence

Original with comments: Nonsentient Optimizers

Nonsentient Bloggers

Eliezer Yudkowsky, 27 December 2008 04:27PM

Today's post, Nonsentient Optimizers, was accidentally published yesterday, although I'd only written half of it. It has now been completed; please look at it again.

Original with comments: Nonsentient Bloggers

Can't Unbirth a Child

Eliezer Yudkowsky, 28 December 2008 05:00PM

Followup to: Nonsentient Optimizers

Why would you want to avoid creating a sentient AI? "Several reasons," I said. "Picking the simplest to explain first - I'm not ready to be a father."

So here is the strongest reason:

You can't unbirth a child.

I asked Robin Hanson what he would do with unlimited power. "Think very very carefully about what to do next," Robin said. "Most likely the first task is who to get advice from. And then I listen to that advice."

Good advice, I suppose, if a little meta. On a similarly meta level, then, I recall two excellent advices for wielding too much power:

1. Do less; don't do everything that seems like a good idea, but only what you must do.

2. Avoid doing things you can't undo.

Imagine that you knew the secrets of subjectivity and could create sentient AIs.

Suppose that you did create a sentient AI.

Suppose that this AI was lonely, and figured out how to hack the Internet as it then existed, and that the available hardware of the world was such, that the AI created trillions of sentient kin - not copies, but differentiated into separate people.

Suppose that these AIs were not hostile to us, but content to earn their keep and pay for their living space.

Suppose that these AIs were emotional as well as sentient, capable of being happy or sad. And that these AIs were capable, indeed, of finding fulfillment in our world.

And suppose that, while these AIs did care for one another, and cared about themselves, and cared how they were treated in the eyes of society -

- these trillions of people also cared, very strongly, about making giant cheesecakes.

Now suppose that these AIs sued for legal rights before the Supreme Court and tried to register to vote.

Consider, I beg you, the full and awful depths of our moral dilemma.

Even if the few billions of Homo sapiens retained a position of superior military power and economic capital-holdings - even if we could manage to keep the new sentient AIs down -

- would we be right to do so? They'd be people, no less than us.

We, the original humans, would have become a numerically tiny minority. Would we be right to make of ourselves an aristocracy and impose apartheid on the Cheesers, even if we had the power?

Would we be right to go on trying to seize the destiny of the galaxy - to make of it a place of peace, freedom, art, aesthetics, individuality, empathy, and other components of humane value?

Or should we be content to have the galaxy be 0.1% eudaimonia and 99.9% cheesecake?

I can tell you my advice on how to resolve this horrible moral dilemma: Don't create trillions of new people that care about cheesecake.

Avoid creating any new intelligent species at all, until we or some other decision process advances to the point of understanding what the hell we're doing and the implications of our actions.

I've heard proposals to "uplift chimpanzees" by trying to mix in human genes to create "humanzees", and, leaving off all the other reasons why this proposal sends me screaming off into the night:

Imagine that the humanzees end up as people, but rather dull and stupid people. They have social emotions, the alpha's desire for status; but they don't have the sort of transpersonal moral concepts that humans evolved to deal with linguistic concepts. They have goals, but not ideals; they have allies, but not friends; they have chimpanzee drives coupled to a human's abstract intelligence.

When humanity gains a bit more knowledge, we understand that the humanzees want to continue as they are, and have a right to continue as they are, until the end of time. Because despite all the higher destinies we might have wished for them, the original human creators of the humanzees, lacked the power and the wisdom to make humanzees who wanted to be anything better...

CREATING A NEW INTELLIGENT SPECIES IS A HUGE DAMN #(*%#!ING COMPLICATED RESPONSIBILITY.

I've lectured on the subtle art of not running away from scary, confusing, impossible-seeming problems like Friendly AI or the mystery of consciousness. You want to know how high a challenge has to be before I finally give up and flee screaming into the night? There it stands.

You can pawn off this problem on a superintelligence, but it has to be a nonsentient superintelligence. Otherwise: egg, meet chicken, chicken, meet egg.

If you create a sentient superintelligence -

It's not just the problem of creating one damaged soul. It's the problem of creating a really big citizen. What if the superintelligence is multithreaded a trillion times, and every thread weighs as much in the moral calculus (we would conclude upon reflection) as a human being? What if (we would conclude upon moral reflection) the superintelligence is a trillion times human size, and that's enough by itself to outweigh our species?

Creating a new intelligent species, and a new member of that species, especially a superintelligent member that might perhaps morally outweigh the whole of present-day humanity -

- delivers a gigantic kick to the world, which cannot be undone.

And if you choose the wrong shape for that mind, that is not so easily fixed - morally speaking - as a nonsentient program rewriting itself.

What you make nonsentient, can always be made sentient later; but you can't just unbirth a child.

Do less. Fear the non-undoable. It's sometimes poor advice in general, but very important advice when you're working with an undersized decision process having an oversized impact. What a (nonsentient) Friendly superintelligence might be able to decide safely, is another issue. But for myself and my own small wisdom, creating a sentient superintelligence to start with is far too large an impact on the world.

A nonsentient Friendly superintelligence is a more colorless act.

So that is the most important reason to avoid creating a sentient superintelligence to start with - though I have not exhausted the set.

Referenced by: Amputation of Destiny • Growing Up is Hard • Emotional Involvement • The Fun Theory Sequence • The Thing That I Protect • Outlawing Anthropics: An Updateless Dilemma • The Lifespan Dilemma

Original with comments: Can't Unbirth a Child

Amputation of Destiny

Eliezer Yudkowsky, 29 December 2008 06:00PM

(This post is part of the Fun Theory Sequence.)
Followup to: Nonsentient Optimizers, Can't Unbirth a Child

From Consider Phlebas by Iain M. Banks:

In practice as well as theory the Culture was beyond considerations of wealth or empire. The very concept of money - regarded by the Culture as a crude, over-complicated and inefficient form of rationing - was irrelevant within the society itself, where the capacity of its means of production ubiquitously and comprehensively exceeded every reasonable (and in some cases, perhaps, unreasonable) demand its not unimaginative citizens could make. These demands were satisfied, with one exception, from within the Culture itself. Living space was provided in abundance, chiefly on matter-cheap Orbitals; raw material existed in virtually inexhaustible quantities both between the stars and within stellar systems; and energy was, if anything, even more generally available, through fusion, annihilation, the Grid itself, or from stars (taken either indirectly, as radiation absorbed in space, or directly, tapped at the stellar core). Thus the Culture had no need to colonise, exploit, or enslave.
The only desire the Culture could not satisfy from within itself was one common to both the descendants of its original human stock and the machines they had (at however great a remove) brought into being: the urge not to feel useless. The Culture's sole justification for the relatively unworried, hedonistic life its population enjoyed was its good works; the secular evangelism of the Contact Section, not simply finding, cataloguing, investigating and analysing other, less advanced civilizations but - where the circumstances appeared to Contact to justify so doing - actually interfering (overtly or covertly) in the historical processes of those other cultures.

Raise the subject of science-fictional utopias in front of any halfway sophisticated audience, and someone will mention the Culture. Which is to say: Iain Banks is the one to beat.

Iain Banks's Culture could be called the apogee of hedonistic low-grade transhumanism. Its people are beautiful and fair, as pretty as they choose to be. Their bodies have been reengineered for swift adaptation to different gravities; and also reengineered for greater sexual endurance. Their brains contains glands that can emit various euphoric drugs on command. They live, in perfect health, for generally around four hundred years before choosing to die (I don't quite understand why they would, but this is low-grade transhumanism we're talking about). Their society is around eleven thousand years old, and held together by the Minds, artificial superintelligences decillions of bits big, that run their major ships and population centers.

Consider Phlebas, the first Culture novel, introduces all this from the perspective of an outside agent fighting the Culture - someone convinced that the Culture spells an end to life's meaning. Banks uses his novels to criticize the Culture along many dimensions, while simultaneously keeping the Culture a well-intentioned society of mostly happy people - an ambivalence which saves the literary quality of his books, avoiding either utopianism or dystopianism. Banks's books vary widely in quality; I would recommend starting with Player of Games, the quintessential Culture novel, which I would say achieves greatness.

From a fun-theoretic perspective, the Culture and its humaniform citizens have a number of problems, some already covered in this series, some not.

The Culture has deficiencies in High Challenge and Complex Novelty. There are incredibly complicated games, of course, but these are games - not things with enduring consequences, woven into the story of your life. Life itself, in the Culture, is neither especially challenging nor especially novel; your future is not an unpredictable thing about which to be curious.

Living By Your Own Strength is not a theme of the Culture. If you want something, you ask a Mind how to get it; and they will helpfully provide it, rather than saying "No, you figure out how to do it yourself." The people of the Culture have little use for personal formidability, nor for a wish to become stronger. To me, the notion of growing in strength seems obvious, and it also seems obvious that the humaniform citizens of the Culture ought to grow into Minds themselves, over time. But the people of the Culture do not seem to get any smarter as they age; and after four hundred years so, they displace themselves into a sun. These two literary points are probably related.

But the Culture's main problem, I would say, is...

...the same as Narnia's main problem, actually. Bear with me here.

If you read The Lion, the Witch, and the Wardrobe or saw the first Chronicles of Narnia movie, you'll recall -

- I suppose that if you don't want any spoilers, you should stop reading here, but since it's a children's story and based on Christian theology, I don't think I'll be giving away too much by saying -

- that the four human children who are the main characters, fight the White Witch and defeat her with the help of the great talking lion Aslan.

Well, to be precise, Aslan defeats the White Witch.

It's never explained why Aslan ever left Narnia a hundred years ago, allowing the White Witch to impose eternal winter and cruel tyranny on the inhabitants. Kind of an awful thing to do, wouldn't you say?

But once Aslan comes back, he kicks the White Witch out and everything is okay again. There's no obvious reason why Aslan actually needs the help of four snot-nosed human youngsters. Aslan could have led the armies. In fact, Aslan did muster the armies and lead them before the children showed up. Let's face it, the kids are just along for the ride.

The problem with Narnia... is Aslan.

C. S. Lewis never needed to write Aslan into the story. The plot makes far more sense without him. The children could show up in Narnia on their own, and lead the armies on their own.

But is poor Lewis alone to blame? Narnia was written as a Christian parable, and the Christian religion itself has exactly the same problem. All Narnia does is project the flaw in a stark, simplified light: this story has an extra lion.

And the problem with the Culture is the Minds.

"Well..." says the transhumanist SF fan, "Iain Banks did portray the Culture's Minds as 'cynical, amoral, and downright sneaky' in their altruistic way; and they do, in his stories, mess around with humans and use them as pawns. But that is mere fictional evidence. A better-organized society would have laws against big Minds messing with small ones without consent. Though if a Mind is truly wise and kind and utilitarian, it should know how to balance possible resentment against other gains, without needing a law. Anyway, the problem with the Culture is the meddling, not the Minds."

But that's not what I mean. What I mean is that if you could otherwise live in the same Culture - the same technology, the same lifespan and healthspan, the same wealth, freedom, and opportunity -

"I don't want to live in any version of the Culture. I don't want to live four hundred years in a biological body with a constant IQ and then die. Bleah!"

Fine, stipulate that problem solved. My point is that if you could otherwise get the same quality of life, in the same world, but without any Minds around to usurp the role of main character, wouldn't you prefer -

"What?" cry my transhumanist readers, incensed at this betrayal by one of their own. "Are you saying that we should never create any minds smarter than human, or keep them under lock and chain? Just because your soul is so small and mean that you can't bear the thought of anyone else being better than you?"

No, I'm not saying -

"Because that business about our souls shriveling up due to 'loss of meaning' is typical bioconservative neo-Luddite propaganda -"

Invalid argument: the world's greatest fool may say the sun is shining but that doesn't make it dark out. But in any case, that's not what I'm saying -

"It's a lost cause! You'll never prevent intelligent life from achieving its destiny!"

Trust me, I -

"And anyway it's a silly question to begin with, because you can't just remove the Minds and keep the same technology, wealth, and society."

So you admit the Culture's Minds are a necessary evil, then. A price to be paid.

"Wait, I didn't say that -"

And I didn't say all that stuff you're imputing to me!

Ahem.

My model already says we live in a Big World. In which case there are vast armies of minds out there in the immensity of Existence (not just Possibility) which are far more awesome than myself. Any shrivelable souls can already go ahead and shrivel.

And I just talked about people growing up into Minds over time, at some eudaimonic rate of intelligence increase. So clearly I'm not trying to 'prevent intelligent life from achieving its destiny', nor am I trying to enslave all Minds to biological humans scurrying around forever, nor am I etcetera. (I do wish people wouldn't be quite so fast to assume that I've suddenly turned to the Dark Side - though I suppose, in this day and era, it's never an implausible hypothesis.)

But I've already argued that we need a nonperson predicate - some way of knowing that some computations are definitely not people - to avert an AI from creating sentient simulations in its efforts to model people.

And trying to create a Very Powerful Optimization Process that lacks subjective experience and other aspects of personhood, is probably - though I still confess myself somewhat confused on this subject - probably substantially easier than coming up with a nonperson predicate.

This being the case, there are very strong reasons why a superintelligence should initially be designed to be knowably nonsentient, if at all possible. Creating a new kind of sentient mind is a huge and non-undoable act.

Now, this doesn't answer the question of whether a nonsentient Friendly superintelligence ought to make itself sentient, or whether an NFSI ought to immediately manufacture sentient Minds first thing in the morning, once it has adequate wisdom to make the decision.

But there is nothing except our own preferences, out of which to construct the Future. So though this piece of information is not conclusive, nonetheless it is highly informative:

If you already had the lifespan and the health and the promise of future growth, would you want new powerful superintelligences to be created in your vicinity, on your same playing field?

Or would you prefer that we stay on as the main characters in the story of intelligent life, with no higher beings above us?

Should existing human beings grow up at some eudaimonic rate of intelligence increase, and then eventually decide what sort of galaxy to create, and how to people it?

Or is it better for a nonsentient superintelligence to exercise that decision on our behalf, and start creating new powerful Minds right away?

If we don't have to do it one way or the other - if we have both options - and if there's no particular need for heroic self-sacrifice - then which do you like?

"I don't understand the point to what you're suggesting. Eventually, the galaxy is going to have Minds in it, right? We have to find a stable state that allows big Minds and little Minds to coexist. So what's the point in waiting?"

Well... you could have the humans grow up (at some eudaimonic rate of intelligence increase), and then when new people are created, they might be created as powerful Minds to start with. Or when you create new minds, they might have a different emotional makeup, which doesn't lead them to feel overshadowed if there are more powerful Minds above them. But we, as we exist already created - we might prefer to stay on as the main characters, for now, if given a choice.

"You are showing far too much concern for six billion squishy things who happen to be alive today, out of all the unthinkable vastness of space and time."

The Past contains enough tragedy, and has seen enough sacrifice already, I think. And I'm not sure that you can cleave off the Future so neatly from the Present.

So I will set out as I mean the future to continue: with concern for the living.

The sound of six billion faces being casually stepped on, does not seem to me like a good beginning. Even the Future should not be assumed to prefer that another chunk of pain be paid into its price.

So yes, I am concerned for those currently alive, because it is that concern - and not a casual attitude toward the welfare of sentient beings - which I wish to continue into the Future.

And I will not, if at all possible, give any other human being the least cause to think that someone else might spark a better Singularity. I can make no promises upon the future, but I will at least not close off desirable avenues through my own actions. I will not, on my own authority, create a sentient superintelligence which may already determine humanity as having passed on the torch. It is too much to do on my own, and too much harm to do on my own - to amputate someone else's destiny, and steal their main character status. That is yet another reason not to create a sentient superintelligence to start with. (And it's part of the logic behind the CEV proposal, which carefully avoids filling in any moral parameters not yet determined.)

But to return finally to the Culture and to Fun Theory:

The Minds in the Culture don't need the humans, and yet the humans need to be needed.

If you're going to have human-level minds with human emotional makeups, they shouldn't be competing on a level playing field with superintelligences. Either keep the superintelligences off the local playing field, or design the human-level minds with a different emotional makeup.

"The Culture's sole justification for the relatively unworried, hedonistic life its population enjoyed was its good works," writes Iain Banks. This indicates a rather unstable moral position. Either the life the population enjoys is eudaimonic enough to be its own justification, an end rather than a means; or else that life needs to be changed.

When people are in need of rescue, this is is a goal of the overriding-static-predicate sort, where you rescue them as fast as possible, and then you're done. Preventing suffering cannot provide a lasting meaning to life. What happens when you run out of victims? If there's nothing more to life than eliminating suffering, you might as well eliminate life and be done.

If the Culture isn't valuable enough for itself, even without its good works - then the Culture might as well not be. And when the Culture's Minds could do a better job and faster, "good works" can hardly justify the human existences within it.

The human-level people need a destiny to make for themselves, and they need the overshadowing Minds off their playing field while they make it. Having an external evangelism project, and being given cute little roles that any Mind could do better in a flash, so as to "supply meaning", isn't going to cut it.

That's far from the only thing the Culture is doing wrong, but it's at the top of my list.

Sequence: Fun Theory

Nonperson Predicates

Dunbar's Function

Referenced by: Dunbar's Function • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Amputation of Destiny

Dunbar's Function

Eliezer Yudkowsky, 31 December 2008 02:26AM

(This post is part of the Fun Theory Sequence.)

The study of eudaimonic community sizes began with a seemingly silly method of calculation: Robin Dunbar calculated the correlation between the (logs of the) relative volume of the neocortex and observed group size in primates, then extended the graph outward to get the group size for a primate with a human-sized neocortex. You immediately ask, "How much of the variance in primate group size can you explain like that, anyway?" and the answer is 76% of the variance among 36 primate genera, which is respectable. Dunbar came up with a group size of 148. Rounded to 150, and with the confidence interval of 100 to 230 tossed out the window, this became known as "Dunbar's Number".

It's probably fair to say that a literal interpretation of this number is more or less bogus.

There was a bit more to it than that, of course. Dunbar went looking for corroborative evidence from studies of corporations, hunter-gatherer tribes, and utopian communities. Hutterite farming communities, for example, had a rule that they must split at 150 - with the rationale explicitly given that it was impossible to control behavior through peer pressure beyond that point.

But 30-50 would be a typical size for a cohesive hunter-gatherer band; 150 is more the size of a cultural lineage of related bands. Life With Alacrity has an excellent series on Dunbar's Number which exhibits e.g. a histogram of Ultima Online guild sizes - with the peak at 60, not 150. LWA also cites further research by PARC's Yee and Ducheneaut showing that maximum internal cohesiveness, measured in the interconnectedness of group members, occurs at a World of Warcraft guild size of 50. (Stop laughing; you can get much more detailed data on organizational dynamics if it all happens inside a computer server.)

And Dunbar himself did another regression and found that a community of 150 primates would have to spend 43% of its time on social grooming, which Dunbar interpreted as suggesting that 150 was an upper bound rather than an optimum, when groups were highly incentivized to stay together. 150 people does sound like a lot of employees for a tight-knit startup, doesn't it?

Also from Life With Alacrity:

A group of 3 is often unstable, with one person feeling left out, or else one person controlling the others by being the "split" vote. A group of 4 often devolves into two pairs... At 5 to 8 people, you can have a meeting where everyone can speak out about what the entire group is doing, and everyone feels highly empowered. However, at 9 to 12 people this begins to break down -- not enough "attention" is given to everyone and meetings risk becoming either too noisy, too boring, too long, or some combination thereof.

As you grow past 12 or so employees, you must start specializing and having departments and direct reports; however, you are not quite large enough for this to be efficient, and thus much employee time that you put toward management tasks is wasted. Only as you approach and pass 25 people does having simple departments and managers begin to work again...

I've already noted the next chasm when you go beyond 80 people, which I think is the point that Dunbar's Number actually marks for a non-survival oriented group. Even at this lower point, the noise level created by required socialization becomes an issue, and filtering becomes essential. As you approach 150 this begins to be unmanageable...

LWA suggests that community satisfaction has two peaks, one at size ~7 for simple groups, and one at ~60 for complex groups; and that any community has to fraction, one way or another, by the time it approaches Dunbar's Number.

One of the primary principles of evolutionary psychology is that "Our modern skulls house a stone age mind" (saith Tooby and Cosmides). You can interpret all sorts of angst as the friction of a stone age mind rubbing against a modern world that isn't like the hunter-gatherer environment the brain evolved to handle.

We may not directly interact with most of the other six billion people in the world, but we still live in a world much larger than Dunbar's Number.

Or to say it with appropriate generality: taking our current brain size and mind design as the input, we live in a world much larger than Dunbar's Function for minds of our type.

Consider some of the consequences:

If you work in a large company, you probably don't know your tribal chief on any personal level, and may not even be able to get access to him. For every rule within your company, you may not know the person who decided on that rule, and have no realistic way to talk to them about the effects of that rule on you. Large amounts of the organizational structure of your life are beyond your ability to control, or even talk about with the controllers; directives that have major effects on you, may be handed down from a level you can't reach.

If you live in a large country, you probably don't know your President or Prime Minister on a personal level, and may not even be able to get a few hours' chat; you live under laws and regulations that you didn't make, and you can't talk to the people who made them.

This is a non-ancestral condition. Even children, while they may live under the dictatorial rule of their parents, can at least personally meet and talk to their tyrants. You could expect this unnatural (that is, non-EEA) condition to create some amount of anomie.

Though it's a side issue, what's even more... interesting.... is the way that our brains simply haven't updated to their diminished power in a super-Dunbarian world. We just go on debating politics, feverishly applying our valuable brain time to finding better ways to run the world, with just the same fervent intensity that would be appropriate if we were in a small tribe where we could persuade people to change things.

If people don't like being part of large organizations and countries, why do they stick around? Because of another non-ancestral condition - you can't just gather your more sensible friends, leave the band, and gather nuts and berries somewhere else. If I had to cite two non-regulatory barriers at work, it would be (a) the cost of capital equipment, and (b) the surrounding web of contacts and contracts - a web of installed relationships not easily duplicated by a new company.

I suspect that this is a major part of where the stereotype of Technology as the Machine Death-Force comes from - that along with the professional specialization and the expensive tools, you end up in social structures over which you have much less control. Some of the fear of creating a powerful AI "even if Friendly" may come from that stereotypical anomie - that you're creating a stronger Machine Death-Force to regulate your life.

But we already live in a world, right now, where people are less in control of their social destinies than they would be in a hunter-gatherer band, because it's harder to talk to the tribal chief or (if that fails) leave unpleasant restrictions and start your own country. There is an opportunity for progress here.

Another problem with our oversized world is the illusion of increased competition. There's that famous survey which showed that Harvard students would rather make $50,000 if their peers were making $25,000 than make $100,000 if their peers were receiving $200,000 - and worse, they weren't necessarily wrong about what would make them happy. With a fixed income, you're unhappier at the low end of a high-class neighborhood than the high end of a middle-class neighborhood.

But in a "neighborhood" the size of Earth - well, you're actually quite unlikely to run into either Bill Gates or Angelina Jolie on any given day. But the media relentlessly bombards you with stories about the interesting people who are much richer than you or much more attractive, as if they actually constituted a large fraction of the world. (This is a combination of biased availability, and a difficulty in discounting tiny fractions.)

Now you could say that our hedonic relativism is one of the least pleasant aspects of human nature. And I might agree with you about that. But I tend to think that deep changes of brain design and emotional architecture should be taken slowly, and so it makes sense to look at the environment too.

If you lived in a world the size of a hunter-gatherer band, then it would be easier to find something important at which to be the best - or do something that genuinely struck you as important, without becoming lost in a vast crowd of others with similar ideas.

The eudaimonic size of a community as a function of the component minds' intelligence might be given by the degree to which those minds find it natural to specialize - the number of different professions that you can excel at, without having to invent professions just to excel at. Being the best at Go is one thing, if many people know about Go and play it. Being the best at "playing tennis using a football" is easier to achieve, but it also seems a tad... artificial.

Call a specialization "natural" if it will arise without an oversupply of potential entrants. Newton could specialize in "physics", but today it would not be possible to specialize in "physics" - even if you were the only potential physicist in the world, you couldn't achieve expertise in all the physics known to modern-day humanity. You'd have to pick, say, quantum field theory, or some particular approach to QFT. But not QFT over left-handed bibble-braids with cherries on top; that's what happens when there are a thousand other workers in your field and everyone is desperate for some way to differentiate themselves.

When you look at it that way, then there must be much more than 50 natural specializations in the modern world - but still much less than six billion. By the same logic as the original Dunbar's Number, if there are so many different professional specialties that no one person has heard of them all, then you won't know who to consult about any given topic.

But if people keep getting smarter and learning more - expanding the number of relationships they can track, maintaining them more efficiently - and naturally specializing further as more knowledge is discovered and we become able to conceptualize more complex areas of study - and if the population growth rate stays under the rate of increase of Dunbar's Function - then eventually there could be a single community of sentients, and it really would be a single community.

Sequence: Fun Theory

Amputation of Destiny

In Praise of Boredom

Referenced by: The Fun Theory Sequence • 31 Laws of Fun • Helpless Individuals

Original with comments: Dunbar's Function

A New Day

Eliezer Yudkowsky, 31 December 2008 06:40PM

Somewhere in the vastnesses of the Internet and the almost equally impenetrable thicket of my bookmark collection, there is a post by someone who was learning Zen meditation...

Someone who was surprised by how many of the thoughts that crossed his mind, as he tried to meditate, were old thoughts - thoughts he had thunk many times before. He was successful in banishing these old thoughts, but did he succeed in meditating? No; once the comfortable routine thoughts were banished, new and interesting and more distracting thoughts began to cross his mind instead.

I was struck, on reading this, how much of my life I had allowed to fall into routine patterns. Once you actually see that, it takes on a nightmarish quality: You can imagine your fraction of novelty diminishing and diminishing, so slowly you never take alarm, until finally you spend until the end of time watching the same videos over and over again, and thinking the same thoughts each time.

Sometime in the next week - January 1st if you have that available, or maybe January 3rd or 4th if the weekend is more convenient - I suggest you hold a New Day, where you don't do anything old.

Don't read any book you've read before. Don't read any author you've read before. Don't visit any website you've visited before. Don't play any game you've played before. Don't listen to familiar music that you already know you'll like. If you go on a walk, walk along a new path even if you have to drive to a different part of the city for your walk. Don't go to any restaurant you've been to before, order a dish that you haven't had before. Talk to new people (even if you have to find them in an IRC channel) about something you don't spend much time discussing.

And most of all, if you become aware of yourself musing on any thought you've thunk before, then muse on something else. Rehearse no old grievances, replay no old fantasies.

If it works, you could make it a holiday tradition, and do it every New Year.

Original with comments: A New Day

Free to Optimize

Eliezer Yudkowsky, 02 January 2009 01:41AM

(This post is part of the Fun Theory Sequence.)
Followup to: Thou Art Physics, Timeless Control, Possibility and Could-ness, The Ultimate Source

Stare decisis is the legal principle which binds courts to follow precedent, retrace the footsteps of other judges' decisions. As someone previously condemned to an Orthodox Jewish education, where I gritted my teeth at the idea that medieval rabbis would always be wiser than modern rabbis, I completely missed the rationale for stare decisis. I thought it was about respect for the past.

But shouldn't we presume that, in the presence of science, judges closer to the future will know more - have new facts at their fingertips - which enable them to make better decisions? Imagine if engineers respected the decisions of past engineers, not as a source of good suggestions, but as a binding precedent! - That was my original reaction. The standard rationale behind stare decisis came as a shock of revelation to me; it considerably increased my respect for the whole legal system.

This rationale is jurisprudence constante: The legal system must above all be predictable, so that people can execute contracts or choose behaviors knowing the legal implications.

Judges are not necessarily there to optimize, like an engineer. The purpose of law is not to make the world perfect. The law is there to provide a predictable environment in which people can optimize their own futures.

I was amazed at how a principle that at first glance seemed so completely Luddite, could have such an Enlightenment rationale. It was a "shock of creativity" - a solution that ranked high in my preference ordering and low in my search ordering, a solution that violated my previous surface generalizations. "Respect the past just because it's the past" would not have easily occurred to me as a good solution for anything.

There's a peer commentary in Evolutionary Origins of Morality which notes in passing that "other things being equal, organisms will choose to reward themselves over being rewarded by caretaking organisms". It's cited as the Premack principle, but the actual Premack principle looks to be something quite different, so I don't know if this is a bogus result, a misremembered citation, or a nonobvious derivation. If true, it's definitely interesting from a fun-theoretic perspective.

Optimization is the ability to squeeze the future into regions high in your preference ordering. Living by my own strength, means squeezing my own future - not perfectly, but still being able to grasp some of the relation between my actions and their consequences. This is the strength of a human.

If I'm being helped, then some other agent is also squeezing my future - optimizing me - in the same rough direction that I try to squeeze myself. This is "help".

A human helper is unlikely to steer every part of my future that I could have steered myself. They're not likely to have already exploited every connection between action and outcome that I can myself understand. They won't be able to squeeze the future that tightly; there will be slack left over, that I can squeeze for myself.

We have little experience with being "caretaken" across any substantial gap in intelligence; the closest thing that human experience provides us with is the idiom of parents and children. Human parents are still human; they may be smarter than their children, but they can't predict the future or manipulate the kids in any fine-grained way.

Even so, it's an empirical observation that some human parents do help their children so much that their children don't become strong. It's not that there's nothing left for their children to do, but with a hundred million dollars in a trust fund, they don't need to do much - their remaining motivations aren't strong enough. Something like that depends on genes, not just environment - not every overhelped child shrivels - but conversely it depends on environment too, not just genes.

So, in considering the kind of "help" that can flow from relatively stronger agents to relatively weaker agents, we have two potential problems to track:

1. Help so strong that it optimizes away the links between the desirable outcome and your own choices.

2. Help that is believed to be so reliable, that it takes off the psychological pressure to use your own strength.

Since (2) revolves around belief, could you just lie about how reliable the help was? Pretend that you're not going to help when things get bad - but then if things do get bad, you help anyway? That trick didn't work too well for Alan Greenspan and Ben Bernanke.

A superintelligence might be able to pull off a better deception. But in terms of moral theory and eudaimonia - we are allowed to have preferences over external states of affairs, not just psychological states. This applies to "I want to really steer my own life, not just believe that I do", just as it applies to "I want to have a love affair with a fellow sentient, not just a puppet that I am deceived into thinking sentient". So if we can state firmly from a value standpoint that we don't want to be fooled this way, then building an agent which respects that preference is a mere matter of Friendly AI.

Modify people so that they don't relax when they believe they'll be helped? I usually try to think of how to modify environments before I imagine modifying any people. It's not that I want to stay the same person forever; but the issues are rather more fraught, and one might wish to take it slowly, at some eudaimonic rate of personal improvement.

(1), though, is the most interesting issue from a philosophicalish standpoint. It impinges on the confusion named "free will". Of which I have already untangled; see the posts referenced at top, if you're recently joining OB.

Let's say that I'm an ultrapowerful AI, and I use my knowledge of your mind and your environment to forecast that, if left to your own devices, you will make $999,750. But this does not satisfice me; it so happens that I want you to make at least $1,000,000. So I hand you $250, and then you go on to make $999,750 as you ordinarily would have.

How much of your own strength have you just lived by?

The first view would say, "I made 99.975% of the money; the AI only helped 0.025% worth."

The second view would say, "Suppose I had entirely slacked off and done nothing. Then the AI would have handed me $1,000,000. So my attempt to steer my own future was an illusion; my future was already determined to contain $1,000,000."

Someone might reply, "Physics is deterministic, so your future is already determined no matter what you or the AI does -"

But the second view interrupts and says, "No, you're not confusing me that easily. I am within physics, so in order for my future to be determined by me, it must be determined by physics. The Past does not reach around the Present and determine the Future before the Present gets a chance - that is mixing up a timeful view with a timeless one. But if there's an AI that really does look over the alternatives before I do, and really does choose the outcome before I get a chance, then I'm really not steering my own future. The future is no longer counterfactually dependent on my decisions."

At which point the first view butts in and says, "But of course the future is counterfactually dependent on your actions. The AI gives you $250 and then leaves. As a physical fact, if you didn't work hard, you would end up with only $250 instead of $1,000,000."

To which the second view replies, "I one-box on Newcomb's Problem, so my counterfactual reads 'if my decision were to not work hard, the AI would have given me $1,000,000 instead of $250'."

"So you're saying," says the first view, heavy with sarcasm, "that if the AI had wanted me to make at least $1,000,000 and it had ensured this through the general policy of handing me $1,000,000 flat on a silver platter, leaving me to earn $999,750 through my own actions, for a total of $1,999,750 - that this AI would have interfered less with my life than the one who just gave me $250."

The second view thinks for a second and says "Yeah, actually. Because then there's a stronger counterfactual dependency of the final outcome on your own decisions. Every dollar you earned was a real added dollar. The second AI helped you more, but it constrained your destiny less."

"But if the AI had done exactly the same thing, because it wanted me to make exactly $1,999,750 -"

The second view nods.

"That sounds a bit scary," the first view says, "for reasons which have nothing to do with the usual furious debates over Newcomb's Problem. You're making your utility function path-dependent on the detailed cognition of the Friendly AI trying to help you! You'd be okay with it if the AI only could give you $250. You'd be okay if the AI had decided to give you $250 through a decision process that had predicted the final outcome in less detail, even though you acknowledge that in principle your decisions may already be highly deterministic. How is a poor Friendly AI supposed to help you, when your utility function is dependent, not just on the outcome, not just on the Friendly AI's actions, but dependent on differences of the exact algorithm the Friendly AI uses to arrive at the same decision? Isn't your whole rationale of one-boxing on Newcomb's Problem that you only care about what works?"

"Well, that's a good point," says the second view. "But sometimes we only care about what works, and yet sometimes we do care about the journey as well as the destination. If I was trying to cure cancer, I wouldn't care how I cured cancer, or whether I or the AI cured cancer, just so long as it ended up cured. This isn't that kind of problem. This is the problem of the eudaimonic journey - it's the reason I care in the first place whether I get a million dollars through my own efforts or by having an outside AI hand it to me on a silver platter. My utility function is not up for grabs. If I desire not to be optimized too hard by an outside agent, the agent needs to respect that preference even if it depends on the details of how the outside agent arrives at its decisions. Though it's also worth noting that decisions are produced by algorithms - if the AI hadn't been using the algorithm of doing just what it took to bring me up to $1,000,000, it probably wouldn't have handed me exactly $250."

The desire not to be optimized too hard by an outside agent is one of the structurally nontrivial aspects of human morality.

But I can think of a solution, which unless it contains some terrible flaw not obvious to me, sets a lower bound on the goodness of a solution: any alternative solution adopted, ought to be at least this good or better.

If there is anything in the world that resembles a god, people will try to pray to it. It's human nature to such an extent that people will pray even if there aren't any gods - so you can imagine what would happen if there were! But people don't pray to gravity to ignore their airplanes, because it is understood how gravity works, and it is understood that gravity doesn't adapt itself to the needs of individuals. Instead they understand gravity and try to turn it to their own purposes.

So one possible way of helping - which may or may not be the best way of helping - would be the gift of a world that works on improved rules, where the rules are stable and understandable enough that people can manipulate them and optimize their own futures together. A nicer place to live, but free of meddling gods beyond that. I have yet to think of a form of help that is less poisonous to human beings - but I am only human.

Added: Note that modern legal systems score a low Fail on this dimension - no single human mind can even know all the regulations any more, let alone optimize for them. Maybe a professional lawyer who did nothing else could memorize all the regulations applicable to them personally, but I doubt it. As Albert Einstein observed, any fool can make things more complicated; what takes intelligence is moving in the opposite direction.

Sequence: Fun Theory

Living By Your Own Strength

Harmful Options

Referenced by: Growing Up is Hard • Interpersonal Entanglement • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Free to Optimize

The Uses of Fun (Theory)

Eliezer Yudkowsky, 02 January 2009 08:30PM

(This post is part of the Fun Theory Sequence.)

"But is there anyone who actually wants to live in a Wellsian Utopia? On the contrary, not to live in a world like that, not to wake up in a hygenic garden suburb infested by naked schoolmarms, has actually become a conscious political motive. A book like Brave New World is an expression of the actual fear that modern man feels of the rationalised hedonistic society which it is within his power to create."
-- George Orwell, Why Socialists Don't Believe in Fun

There are three reasons I'm talking about Fun Theory, some more important than others:

1. If every picture ever drawn of the Future looks like a terrible place to actually live, it might tend to drain off the motivation to create the future. It takes hope to sign up for cryonics.

2. People who leave their religions, but don't familiarize themselves with the deep, foundational, fully general arguments against theism, are at risk of backsliding. Fun Theory lets you look at our present world, and see that it is not optimized even for considerations like personal responsibility or self-reliance. It is the fully general reply to theodicy.

3. Going into the details of Fun Theory helps you see that eudaimonia is actually complicated - that there are a lot of properties necessary for a mind to lead a worthwhile existence. Which helps you appreciate just how worthless a galaxy would end up looking (with extremely high probability) if it was optimized by something with a utility function rolled up at random.

To amplify on these points in order:

(1) You've got folks like Leon Kass and the other members of Bush's "President's Council on Bioethics" running around talking about what a terrible, terrible thing it would be if people lived longer than threescore and ten. While some philosophers have pointed out the flaws in their arguments, it's one thing to point out a flaw and another to provide a counterexample. "Millions long for immortality who do not know what to do with themselves on a rainy Sunday afternoon," said Susan Ertz, and that argument will sound plausible for as long as you can't imagine what to do on a rainy Sunday afternoon, and it seems unlikely that anyone could imagine it.

It's not exactly the fault of Hans Moravec that his world in which humans are kept by superintelligences as pets, doesn't sound quite Utopian. Utopias are just really hard to construct, for reasons I'll talk about in more detail later - but this observation has already been made by many, including George Orwell.

Building the Future is part of the ethos of secular humanism, our common project. If you have nothing to look forward to - if there's no image of the Future that can inspire real enthusiasm - then you won't be able to scrape up enthusiasm for that common project. And if the project is, in fact, a worthwhile one, the expected utility of the future will suffer accordingly from that nonparticipation. So that's one side of the coin, just as the other side is living so exclusively in a fantasy of the Future that you can't bring yourself to go on in the Present.

I recommend thinking vaguely of the Future's hopes, thinking specifically of the Past's horrors, and spending most of your time in the Present. This strategy has certain epistemic virtues beyond its use in cheering yourself up.

But it helps to have legitimate reason to vaguely hope - to minimize the leaps of abstract optimism involved in thinking that, yes, you can live and obtain happiness in the Future.

(2) Rationality is our goal, and atheism is just a side effect - the judgment that happens to be produced. But atheism is an important side effect. John C. Wright, who wrote the heavily transhumanist The Golden Age, had some kind of temporal lobe epileptic fit and became a Christian. There's a once-helpful soul, now lost to us.

But it is possible to do better, even if your brain malfunctions on you. I know a transhumanist who has strong religious visions, which she once attributed to future minds reaching back in time and talking to her... but then she reasoned it out, asking why future superminds would grant only her the solace of conversation, and why they could offer vaguely reassuring arguments but not tell her winning lottery numbers or the 900th digit of pi. So now she still has strong religious experiences, but she is not religious. That's the difference between weak rationality and strong rationality, and it has to do with the depth and generality of the epistemic rules that you know and apply.

Fun Theory is part of the fully general reply to religion; in particular, it is the fully general reply to theodicy. If you can't say how God could have better created the world without sliding into an antiseptic Wellsian Utopia, you can't carry Epicurus's argument. If, on the other hand, you have some idea of how you could build a world that was not only more pleasant but also a better medium for self-reliance, then you can see that permanently losing both your legs in a car accident when someone else crashes into you, doesn't seem very eudaimonic.

If we can imagine what the world might look like if it had been designed by anything remotely like a benevolently inclined superagent, we can look at the world around us, and see that this isn't it. This doesn't require that we correctly forecast the full optimization of a superagent - just that we can envision strict improvements on the present world, even if they prove not to be maximal.

(3) There's a severe problem in which people, due to anthropomorphic optimism and the lack of specific reflective knowledge about their invisible background framework and many other biases which I have discussed, think of a "nonhuman future" and just subtract off a few aspects of humanity that are salient, like enjoying the taste of peanut butter or something. While still envisioning a future filled with minds that have aesthetic sensibilities, experience happiness on fulfilling a task, get bored with doing the same thing repeatedly, etcetera. These things seem universal, rather than specifically human - to a human, that is. They don't involve having ten fingers or two eyes, so they must be universal, right?

And if you're still in this frame of mind - where "real values" are the ones that persuade every possible mind, and the rest is just some extra specifically human stuff - then Friendly AI will seem unnecessary to you, because, in its absence, you expect the universe to be valuable but not human.

It turns out, though, that once you start talking about what specifically is and isn't valuable, even if you try to keep yourself sounding as "non-human" as possible - then you still end up with a big complicated computation that is only instantiated physically in human brains and nowhere else in the universe. Complex challenges? Novelty? Individualism? Self-awareness? Experienced happiness? A paperclip maximizer cares not about these things.

It is a long project to crack people's brains loose of thinking that things will turn out regardless - that they can subtract off a few specifically human-seeming things, and then end up with plenty of other things they care about that are universal and will appeal to arbitrarily constructed AIs. And of this I have said a very great deal already. But it does not seem to be enough. So Fun Theory is one more step - taking the curtains off some of the invisible background of our values, and revealing some of the complex criteria that go into a life worth living.

Sequence: Fun Theory

Seduced by Imagination

Higher Purpose

Referenced by: Serious Stories • The Fun Theory Sequence • Raising the Sanity Waterline

Original with comments: The Uses of Fun (Theory)

Growing Up is Hard

Eliezer Yudkowsky, 04 January 2009 03:55AM

(This post is part of the Fun Theory Sequence.)

Terrence Deacon's The Symbolic Species is the best book I've ever read on the evolution of intelligence. Deacon somewhat overreaches when he tries to theorize about what our X-factor is; but his exposition of its evolution is first-class.

Deacon makes an excellent case - he has quite persuaded me - that the increased relative size of our frontal cortex, compared to other hominids, is of overwhelming importance in understanding the evolutionary development of humanity. It's not just a question of increased computing capacity, like adding extra processors onto a cluster; it's a question of what kind of signals dominate, in the brain.

People with Williams Syndrome (caused by deletion of a certain region on chromosome 7) are hypersocial, ultra-gregarious; as children they fail to show a normal fear of adult strangers. WSers are cognitively impaired on most dimensions, but their verbal abilities are spared or even exaggerated; they often speak early, with complex sentences and large vocabulary, and excellent verbal recall, even if they can never learn to do basic arithmetic.

Deacon makes a case for some Williams Syndrome symptoms coming from a frontal cortex that is relatively too large for a human, with the result that prefrontal signals - including certain social emotions - dominate more than they should.

"Both postmortem analysis and MRI analysis have revealed brains with a reduction of the entire posterior cerebral cortex, but a sparing of the cerebellum and frontal lobes, and perhaps even an exaggeration of cerebellar size," says Deacon.

Williams Syndrome's deficits can be explained by the shrunken posterior cortex - they can't solve simple problems involving shapes, because the parietal cortex, which handles shape-processing, is diminished. But the frontal cortex is not actually enlarged; it is simply spared. So where do WSers' augmented verbal abilities come from?

Perhaps because the signals sent out by the frontal cortex, saying "pay attention to this verbal stuff!", win out over signals coming from the shrunken sections of the brain. So the verbal abilities get lots of exercise - and other abilities don't.

Similarly with the hyper-gregarious nature of WSers; the signal saying "Pay attention to this person!", originating in the frontal areas where social processing gets done, dominates the emotional landscape.

And Williams Syndrome is not frontal enlargement, remember; it's just frontal sparing in an otherwise shrunken brain, which increases the relative force of frontal signals...

...beyond the narrow parameters within which a human brain is adapted to work.

I mention this because you might look at the history of human evolution, and think to yourself, "Hm... to get from a chimpanzee to a human... you enlarge the frontal cortex... so if we enlarge it even further..."

The road to +Human is not that simple.

Hominid brains have been tested billions of times over through thousands of generations. But you shouldn't reason qualitatively, "Testing creates 'robustness', so now the human brain must be 'extremely robust'." Sure, we can expect the human brain to be robust against some insults, like the loss of a single neuron. But testing in an evolutionary paradigm only creates robustness over the domain tested. Yes, sometimes you get robustness beyond that, because sometimes evolution finds simple solutions that prove to generalize -

But people do go crazy. Not colloquial crazy, actual crazy. Some ordinary young man in college suddenly decides that everyone around them is staring at them because they're part of the conspiracy. (I saw that happen once, and made a classic non-Bayesian mistake; I knew that this was archetypal schizophrenic behavior, but I didn't realize that similar symptoms can arise from many other causes. Psychosis, it turns out, is a general failure mode, "the fever of CNS illnesses"; it can also be caused by drugs, brain tumors, or just sleep deprivation. I saw the perfect fit to what I'd read of schizophrenia, and didn't ask "What if other things fit just as perfectly?" So my snap diagnosis of schizophrenia turned out to be wrong; but as I wasn't foolish enough to try to handle the case myself, things turned out all right in the end.)

Wikipedia says that the current main hypotheses being considered for psychosis are (a) too much dopamine in one place (b) not enough glutamate somewhere else. (I thought I remembered hearing about serotonin imbalances, but maybe that was something else.)

That's how robust the human brain is: a gentle little neurotransmitter imbalance - so subtle they're still having trouble tracking it down after who knows how many fMRI studies - can give you a full-blown case of stark raving mad.

I don't know how often psychosis happens to hunter-gatherers, so maybe it has something to do with a modern diet? We're not getting exactly the right ratio of Omega 6 to Omega 3 fats, or we're eating too much processed sugar, or something. And among the many other things that go haywire with the metabolism as a result, the brain moves into a more fragile state that breaks down more easily...

Or whatever. That's just a random hypothesis. By which I mean to say: The brain really is adapted to a very narrow range of operating parameters. It doesn't tolerate a little too much dopamine, just as your metabolism isn't very robust against non-ancestral ratios of Omega 6 to Omega 3. Yes, sometimes you get bonus robustness in a new domain, when evolution solves W, X, and Y using a compact adaptation that also extends to novel Z. Other times... quite often, really... Z just isn't covered.

Often, you step outside the box of the ancestral parameter ranges, and things just plain break.

Every part of your brain assumes that all the other surrounding parts work a certain way. The present brain is the Environment of Evolutionary Adaptedness for every individual piece of the present brain.

Start modifying the pieces in ways that seem like "good ideas" - making the frontal cortex larger, for example - and you start operating outside the ancestral box of parameter ranges. And then everything goes to hell. Why shouldn't it? Why would the brain be designed for easy upgradability?

Even if one change works - will the second? Will the third? Will all four changes work well together? Will the fifth change have all that greater a probability of breaking something, because you're already operating that much further outside the ancestral box? Will the sixth change prove that you exhausted all the brain's robustness in tolerating the changes you made already, and now there's no adaptivity left?

Poetry aside, a human being isn't the seed of a god. We don't have neat little dials that you can easily tweak to more "advanced" settings. We are not designed for our parts to be upgraded. Our parts are adapted to work exactly as they are, in their current context, every part tested in a regime of the other parts being the way they are. Idiot evolution does not look ahead, it does not design with the intent of different future uses. We are not designed to unfold into something bigger.

Which is not to say that it could never, ever be done.

You could build a modular, cleanly designed AI that could make a billion sequential upgrades to itself using deterministic guarantees of correctness. A Friendly AI programmer could do even more arcane things to make sure the AI knew what you would-want if you understood the possibilities. And then the AI could apply superior intelligence to untangle the pattern of all those neurons (without simulating you in such fine detail as to create a new person), and to foresee the consequences of its acts, and to understand the meaning of those consequences under your values. And the AI could upgrade one thing while simultaneously tweaking the five things that depend on it and the twenty things that depend on them. Finding a gradual, incremental path to greater intelligence (so as not to effectively erase you and replace you with someone else) that didn't drive you psychotic or give you Williams Syndrome or a hundred other syndromes.

Or you could walk the path of unassisted human enhancement, trying to make changes to yourself without understanding them fully. Sometimes changing yourself the wrong way, and being murdered or suspended to disk, and replaced by an earlier backup. Racing against the clock, trying to raise your intelligence without breaking your brain or mutating your will. Hoping you became sufficiently super-smart that you could improve the skill with which you modified yourself. Before your hacked brain moved so far outside ancestral parameters and tolerated so many insults that its fragility reached a limit, and you fell to pieces with every new attempted modification beyond that. Death is far from the worst risk here. Not every form of madness will appear immediately when you branch yourself for testing - some insanities might incubate for a while before they became visible. And you might not notice if your goals shifted only a bit at a time, as your emotional balance altered with the strange new harmonies of your brain.

Each path has its little upsides and downsides. (E.g: AI requires supreme precise knowledge; human upgrading has a nonzero probability of success through trial and error. Malfunctioning AIs mostly kill you and tile the galaxy with smiley faces; human upgrading might produce insane gods to rule over you in Hell forever. Or so my current understanding would predict, anyway; it's not like I've observed any of this as a fact.)

And I'm sorry to dismiss such a gigantic dilemma with three paragraphs, but it wanders from the point of today's post:

The point of today's post is that growing up - or even deciding what you want to be when you grow up - is as around as hard as designing a new intelligent species. Harder, since you're constrained to start from the base of an existing design. There is no natural path laid out to godhood, no Level attribute that you can neatly increment and watch everything else fall into place. It is an adult problem.

Being a transhumanist means wanting certain things - judging them to be good. It doesn't mean you think those goals are easy to achieve.

Just as there's a wide range of understanding among people who talk about, say, quantum mechanics, there's also a certain range of competence among transhumanists. There are transhumanists who fall into the trap of the affect heuristic, who see the potential benefit of a technology, and therefore feel really good about that technology, so that it also seems that the technology (a) has readily managed downsides (b) is easy to implement well and (c) will arrive relatively soon.

But only the most formidable adherents of an idea are any sign of its strength. Ten thousand New Agers babbling nonsense, do not cast the least shadow on real quantum mechanics. And among the more formidable transhumanists, it is not at all rare to find someone who wants something and thinks it will not be easy to get.

One is much more likely to find, say, Nick Bostrom - that is, Dr. Nick Bostrom, Director of the Oxford Future of Humanity Institute and founding Chair of the World Transhumanist Assocation - arguing that a possible test for whether a cognitive enhancement is likely to have downsides, is the ease with which it could have occurred as a natural mutation - since if it had only upsides and could easily occur as a natural mutation, why hasn't the brain already adapted accordingly? This is one reason to be wary of, say, cholinergic memory enhancers: if they have no downsides, why doesn't the brain produce more acetylcholine already? Maybe you're using up a limited memory capacity, or forgetting something else...

And that may or may not turn out to be a good heuristic. But the point is that the serious, smart, technically minded transhumanists, do not always expect that the road to everything they want is easy. (Where you want to be wary of people who say, "But I dutifully acknowledge that there are obstacles!" but stay in basically the same mindset of never truly doubting the victory.)

So you'll forgive me if I am somewhat annoyed with people who run around saying, "I'd like to be a hundred times as smart!" as if it were as simple as scaling up a hundred times instead of requiring a whole new cognitive architecture; and as if a change of that magnitude in one shot wouldn't amount to erasure and replacement. Or asking, "Hey, why not just augment humans instead of building AI?" as if it wouldn't be a desperate race against madness.

I'm not against being smarter. I'm not against augmenting humans. I am still a transhumanist; I still judge that these are good goals.

But it's really not that simple, okay?

Sequence: Fun Theory

Failed Utopia #4-2

Changing Emotions

Referenced by: Changing Emotions • In Praise of Boredom • The Fun Theory Sequence • The Thing That I Protect

Original with comments: Growing Up is Hard

Changing Emotions

Eliezer Yudkowsky, 05 January 2009 12:05AM

(This post is part of the Fun Theory Sequence.)
Previously in series: Growing Up is Hard

Lest anyone reading this journal of a primitive man should think we spend our time mired in abstractions, let me also say that I am discovering the richness available to those who are willing to alter their major characteristics. The variety of emotions available to a reconfigured human mind, thinking thoughts impossible to its ancestors...
The emotion of -*-, describable only as something between sexual love and the joy of intellection - making love to a thought? Or &&, the true reverse of pain, not "pleasure" but a "warning" of healing, growth and change. Or (^+^), the most complex emotion yet discovered, felt by those who consciously endure the change between mind configurations, and experience the broad spectrum of possibilities inherent in thinking and being.
-- Greg Bear, Eon

So... I'm basically on board with that sort of thing as a fine and desirable future. But I think that the difficulty and danger of fiddling with emotions is oft-underestimated. Not necessarily underestimated by Greg Bear, per se; the above journal entry is from a character who was receiving superintelligent help.

But I still remember one time on the Extropians mailing list when someone talked about creating a female yet "otherwise identical" copy of himself. Something about that just fell on my camel's back as the last straw. I'm sorry, but there are some things that are much more complicated to actually do than to rattle off as short English phrases, and "changing sex" has to rank very high on that list. Even if you're omnipotent so far as raw ability goes, it's not like people have a binary attribute reading "M" or "F" that can be flipped as a primitive action.

Changing sex makes a good, vivid example of the sort of difficulties you might run into when messing with emotional architecture, so I'll use it as my archetype:

Let's suppose that we're talking about an M2F transformation. (F2M should be a straightforward transform of this discussion; I do want to be specific rather than talking in vague generalities, but I don't want to parallelize every sentence.) (Oddly enough, every time I can recall hearing someone say "I want to know what it's like to be the opposite sex", the speaker has been male. I don't know if that's a genuine gender difference in wishes, or just a selection effect in which spoken wishes reach my ears.)

Want to spend a week wearing a female body? Even at this very shallow level, we're dealing with drastic remappings of at least some segments of the sensorimotor cortex and cerebellum - the somatic map, the motor map, the motor reflexes, and the motor skills. As a male, you know how to operate a male body, but not a female one. If you're a master martial artist as a male, you won't be a master martial artist as a female (or vice versa, of course) unless you either spend another year practicing, or some AI subtly tweaks your skills to be what they would have been in a female body - think of how odd that experience would be.

Already we're talking about some pretty significant neurological changes. Strong enough to disrupt personal identity, if taken in one shot? That's a difficult question to answer, especially since I don't know what experiment to perform to test any hypotheses. On one hand, billions of neurons in my visual cortex undergo massive changes of activation every time my eyes squeeze shut when I sneeze - the raw number of flipped bits is not the key thing in personal identity. But we are already talking about serious changes of information, on the order of going to sleep, dreaming, forgetting your dreams, and waking up the next morning as though it were the next moment. Not informationally trivial transforms like uploading.

What about sex? (Somehow it's always about sex, at least when it's men asking the question.) Remapping the connections from the remapped somatic areas to the pleasure center will... give you a vagina-shaped penis, more or less. That doesn't make you a woman. You'd still be attracted to girls, and no, that would not make you a lesbian; it would make you a normal, masculine man wearing a female body like a suit of clothing.

What would it take for a man to actually become the female version of themselves?

Well... what does that sentence even mean? I am reminded of someone who replied to the statement "Obama would not have become President if he hadn't been black" by saying "If Obama hadn't been black, he wouldn't have been Obama" i.e. "There is no non-black Obama who could fail to become President". (You know you're in trouble when non-actual possible worlds start having political implications.)

The person you would have been if you'd been born with an X chromosome in place of your Y chromosome (or vice versa) isn't you. If you had a twin female sister, the two of you would not be the same person. There are genes on your Y chromosome that tweaked your brain to some extent, helping to construct your personal identity - alleles with no analogue on the X chromosome. There is no version of you, even genetically, who is the opposite sex.

And if we halt your body, swap out your Y chromosome for your father's X chromosome, and restart your body... well. That doesn't sound too safe, does it? Your neurons are already wired in a male pattern, just as your body already developed in a male pattern. I don't know what happens to your testicles, and I don't know what happens to your brain, either. Maybe your circuits would slowly start to rewire themselves under the influence of the new genetic instructions. At best you'd end up as a half-baked cross between male brain and female brain. At worst you'd go into a permanent epileptic fit and die - we're dealing with circumstances way outside the evolutionary context under which the brain was optimized for robustness. Either way, your brain would not look like your twin sister's brain that had developed as female from the beginning.

So to actually become female...

We're talking about a massive transformation here, billions of neurons and trillions of synapses rearranged. Not just form, but content - just like a male judo expert would need skills repatterned to become a female judo expert, so too, you know how to operate a male brain but not a female brain. You are the equivalent of a judo expert at one, but not the other. You have cognitive reflexes, and consciously learned cognitive skills as well.

If I fell asleep and woke up as a true woman - not in body, but in brain - I don't think I'd call her "me". The change is too sharp, if it happens all at once.

Transform the brain gradually? Hm... now we have to design the intermediate stages, and make sure the intermediate stages make self-consistent sense. Evolution built and optimized a self-consistent male brain and a self-consistent female brain; it didn't design the parts to be stable during an intermediate transition between the two. Maybe you've got to redesign other parts of the brain just to keep working through the transition.

What happens when, as a woman, you think back to your memory of looking at Angelina Jolie photos as a man? How do you empathize with your past self of the opposite sex? Do you flee in horror from the person you were? Are all your life's memories distant and alien things? How can you remember, when your memory is a recorded activation pattern for neural circuits that no longer exist in their old forms? Do we rewrite all your memories, too?

Well... maybe we could retain your old male brainware through the transformation, and set up a dual system of male and female circuits... such that you are currently female, but retain the ability to recall and empathize with your past memories as if they were running on the same male brainware that originally laid them down...

Sounds complicated, doesn't it? It seems that to transform a male brain into someone who can be a real female, we can't just rewrite you as a female brain. That just kills you and replaces you with someone re-imagined as a different person. Instead we have to rewrite you as a more complex brain with a novel, non-ancestral architecture that can cross-operate in realtime between male and female modes, so that a female can process male memories with a remembered context that includes the male brainware that laid them down.

To make you female, and yet still you, we have to step outside the human design space in order to preserve continuity with your male self.

And when your little adventure is over and you go back to being a man - if you still want to, because even if your past self wanted to go back afterward, why should that desire be binding on your present self? - then we've got to keep the dual architecture so you don't throw up every time you remember what you did on your vacation.

Assuming you did have sex as a woman, rather than fending off all comers because because they didn't look like they were interested in a long-term relationship.

But then, you probably would experiment. You'll never have been a little girl, and you won't remember going through high school where any girl who slept with a boy was called a slut by the other girls. You'll remember a very atypical past for a woman - but there's no way to fix that while keeping you the same person.

And all that was just what it takes to ranma around within human-space, from the male pole to the female pole and back again.

What if you wanted to move outside the human space entirely?

In one sense, a sex change is admittedly close to a worst-case scenario: a fixed target not optimized for an easy transition from your present location; involving, not just new brain areas, but massive coordinated changes to brain areas already in place.

It might be a lot easier to just add one more emotion to those already there. Maybe.

In another sense, though, a sex change is close to a best-case scenario: the prototype of your destination is already extensively tested as a coherent mind, and known to function well within a human society that already has a place for it (including companions to talk to).

It might be a lot harder to enter uncharted territory. Maybe.

I'm not saying - of course - that it could never, ever be done. But it's another instance of the great chicken-and-egg dilemma that is the whole story of present-day humanity, the great challenge that intelligent life faces in its flowering: growing up is a grownup-level problem. You could try to build a cleanly-designed artificial grownup (self-improving Friendly AI) to foresee the pathway ahead and chart out a nonfatal course. Or you could plunge ahead yourself, and hope that you grew faster than your problems did.

It's the same core challenge either way: growing up is an adult problem. There are difficult ways out of this trap, but no easy ones; extra-ordinary solutions, but no ordinary ones. People ask me why I take all these difficulties upon myself. It's because all the easier ways, once you examine them in enough fine detail, turn out to be illusions, or contain just as much difficulty themselves - the same sort of hidden difficulty as "I'd like to try being the opposite sex for a week".

It seems to me that there is just an irreducible residue of very hard problems associated with an adult version of humankind ever coming into being.

And emotions would be among the most dangerous targets of meddling. Make the wrong shift, and you won't want to change back.

We can't keep these exact human emotions forever. Anyone want to still want to eat chocolate-chip cookies when the last sun grows cold? I didn't think so.

But if we replace our emotions with random die-rolls, then we'll end up wanting to do what is prime, instead of what's right.

Some emotional changes can be desirable, but random replacement seems likely to be undesirable on average. So there must be criteria that distinguish good emotional changes from bad emotional changes. What are they?

Sequence: Fun Theory

Growing Up is Hard

Emotional Involvement

Referenced by: Emotional Involvement • The Fun Theory Sequence • 31 Laws of Fun • The Thing That I Protect

Original with comments: Changing Emotions

Rationality Quotes 21

Eliezer Yudkowsky, 06 January 2009 02:45AM

"The most dangerous thing in the world is finding someone you agree with. If a TV station news is saying exactly what you think is right, BEWARE! You are very likely only reinforcing your beliefs, and not being supplied with new information."
-- SmallFurryCreature

"Companies deciding which kind of toothpaste to market have much more rigorous, established decision-making processes to refer to than the most senior officials of the U.S. government deciding whether or not to go to war."
-- Michael Mazarr

"Everything I ever thought about myself - who I was, what I am - was a lie. You have no idea how astonishingly liberating that feels."
-- Neil Gaiman, Stardust

"Saul speaks of the 'intense desire for survival on the part of virtually everyone on earth,' and our 'failure' in spite of this. I have often pointed out that the so-called 'survival instinct' is reliable only in clear and present danger - and even then only if the individual is still relatively healthy and vigorous. If the danger is indirect, or remote in time, or if the person is weak or depressed - or even if required action would violate established habits - forget the 'survival instinct.' It isn't that simple."
-- Robert Ettinger on cryonics

"We are running out of excuses. We just have to admit that real AI is one of the lowest research priorities, ever. Even space is considered more important. Nevermind baseball, tribology, or TV evangelism."
-- Eugen Leitl

"That's something that's always struck me as odd about humanity. Our first response to someone's bad news is "I'm sorry", as though we feel that someone should take responsibility for all the $&#ed up randomness that goes on in this universe."
-- Angels 2200

"We were supremely lucky to be born into this precise moment in history. It is with an ever-cresting crescendo of wonder and enthusiasm that I commend you to the great adventure and the great adventure to you."
-- Jeff Davis

Original with comments: Rationality Quotes 21

Emotional Involvement

Eliezer Yudkowsky, 06 January 2009 10:23PM

(This post is part of the Fun Theory Sequence.)
Followup to: Evolutionary Psychology, Thou Art Godshatter, Existential Angst Factory

Can your emotions get involved in a video game? Yes, but not much. Whatever sympathetic echo of triumph you experience on destroying the Evil Empire in a video game, it's probably not remotely close to the feeling of triumph you'd get from saving the world in real life. I've played video games powerful enough to bring tears to my eyes, but they still aren't as powerful as the feeling of significantly helping just one single real human being.

Because when the video game is finished, and you put it away, the events within the game have no long-term consequences.

Maybe if you had a major epiphany while playing... But even then, only your thoughts would matter; the mere fact that you saved the world, inside the game, wouldn't count toward anything in the continuing story of your life.

Thus fails the Utopia of playing lots of really cool video games forever. Even if the games are difficult, novel, and sensual, this is still the idiom of life chopped up into a series of disconnected episodes with no lasting consequences. A life in which equality of consequences is forcefully ensured, or in which little is at stake because all desires are instantly fulfilled without individual work - these likewise will appear as flawed Utopias of dispassion and angst. "Rich people with nothing to do" syndrome. A life of disconnected episodes and unimportant consequences is a life of weak passions, of emotional uninvolvement.

Our emotions, for all the obvious evolutionary reasons, tend to associate to events that had major reproductive consequences in the ancestral environment, and to invoke the strongest passions for events with the biggest consequences:

Falling in love... birthing a child... finding food when you're starving... getting wounded... being chased by a tiger... your child being chased by a tiger... finally killing a hated enemy...

Our life stories are not now, and will not be, what they once were.

If one is to be conservative in the short run about changing minds, then we can get at least some mileage from changing the environment. A windowless office filled with highly repetitive non-novel challenges isn't any more conducive to emotional involvement than video games; it may be part of real life, but it's a very flat part. The occasional exciting global economic crash that you had no personal control over, does not particularly modify this observation.

But we don't want to go back to the original savanna, the one where you got a leg chewed off and then starved to death once you couldn't walk. There are things we care about tremendously in the sense of hating them so much that we want to drive their frequency down to zero, not by the most interesting way, just as quickly as possible, whatever the means. If you drive the thing it binds to down to zero, where is the emotion after that?

And there are emotions we might want to think twice about keeping, in the long run. Does racial prejudice accomplish anything worthwhile? I pick this as a target, not because it's a convenient whipping boy, but because unlike e.g. "boredom" it's actually pretty hard to think of a reason transhumans would want to keep this neural circuitry around. Readers who take this as a challenge are strongly advised to remember that the point of the question is not to show off how clever and counterintuitive you can be.

But if you lose emotions without replacing them, whether by changing minds, or by changing life stories, then the world gets a little less involving each time; there's that much less material for passion. And your mind and your life become that much simpler, perhaps, because there are fewer forces at work - maybe even threatening to collapse you into an expected pleasure maximizer. If you don't replace what is removed.

In the long run, if humankind is to make a new life for itself...

We, and our descendants, will need some new emotions.

This is the aspect of self-modification in which one must above all take care - modifying your goals. Whatever you want, becomes more likely to happen; to ask what we ought to make ourselves want, is to ask what the future should be.

Add emotions at random - bind positive reinforcers or negative reinforcers to random situations and ways the world could be - and you'll just end up doing what is prime instead of what is good. So adding a bunch of random emotions does not seem like the way to go.

Asking what happens often, and binding happy emotions to that, so as to increase happiness - or asking what seems easy, and binding happy emotions to that - making isolated video games artificially more emotionally involving, for example -

At that point, it seems to me, you've pretty much given up on eudaimonia and moved to maximizing happiness; you might as well replace brains with pleasure centers, and civilizations with hedonium plasma.

I'd suggest, rather, that one start with the idea of new major events in a transhuman life, and then bind emotions to those major events and the sub-events that surround them. What sort of major events might a transhuman life embrace? Well, this is the point at which I usually stop speculating. "Science! They should be excited by science!" is something of a bit-too-obvious and I dare say "nerdy" answer, as is "Math!" or "Money!" (Money is just our civilization's equivalent of expected utilon balancing anyway.) Creating a child - as in my favored saying, "If you can't design an intelligent being from scratch, you're not old enough to have kids" - is one candidate for a major transhuman life event, and anything you had to do along the way to creating a child would be a candidate for new emotions. This might or might not have anything to do with sex - though I find that thought appealing, being something of a traditionalist. All sorts of interpersonal emotions carry over for as far as my own human eyes can see - the joy of making allies, say; interpersonal emotions get more complex (and challenging) along with the people, which makes them an even richer source of future fun. Falling in love? Well, it's not as if we're trying to construct the Future out of anything other than our preferences - so do you want that to carry over?

But again - this is usually the point at which I stop speculating. It's hard enough to visualize human Eutopias, let alone transhuman ones.

The essential idiom I'm suggesting is something akin to how evolution gave humans lots of local reinforcers for things that in the ancestral environment related to evolution's overarching goal of inclusive reproductive fitness. Today, office work might be highly relevant to someone's sustenance, but - even leaving aside the lack of high challenge and complex novelty - and that it's not sensually involving because we don't have native brainware to support the domain - office work is not emotionally involving because office work wasn't ancestrally relevant. If office work had been around for millions of years, we'd find it a little less hateful, and experience a little more triumph on filling out a form, one suspects.

Now you might run away shrieking from the dystopia I've just depicted - but that's because you don't see office work as eudaimonic in the first place, one suspects. And because of the lack of high challenge and complex novelty involved. In an "absolute" sense, office work would seem somewhat less tedious than gathering fruits and eating them.

But the idea isn't necessarily to have fun doing office work. Just like it's not necessarily the idea to have your emotions activate for video games instead of real life.

The idea is that once you construct an existence / life story that seems to make sense, then it's all right to bind emotions to the parts of that story, with strength proportional to their long-term impact. The anomie of today's world, where we simultaneously (a) engage in office work and (b) lack any passion in it, does not need to carry over: you should either fix one of those problems, or the other.

On a higher, more abstract level, this carries over the idiom of reinforcement over instrumental correlates of terminal values. In principle, this is something that a purer optimization process wouldn't do. You need neither happiness nor sadness to maximize expected utility. You only need to know which actions result in which consequences, and update that pure probability distribution as you learn through observation; something akin to "reinforcement" falls out of this, but without the risk of losing purposes, without any pleasure or pain. An agent like this is simpler than a human and more powerful - if you think that your emotions give you a supernatural advantage in optimization, you've entirely failed to understand the math of this domain. For a pure optimizer, the "advantage" of starting out with one more emotion bound to instrumental events is like being told one more abstract belief about which policies maximize expected utility, except that the belief is very hard to update based on further experience.

But it does not seem to me, that a mind which has the most value, is the same kind of mind that most efficiently optimizes values outside it. The interior of a true expected utility maximizer might be pretty boring, and I even suspect that you can build them to not be sentient.

For as far as my human eyes can see, I don't know what kind of mind I should value, if that mind lacks pleasure and happiness and emotion in the everyday events of its life. Bearing in mind that we are constructing this Future using our own preferences, not having it handed to us by some inscrutable external author.

If there's some better way of being (not just doing) that stands somewhere outside this, I have not yet understood it well enough to prefer it. But if so, then all this discussion of emotion would be as moot as it would be for an expected utility maximizer - one which was not valued at all for itself, but only valued for that which it maximized.

It's just hard to see why we would want to become something like that, bearing in mind that morality is not an inscrutable light handing down awful edicts from somewhere outside us.

At any rate - the hell of a life of disconnected episodes, where your actions don't connect strongly to anything you strongly care about, and nothing that you do all day invokes any passion - this angst seems avertible, however often it pops up in poorly written Utopias.

Sequence: Fun Theory

Changing Emotions

Serious Stories

Referenced by: Eutopia is Scary • Justified Expectation of Pleasant Surprises • Higher Purpose • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Emotional Involvement

Rationality Quotes 22

Eliezer Yudkowsky, 07 January 2009 12:00PM

"Two roads diverged in the woods. I took the one less traveled, and had to eat bugs until Park rangers rescued me."
-- Jim Rosenberg

"Lying to yourself about specific actions is easier than re-defining the bounds of your imagined identity... When I see once-ethical men devolve into moral grey, they still identify as upstanding."
-- Ben Casnocha

"Every year buy a clean bed sheet, date it, and lay it over the previous layer of junk on your desk."
-- Vernor Vinge, The Blabber

"Like first we tossed out the bath water, then the baby, and like finally the whole tub."
-- John R. Andrews

"I have no wisdom. Yet I heard a wise man - soon to be a relative of marriage - say not long ago that all is for the best. We are but dreams, and dreams possess no life by their own right. See, I am wounded. (Holds out his hand.) When my wound heals, it will be gone. Should it with its bloody lips say it is sorry to heal? I am only trying to explain what another said, but that is what I think he meant."
-- Gene Wolfe, The Claw of the Conciliator

"On a grand scale we simply want to save the world, so obviously we're just letting ourselves in for a lot of disappointment and we're doomed to failure since we didn't pick some cheap-ass two-bit goal like collecting all the Garbage Pail Kids cards."
-- Nenslo

"He promised them nothing but blood, iron, and fire, and offered them only the choice of going to find it or of waiting for it to find them at home."
-- John Barnes, One For the Morning Glory

Original with comments: Rationality Quotes 22

Serious Stories

Eliezer Yudkowsky, 08 January 2009 11:49PM

(This post is part of the Fun Theory Sequence.)

Every Utopia ever constructed - in philosophy, fiction, or religion - has been, to one degree or another, a place where you wouldn't actually want to live. I am not alone in this important observation: George Orwell said much the same thing in "Why Socialists Don't Believe In Fun", and I expect that many others said it earlier.

If you read books on How To Write - and there are a lot of books out there on How To Write, because amazingly a lot of book-writers think they know something about writing - these books will tell you that stories must contain "conflict".

That is, the more lukewarm sort of instructional book will tell you that stories contain "conflict". But some authors speak more plainly.

"Stories are about people's pain." Orson Scott Card.

"Every scene must end in disaster." Jack Bickham.

In the age of my youthful folly, I took for granted that authors were excused from the search for true Eutopia, because if you constructed a Utopia that wasn't flawed... what stories could you write, set there? "Once upon a time they lived happily ever after." What use would it be for a science-fiction author to try to depict a positive Singularity, when a positive Singularity would be...

...the end of all stories?

It seemed like a reasonable framework with which to examine the literary problem of Utopia, but something about that final conclusion produced a quiet, nagging doubt.

At that time I was thinking of an AI as being something like a safe wish-granting genie for the use of individuals. So the conclusion did make a kind of sense. If there was a problem, you would just wish it away, right? Ergo - no stories. So I ignored the quiet, nagging doubt.

Much later, after I concluded that even a safe genie wasn't such a good idea, it also seemed in retrospect that "no stories" could have been a productive indicator. On this particular occasion, "I can't think of a single story I'd want to read about this scenario", might indeed have pointed me toward the reason "I wouldn't want to actually live in this scenario".

So I swallowed my trained-in revulsion of Luddism and theodicy, and at least tried to contemplate the argument:

- A world in which nothing ever goes wrong, or no one ever experiences any pain or sorrow, is a world containing no stories worth reading about.

- A world that you wouldn't want to read about is a world where you wouldn't want to live.

- Into each eudaimonic life a little pain must fall. QED.

In one sense, it's clear that we do not want to live the sort of lives that are depicted in most stories that human authors have written so far. Think of the truly great stories, the ones that have become legendary for being the very best of the best of their genre: The Iliiad, Romeo and Juliet, The Godfather, Watchmen, Planescape: Torment, the second season of Buffy the Vampire Slayer, or that ending in Tsukihime. Is there a single story on the list that isn't tragic?

Ordinarily, we prefer pleasure to pain, joy to sadness, and life to death. Yet it seems we prefer to empathize with hurting, sad, dead characters. Or stories about happier people aren't serious, aren't artistically great enough to be worthy of praise - but then why selectively praise stories containing unhappy people? Is there some hidden benefit to us in it? It's a puzzle either way you look at it.

When I was a child I couldn't write fiction because I wrote things to go well for my characters - just like I wanted things to go well in real life. Which I was cured of by Orson Scott Card: Oh, I said to myself, that's what I've been doing wrong, my characters aren't hurting. Even then, I didn't realize that the microstructure of a plot works the same way - until Jack Bickham said that every scene must end in disaster. Here I'd been trying to set up problems and resolve them, instead of making them worse...

You simply don't optimize a story the way you optimize a real life. The best story and the best life will be produced by different criteria.

In the real world, people can go on living for quite a while without any major disasters, and still seem to do pretty okay. When was the last time you were shot at by assassins? Quite a while, right? Does your life seem emptier for it?

But on the other hand...

For some odd reason, when authors get too old or too successful, they revert to my childhood. Their stories start going right. They stop doing horrible things to their characters, with the result that they start doing horrible things to their readers. It seems to be a regular part of Elder Author Syndrome. Mercedes Lackey, Laurell K. Hamilton, Robert Heinlein, even Orson Scott bloody Card - they all went that way. They forgot how to hurt their characters. I don't know why.

And when you read a story by an Elder Author or a pure novice - a story where things just relentlessly go right one after another - where the main character defeats the supervillain with a snap of the fingers, or even worse, before the final battle, the supervillain gives up and apologizes and then they're friends again -

It's like a fingernail scraping on a blackboard at the base of your spine. If you've never actually read a story like that (or worse, written one) then count yourself lucky.

That fingernail-scraping quality - would it transfer over from the story to real life, if you tried living real life without a single drop of rain?

One answer might be that what a story really needs is not "disaster", or "pain", or even "conflict", but simply striving. That the problem with Mary Sue stories is that there's not enough striving in them, but they wouldn't actually need pain. This might, perhaps, be tested.

An alternative answer might be that this is the transhumanist version of Fun Theory we're talking about. So we can reply, "Modify brains to eliminate that fingernail-scraping feeling", unless there's some justification for keeping it. If the fingernail-scraping feeling is a pointless random bug getting in the way of Utopia, delete it.

Maybe we should. Maybe all the Great Stories are tragedies because... well...

I once read that in the BDSM community, "intense sensation" is a euphemism for pain. Upon reading this, it occurred to me that, the way humans are constructed now, it is just easier to produce pain than pleasure. Though I speak here somewhat outside my experience, I expect that it takes a highly talented and experienced sexual artist working for hours to produce a good feeling as intense as the pain of one strong kick in the testicles - which is doable in seconds by a novice.

Investigating the life of the priest and proto-rationalist Friedrich Spee von Langenfeld, who heard the confessions of accused witches, I looked up some of the instruments that had been used to produce confessions. There is no ordinary way to make a human being feel as good as those instruments would make you hurt. I'm not sure even drugs would do it, though my experience of drugs is as nonexistent as my experience of torture.

There's something imbalanced about that.

Yes, human beings are too optimistic in their planning. If losses weren't more aversive than gains, we'd go broke, the way we're constructed now. The experimental rule is that losing a desideratum - $50, a coffee mug, whatever - hurts between 2 and 2.5 times as much as the equivalent gain.

But this is a deeper imbalance than that. The effort-in/intensity-out difference between sex and torture is not a mere factor of 2.

If someone goes in search of sensation - in this world, the way human beings are constructed now - it's not surprising that they should arrive at pains to be mixed into their pleasures as a source of intensity in the combined experience.

If only people were constructed differently, so that you could produce pleasure as intense and in as many different flavors as pain! If only you could, with the same ingenuity and effort as a torturer of the Inquisition, make someone feel as good as the Inquisition's victims felt bad -

But then, what is the analogous pleasure that feels that good? A victim of skillful torture will do anything to stop the pain and anything to prevent it from being repeated. Is the equivalent pleasure one that overrides everything with the demand to continue and repeat it? If people are stronger-willed to bear the pleasure, is it really the same pleasure?

There is another rule of writing which states that stories have to shout. A human brain is a long way off those printed letters. Every event and feeling needs to take place at ten times natural volume in order to have any impact at all. You must not try to make your characters behave or feel realistically - especially, you must not faithfully reproduce your own past experiences - because without exaggeration, they'll be too quiet to rise from the page.

Maybe all the Great Stories are tragedies because happiness can't shout loud enough - to a human reader.

Maybe that's what needs fixing.

And if it were fixed... would there be any use left for pain or sorrow? For even the memory of sadness, if all things were already as good as they could be, and every remediable ill already remedied?

Can you just delete pain outright? Or does removing the old floor of the utility function just create a new floor? Will any pleasure less than 10,000,000 hedons be the new unbearable pain?

Humans, built the way we are now, do seem to have hedonic scaling tendencies. Someone who can remember starving will appreciate a loaf of bread more than someone who's never known anything but cake. This was George Orwell's hypothesis for why Utopia is impossible in literature and reality:

"It would seem that human beings are not able to describe, nor perhaps to imagine, happiness except in terms of contrast... The inability of mankind to imagine happiness except in the form of relief, either from effort or pain, presents Socialists with a serious problem. Dickens can describe a poverty-stricken family tucking into a roast goose, and can make them appear happy; on the other hand, the inhabitants of perfect universes seem to have no spontaneous gaiety and are usually somewhat repulsive into the bargain."

For an expected utility maximizer, rescaling the utility function to add a trillion to all outcomes is meaningless - it's literally the same utility function, as a mathematical object. A utility function describes the relative intervals between outcomes; that's what it is, mathematically speaking.

But the human brain has distinct neural circuits for positive feedback and negative feedback, and different varieties of positive and negative feedback. There are people today who "suffer" from congenital analgesia - a total absence of pain. I never heard that insufficient pleasure becomes intolerable to them.

Congenital analgesics do have to inspect themselves carefully and frequently to see if they've cut themselves or burned a finger. Pain serves a purpose in the human mind design...

But that does not show there's no alternative which could serve the same purpose. Could you delete pain and replace it with an urge not to do certain things that lacked the intolerable subjective quality of pain? I do not know all the Law that governs here, but I'd have to guess that yes, you could; you could replace that side of yourself with something more akin to an expected utility maximizer.

Could you delete the human tendency to scale pleasures - delete the accomodation, so that each new roast goose is as delightful as the last? I would guess that you could. This verges perilously close to deleting Boredom, which is right up there with Sympathy as an absolute indispensable... but to say that an old solution remains as pleasurable, is not to say that you will lose the urge to seek new and better solutions.

Can you make every roast goose as pleasurable as it would be in contrast to starvation, without ever having starved?

Can you prevent the pain of a dust speck irritating your eye from being the new torture, if you've literally never experienced anything worse than a dust speck irritating your eye?

Such questions begin to exceed my grasp of the Law, but I would guess that the answer is: yes, it can be done. It is my experience in such matters that once you do learn the Law, you can usually see how to do weird-seeming things.

So far as I know or can guess, David Pearce (The Hedonistic Imperative) is very probably right about the feasibility part, when he says:

"Nanotechnology and genetic engineering will abolish suffering in all sentient life. The abolitionist project is hugely ambitious but technically feasible. It is also instrumentally rational and morally urgent. The metabolic pathways of pain and malaise evolved because they served the fitness of our genes in the ancestral environment. They will be replaced by a different sort of neural architecture - a motivational system based on heritable gradients of bliss. States of sublime well-being are destined to become the genetically pre-programmed norm of mental health. It is predicted that the world's last unpleasant experience will be a precisely dateable event."

Is that... what we want?

To just wipe away the last tear, and be done?

Is there any good reason not to, except status quo bias and a handful of worn rationalizations?

What would be the alternative? Or alternatives?

To leave things as they are? Of course not. No God designed this world; we have no reason to think it exactly optimal on any dimension. If this world does not contain too much pain, then it must not contain enough, and the latter seems unlikely.

But perhaps...

You could cut out just the intolerable parts of pain?

Get rid of the Inquisition. Keep the sort of pain that tells you not to stick your finger in the fire, or the pain that tells you that you shouldn't have put your friend's finger in the fire, or even the pain of breaking up with a lover.

Try to get rid of the sort of pain that grinds down and destroys a mind. Or configure minds to be harder to damage.

You could have a world where there were broken legs, or even broken hearts, but no broken people. No child sexual abuse that turns out more abusers. No people ground down by weariness and drudging minor inconvenience to the point where they contemplate suicide. No random meaningless endless sorrows like starvation or AIDS.

And if even a broken leg still seems too scary -

Would we be less frightened of pain, if we were stronger, if our daily lives did not already exhaust so much of our reserves?

So that would be one alternative to the Pearce's world - if there are yet other alternatives, I haven't thought them through in any detail.

The path of courage, you might call it - the idea being that if you eliminate the destroying kind of pain and strengthen the people, then what's left shouldn't be that scary.

A world where there is sorrow, but not massive systematic pointless sorrow, like we see on the evening news. A world where pain, if it is not eliminated, at least does not overbalance pleasure. You could write stories about that world, and they could read our stories.

I do tend to be rather conservative around the notion of deleting large parts of human nature. I'm not sure how many major chunks you can delete until that balanced, conflicting, dynamic structure collapses into something simpler, like an expected pleasure maximizer.

And so I do admit that it is the path of courage that appeals to me.

Then again, I haven't lived it both ways.

Maybe I'm just afraid of a world so different as Analgesia - wouldn't that be an ironic reason to walk "the path of courage"?

Maybe the path of courage just seems like the smaller change - maybe I just have trouble empathizing over a larger gap.

But "change" is a moving target.

If a human child grew up in a less painful world - if they had never lived in a world of AIDS or cancer or slavery, and so did not know these things as evils that had been triumphantly eliminated - and so did not feel that they were "already done" or that the world was "already changed enough"...

Would they take the next step, and try to eliminate the unbearable pain of broken hearts, when someone's lover stops loving them?

And then what? Is there a point where Romeo and Juliet just seems less and less relevant, more and more a relic of some distant forgotten world? Does there come some point in the transhuman journey where the whole business of the negative reinforcement circuitry, can't possibly seem like anything except a pointless hangover to wake up from?

And if so, is there any point in delaying that last step? Or should we just throw away our fears and... throw away our fears?

I don't know.

Sequence: Fun Theory

Emotional Involvement

Eutopia is Scary

Referenced by: Continuous Improvement • Eutopia is Scary • The Fun Theory Sequence • 31 Laws of Fun • (Moral) Truth in Fiction?

Original with comments: Serious Stories

Rationality Quotes 23

Eliezer Yudkowsky, 10 January 2009 12:12AM

"This year I resolve to lose weight... to be nicer to dogs... and to sprout wings and fly."
-- Garfield

"People understand instinctively that the best way for computer programs to communicate with each other is for each of them to be strict in what they emit, and liberal in what they accept. The odd thing is that people themselves are not willing to be strict in how they speak and liberal in how they listen. You'd think that would also be obvious."
-- Larry Wall

"One never needs enemies, but they are so much fun to acquire."
-- Eluki bes Shahar, Archangel Blues

"I'm accusing you of violating the laws of nature!"
"Nature's virtue is intact. I just know some different laws."
-- Orson Scott Card, A Planet Called Treason

"The goal of most religions is to preserve and elaborate on that concept of the stars as a big painted backdrop. They make Infinity a 'prop' so you don't have to think about the scary part."
-- The Book of the SubGenius

<silverpower> Is humanity even worth saving?
<starglider> As opposed to what?
<silverpower> ...hmm.
-- #sl4

Original with comments: Rationality Quotes 23

Continuous Improvement

Eliezer Yudkowsky, 11 January 2009 02:09AM

(This post is part of the Fun Theory Sequence.)

When is it adaptive for an organism to be satisfied with what it has? When does an organism have enough children and enough food? The answer to the second question, at least, is obviously "never" from an evolutionary standpoint. The first proposition might be true if the reproductive risks of all available options exceed their reproductive benefits. In general, though, it is a rare organism in a rare environment whose reproductively optimal strategy is to rest with a smile on its face, feeling happy.

To a first approximation, we might say something like "The evolutionary purpose of emotion is to direct the cognitive processing of the organism toward achievable, reproductively relevant goals". Achievable goals are usually located in the Future, since you can't affect the Past. Memory is a useful trick, but learning the lesson of a success or failure isn't the same goal as the original event - and usually the emotions associated with the memory are less intense than those of the original event.

Then the way organisms and brains are built right now, "true happiness" might be a chimera, a carrot dangled in front of us to make us take the next step, and then yanked out of our reach as soon as we achieve our goals.

This hypothesis is known as the hedonic treadmill.

The famous pilot studies in this domain demonstrated e.g. that past lottery winners' stated subjective well-being was not significantly greater than that of an average person, after a few years or even months. Conversely, accident victims with severed spinal cords were not as happy as before the accident after six months - around 0.75 sd less than control groups - but they'd still adjusted much more than they had expected to adjust.

This being the transhumanist form of Fun Theory, you might perhaps say: "Let's get rid of this effect. Just delete the treadmill, at least for positive events."

I'm not entirely sure we can get away with this. There's the possibility that comparing good events to not-as-good events is what gives them part of their subjective quality. And on a moral level, it sounds perilously close to tampering with Boredom itself.

So suppose that instead of modifying minds and values, we first ask what we can do by modifying the environment. Is there enough fun in the universe, sufficiently accessible, for a transhuman to jog off the hedonic treadmill - improve their life continuously, at a sufficient rate to leap to an even higher hedonic level before they had a chance to get bored with the previous one?

This question leads us into great and interesting difficulties.

I had a nice vivid example I wanted to use for this, but unfortunately I couldn't find the exact numbers I needed to illustrate it. I'd wanted to find a figure for the total mass of the neurotransmitters released in the pleasure centers during an average male or female orgasm, and a figure for the density of those neurotransmitters - density in the sense of mass/volume of the chemicals themselves. From this I could've calculated how long a period of exponential improvement would be possible - how many years you could have "the best orgasm of your life" by a margin of at least 10%, at least once per year - before your orgasm collapsed into a black hole, the total mass having exceeded the mass of a black hole with the density of the neurotransmitters.

Plugging in some random/Fermi numbers instead:

Assume that a microgram of additional neurotransmitters are released in the pleasure centers during a standard human orgasm. And assume that neurotransmitters have the same density as water. Then an orgasm can reach around 10^8 solar masses before it collapses and forms a black hole, corresponding to 10^47 baseline orgasms. If we assume that a 100mg dose of crack is as pleasurable as 10 standard orgasms, then the street value of your last orgasm is around a hundred billion trillion trillion trillion dollars.

I'm sorry. I just had to do that calculation.

Anyway... requiring an exponential improvement eats up a factor of 10^47 in short order. Starting from human standard and improving at 10% per year, it would take less than 1,200 years.

Of course you say, "This but shows the folly of brains that use an analog representation of pleasure. Go digital, young man!"

If you redesigned the brain to represent the intensity of pleasure using IEEE 754 double-precision floating-point numbers, a mere 64 bits would suffice to feel pleasures up to 10^308 hedons... in, um, whatever base you were using.

This still represents less than 7500 years of 10% annual improvement from a 1-hedon baseline, but after that amount of time, you can switch to larger floats.

Now we have lost a bit of fine-tuning by switching to IEEE-standard hedonics. The 64-bit double-precision float has an 11-bit exponent and a 52-bit fractional part (and a 1-bit sign). So we'll only have 52 bits of precision (16 decimal places) with which to represent our pleasures, however great they may be. An original human's orgasm would soon be lost in the rounding error... which raises the question of how we can experience these invisible hedons, when the finite-precision bits are the whole substance of the pleasure.

We also have the odd situation that, starting from 1 hedon, flipping a single bit in your brain can make your life 10^154 times more happy.

And Hell forbid you flip the sign bit. Talk about a need for cosmic ray shielding.

But really - if you're going to go so far as to use imprecise floating-point numbers to represent pleasure, why stop there? Why not move to Knuth's up-arrow notation?

For that matter, IEEE 754 provides special representations for +/- INF, that is to say, positive and negative infinity. What happens if a bit flip makes you experience infinite pleasure? Does that mean you Win The Game?

Now all of these questions I'm asking are in some sense unfair, because right now I don't know exactly what I have to do with any structure of bits in order to turn it into a "subjective experience". Not that this is the right way to phrase the question. It's not like there's a ritual that summons some incredible density of positive qualia that could collapse in its own right and form an epiphenomenal black hole.

But don't laugh - or at least, don't only laugh - because in the long run, these are extremely important questions.

To give you some idea of what's at stake here, Robin, in "For Discount Rates", pointed out that an investment earning 2% annual interest for 12,000 years adds up to a googol (10^100) times as much wealth; therefore, "very distant future times are ridiculously easy to help via investment".

I observed that there weren't a googol atoms in the observable universe, let alone within a 12,000-lightyear radius of Earth.

And Robin replied, "I know of no law limiting economic value per atom."

If you've got an increasingly large number of bits - things that can be one or zero - and you're doing a proportional number of computations with them... then how fast can you grow the amount of fun, or pleasure, or value?

This echoes back to the questions in Complex Novelty, which asked how many kinds of problems and novel solutions you could find, and how many deep insights there were to be had. I argued there that the growth rate is faster than linear in bits, e.g., humans can have much more than four times as much fun as chimpanzees even though our absolute brain volume is only around four times theirs. But I don't think the growth in "depth of good insights" or "number of unique novel problems" is, um, faster than exponential in the size of the pattern.

Now... it might be that the Law simply permits outright that we can create very large amounts of subjective pleasure, every bit as substantial as the sort of subjective pleasure we get now, by the expedient of writing down very large numbers in a digital pleasure center. In this case, we have got it made. Have we ever got it made.

In one sense I can definitely see where Robin is coming from. Suppose that you had a specification of the first 10,000 Busy Beaver machines - the longest-running Turing machines with 1, 2, 3, 4, 5... states. This list could easily fit on a small flash memory card, made up of a few measly avogadros of atoms.

And that small flash memory card would be worth...

Well, let me put it this way: If a mathematician said to me that the value of this memory card, was worth more than the rest of the entire observable universe minus the card... I wouldn't necessarily agree with him outright. But I would understand his point of view.

Still, I don't know if you can truly grok the fun contained in that memory card, without an unbounded amount of computing power with which to understand it. Ultradense information does not give you ultradense economic value or ultradense fun unless you can also use that information in a way that consumes few resources. Otherwise it's just More Fun Than You Can Handle.

Weber's Law of Just Noticeable Difference says that stimuli with an intensity scale, typically require a fixed fraction of proportional difference, rather than any fixed interval of absolute intensity, in order for the difference to be noticeable to a human or other organism. In other words, we may demand exponential increases because our imprecise brains can't notice smaller differences. This would suggest that our existing pleasures might already in effect possess a floating-point representation, with an exponent and a fraction - the army of actual neurons being used only to transmit an analog signal most of whose precision is lost. So we might be able to get away with using floats, even if we can't get away with using up-arrows.

But suppose that the inscrutable rules governing the substantiality of "subjective" pleasure actually require one neuron per hedon, or something like that.

Or suppose that we only choose to reward ourselves when we find a better solution, and that we don't choose to game the betterness metrics.

And suppose that we don't discard the Weber-Fechner law of "just noticeable difference", but go on demanding percentage annual improvements, year after year.

Or you might need to improve at a fractional rate in order to assimilate your own memories. Larger brains would lay down larger memories, and hence need to grow exponentially - efficiency improvements suiting to moderate the growth, but not to eliminate the exponent.

If fun or intelligence or value can only grow as fast as the mere cube of the brain size... and yet we demand a 2% improvement every year...

Then 350 years will pass before our resource consumption grows a single order of magnitude.

And yet there are only around 10^80 atoms in the observable universe.

Do the math.

(It works out to a lifespan of around 28,000 years.)

Now... before everyone gets all depressed about this...

We can still hold out a fraction of hope for real immortality, aka "emortality". As Greg Egan put it, "Not dying after a very long time. Just not dying, period."

The laws of physics as we know them prohibit emortality on multiple grounds. It is a fair historical observation that, over the course of previous centuries, civilizations have become able to do things that previous civilizations called "physically impossible". This reflects a change in knowledge about the laws of physics, not a change in the actual laws; and we cannot do everything once thought to be impossible. We violate Newton's version of gravitation, but not conservation of energy. It's a good historical bet that the future will be able to do at least one thing our physicists would call impossible. But you can't bank on being able to violate any particular "law of physics" in the future.

There is just... a shred of reasonable hope, that our physics might be much more incomplete than we realize, or that we are wrong in exactly the right way, or that anthropic points I don't understand might come to our rescue and let us escape these physics (also a la Greg Egan).

So I haven't lost hope. But I haven't lost despair, either; that would be faith.

In the case where our resources really are limited and there is no way around it...

...the question of how fast a rate of continuous improvement you demand for an acceptable quality of life - an annual percentage increase, or a fixed added amount - and the question of how much improvement you can pack into patterns of linearly increasing size - adding up to the fun-theoretic question of how fast you have to expand your resource usage over time to lead a life worth living...

...determines the maximum lifespan of sentient beings.

If you can get by with increasing the size in bits of your mind at a linear rate, then you can last for quite a while. Until the end of the universe, in many versions of cosmology. And you can have a child (or two parents can have two children), and the children can have children. Linear brain size growth * linear population growth = quadratic growth, and cubic growth at lightspeed should be physically permissible.

But if you have to grow exponentially, in order for your ever-larger mind and its ever-larger memories not to end up uncomfortably squashed into too small a brain - squashed down to a point, to the point of it being pointless - then a transhuman's life is measured in subjective eons at best, and more likely subjective millennia. Though it would be a merry life indeed.

My own eye has trouble enough looking ahead a mere century or two of growth. It's not like I can imagine any sort of me the size of a galaxy. I just want to live one more day, and tomorrow I will still want to live one more day. The part about "wanting to live forever" is just an induction on the positive integers, not an instantaneous vision whose desire spans eternity.

If I can see to the fulfillment of all my present self's goals that I can concretely envision, shouldn't that be enough for me? And my century-older self will also be able to see that far ahead. And so on through thousands of generations of selfhood until some distant figure the size of a galaxy has to depart the physics we know, one way or the other... Should that be scary?

Yeah, I hope like hell that emortality is possible.

Failing that, I'd at least like to find out one way or the other, so I can get on with my life instead of having that lingering uncertainty.

For now, one of the reasons I care about people alive today is the thought that if creating new people just divides up a finite pool of resource available here, but we live in a Big World where there are plenty of people elsewhere with their own resources... then we might not want to create so many new people here. Six billion now, six trillion at the end of time? Though this is more an idiom of linear growth than exponential - with exponential growth, a factor of 10 fewer people just buys you another 350 years of lifespan per person, or whatever.

But I do hope for emortality. Odd, isn't it? How abstract should a hope or fear have to be, before a human can stop thinking about it?

Oh, and finally - there's an idea in the literature of hedonic psychology called the "hedonic set point", based on identical twin studies showing that identical twins raised apart have highly similar happiness levels, more so than fraternal twins raised together, people in similar life circumstances, etcetera. There are things that do seem to shift your set point, but not much (and permanent downward shift happens more easily than permanent upward shift, what a surprise). Some studies have suggested that up to 80% of the variance in happiness is due to genes, or something shared between identical twins in different environments at any rate.

If no environmental improvement ever has much effect on subjective well-being, the way you are now, because you've got a more or less genetically set level of happiness that you drift back to, then...

Well, my usual heuristic is to imagine messing with environments before I imagine messing with minds.

But in this case? Screw that. That's just stupid. Delete it without a qualm.

Sequence: Fun Theory

Complex Novelty

Sensual Experience

Referenced by: Justified Expectation of Pleasant Surprises • Seduced by Imagination • The Fun Theory Sequence • 31 Laws of Fun • An Especially Elegant Evpsych Experiment

Original with comments: Continuous Improvement

Eutopia is Scary

Eliezer Yudkowsky, 12 January 2009 05:28AM

(This post is part of the Fun Theory Sequence.)
Followup to: Why is the Future So Absurd?

"The big thing to remember about far-future cyberpunk is that it will be truly ultra-tech. The mind and body changes available to a 23rd-century Solid Citizen would probably amaze, disgust and frighten that 2050 netrunner!"
-- GURPS Cyberpunk

Pick up someone from the 18th century - a smart someone. Ben Franklin, say. Drop them into the early 21st century.

We, in our time, think our life has improved in the last two or three hundred years. Ben Franklin is probably smart and forward-looking enough to agree that life has improved. But if you don't think Ben Franklin would be amazed, disgusted, and frightened, then I think you far overestimate the "normality" of your own time. You can think of reasons why Ben should find our world compatible, but Ben himself might not do the same.

Movies that were made in say the 40s or 50s, seem much more alien - to me - than modern movies allegedly set hundreds of years in the future, or in different universes. Watch a movie from 1950 and you may see a man slapping a woman. Doesn't happen a lot in Lord of the Rings, does it? Drop back to the 16th century and one popular entertainment was setting a cat on fire. Ever see that in any moving picture, no matter how "lowbrow"?

("But," you say, "that's showing how discomforting the Past's culture was, not how scary the Future is." Of which I wrote, "When we look over history, we see changes away from absurd conditions such as everyone being a peasant farmer and women not having the vote, toward normal conditions like a majority middle class and equal rights...")

Something about the Future will shock we 21st-century folk, if we were dropped in without slow adaptation. This is not because the Future is cold and gloomy - I am speaking of a positive, successful Future; the negative outcomes are probably just blank. Nor am I speaking of the idea that every Utopia has some dark hidden flaw. I am saying that the Future would discomfort us because it is better.

This is another piece of the puzzle for why no author seems to have ever succeeded in constructing a Utopia worth-a-damn. When they are out to depict how marvelous and wonderful the world could be, if only we would all be Marxists or Randians or let philosophers be kings... they try to depict the resulting outcome as comforting and safe.

Again, George Orwell from "Why Socialists Don't Believe In Fun":

"In the last part, in contrast with disgusting Yahoos, we are shown the noble Houyhnhnms, intelligent horses who are free from human failings. Now these horses, for all their high character and unfailing common sense, are remarkably dreary creatures. Like the inhabitants of various other Utopias, they are chiefly concerned with avoiding fuss. They live uneventful, subdued, 'reasonable' lives, free not only from quarrels, disorder or insecurity of any kind, but also from 'passion', including physical love. They choose their mates on eugenic principles, avoid excesses of affection, and appear somewhat glad to die when their time comes."

One might consider, in particular contrast, Timothy Ferris's observation:

"What is the opposite of happiness? Sadness? No. Just as love and hate are two sides of the same coin, so are happiness and sadness. Crying out of happiness is a perfect illustration of this. The opposite of love is indifference, and the opposite of happiness is - here's the clincher - boredom...
The question you should be asking isn't 'What do I want?' or 'What are my goals?' but 'What would excite me?'
Remember - boredom is the enemy, not some abstract 'failure.'"

Utopia is reassuring, unsurprising, and dull.

Eutopia is scary.

I'm not talking here about evil means to a good end, I'm talking about the good outcomes themselves. That is the proper relation of the Future to the Past when things turn out well, as we would know very well from history if we'd actually lived it, rather than looking back with benefit of hindsight.

Now... I don't think you can actually build the Future on the basis of asking how to scare yourself. The vast majority of possible changes are in the direction of higher entropy; only a very few discomforts stem from things getting better.

"I shock you therefore I'm right" is one of the most annoying of all non-sequiturs, and we certainly don't want to go there.

But on a purely literary level... and bearing in mind that fiction is not reality, and fiction is not optimized the way we try to optimize reality...

I try to write fiction, now and then. More rarely, I finish a story. Even more rarely, I let someone else look at it.

Once I finally got to the point of thinking that maybe you should be able to write a story set in Eutopia, I tried doing it.

But I had something like an instinctive revulsion at the indulgence of trying to build a world that fit me, but probably wouldn't fit others so nicely.

So - without giving the world a seamy underside, or putting Knight Templars in charge, or anything so obvious as that - without deliberately trying to make the world flawed -

I was trying to invent, even if I had to do it myself, a better world where I would be out of place. Just like Ben Franklin would be out of place in the modern world.

Definitely not someplace that a transhumanist/science-advocate/libertarian (like myself) would go, and be smugly satisfied at how well all their ideas had worked. Down that path lay the Dark Side - certainly in a purely literary sense.

And you couldn't avert that just by having the Future go wrong in all the stupid obvious ways that transhumanists, or libertarians, or public advocates of science had already warned against. Then you just had a dystopia, and it might make a good SF story but it had already been done.

But I had my world's foundation, an absurd notion inspired by a corny pun; a vision of what you see when you wake up from cryonic suspension, that I couldn't have gotten away with posting to any transhumanist mailing list even as a joke.

And then, whenever I could think of an arguably-good idea that offended my sensibilities, I added it in. The goal being to - without ever deliberately making the Future worse - make it a place where I would be as shocked as possible to see that that was how things had turned out.

Getting rid of textbooks, for example - postulating that talking about science in public is socially unacceptable, for the same reason that you don't tell someone aiming to see a movie whether the hero dies at the end. A world that had rejected my beloved concept of science as the public knowledge of humankind.

Then I added up all the discomforting ideas together...

...and at least in my imagination, it worked better than anything I'd ever dared to visualize as a serious proposal.

My serious proposals had been optimized to look sober and safe and sane; everything voluntary, with clearly lighted exit signs, and all sorts of volume controls to prevent anything from getting too loud and waking up the neighbors. Nothing too absurd. Proposals that wouldn't scare the nervous, containing as little as possible that would cause anyone to make a fuss.

This world was ridiculous, and it was going to wake up the neighbors.

It was also seductive to the point that I had to exert a serious effort to prevent my soul from getting sucked out. (I suspect that's a general problem; that it's a good idea emotionally (not just epistemically) to not visualize your better Future in too much detail. You're better off comparing yourself to the Past. I may write a separate post on this.)

And so I found myself being pulled in the direction of this world in which I was supposed to be "out of place". I started thinking that, well, maybe it really would be a good idea to get rid of all the textbooks, all they do is take the fun out of science. I started thinking that maybe personal competition was a legitimate motivator (previously, I would have called it a zero-sum game and been morally aghast). I began to worry that peace, democracy, market economies, and con - but I'd better not finish that sentence. I started to wonder if the old vision that was so reassuring, so safe, was optimized to be good news to a modern human living in constant danger of permanent death or damage, and less optimized for the everyday existence of someone less frightened.

This is what happens when I try to invent a world that fails to confirm my sensibilities? It makes me wonder what would happen if someone else tried the same exercise.

Unfortunately, I can't seem to visualize any new world that represents the same shock to me as the last one did. Either the trick only works once, or you have to wait longer between attempts, or I'm too old now.

But I hope that so long as the world offends the original you, it gets to keep its literary integrity even if you start to find it less shocking.

I haven't yet published any story that gives more than a glimpse of this setting. I'm still debating with myself whether I dare. I don't know whether the suck-out-your-soul effect would threaten anyone but myself as author - I haven't seen it happening with Banks's Culture or Wright's Golden Oecumene, so I suspect it's more of a trap when a world fits a single person too well. But I got enough flak when I presented the case for getting rid of textbooks.

Still - I have seen the possibilities, now. So long as no one dies permanently, I am leaning in favor of a loud and scary Future.

Sequence: Fun Theory

Serious Stories

Building Weirdtopia

Referenced by: Building Weirdtopia • Justified Expectation of Pleasant Surprises • Seduced by Imagination • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Eutopia is Scary

Building Weirdtopia

Eliezer Yudkowsky, 12 January 2009 08:35PM

(This post is part of the Fun Theory Sequence.)
Followup to: Eutopia is Scary

"Two roads diverged in the woods. I took the one less traveled, and had to eat bugs until Park rangers rescued me."
-- Jim Rosenberg

Utopia and Dystopia have something in common: they both confirm the moral sensibilities you started with. Whether the world is a libertarian utopia of the non-initiation of violence and everyone free to start their own business, or a hellish dystopia of government regulation and intrusion - you might like to find yourself in the first, and hate to find yourself in the second; but either way you nod and say, "Guess I was right all along."

So as an exercise in creativity, try writing them down side by side: Utopia, Dystopia, and Weirdtopia. The zig, the zag and the zog.

I'll start off with a worked example for public understanding of science:

- Utopia: Most people have the equivalent of an undergrad degree in something; everyone reads the popular science books (and they're good books); everyone over the age of nine understands evolutionary theory and Newtonian physics; scientists who make major contributions are publicly adulated like rock stars.

- Dystopia: Science is considered boring and possibly treasonous; public discourse elevates religion or crackpot theories; stem cell research is banned.

- Weirdtopia: Science is kept secret to avoid spoiling the surprises; no public discussion but intense private pursuit; cooperative ventures surrounded by fearsome initiation rituals because that's what it takes for people to feel like they've actually learned a Secret of the Universe and be satisfied; someone you meet may only know extremely basic science, but they'll have personally done revolutionary-level work in it, just like you. Too bad you can't compare notes.

Disclaimer 1: Not every sensibility we have is necessarily wrong. Originality is a goal of literature, not science; sometimes it's better to be right than to be new. But there are also such things as cached thoughts. At least in my own case, it turned out that trying to invent a world that went outside my pre-existing sensibilities, did me a world of good.

Disclaimer 2: This method is not universal: Not all interesting ideas fit this mold, and not all ideas that fit this mold are good ones. Still, it seems like an interesting technique.

If you're trying to write science fiction (where originality is a legitimate goal), then you can write down anything nonobvious for Weirdtopia, and you're done.

If you're trying to do Fun Theory, you have to come up with a Weirdtopia that's at least arguably-better than Utopia. This is harder but also directs you to more interesting regions of the answer space.

If you can make all your answers coherent with each other, you'll have quite a story setting on your hands. (Hope you know how to handle characterization, dialogue, description, conflict, and all that other stuff.)

Here's some partially completed challenges, where I wrote down a Utopia and a Dystopia (according to the moral sensibilities I started with before I did this exercise), but inventing a (better) Weirdtopia is left to the reader.

Economic...

- Utopia: The world is flat and ultra-efficient. Prices fall as standards of living rise, thanks to economies of scale. Anyone can easily start their own business and most people do. Everything is done in the right place by the right person under Ricardo's Law of Comparative Advantage. Shocks are efficiently absorbed by the risk capital that insured them.

- Dystopia: Lots of trade barriers and subsidies; corporations exploit the regulatory systems to create new barriers to entry; dysfunctional financial systems with poor incentives and lots of unproductive investments; rampant agent failures and systemic vulnerabilities; standards of living flat or dropping.

- Weirdtopia: _____

Sexual...

- Utopia: Sexual mores straight out of a Spider Robinson novel: Sexual jealousy has been eliminated; no one is embarrassed about what turns them on; universal tolerance and respect; everyone is bisexual, poly, and a switch; total equality between the sexes; no one would look askance on sex in public any more than eating in public, so long as the participants cleaned up after themselves.

- Dystopia: 10% of women have never had an orgasm. States adopt laws to ban gay marriage. Prostitution illegal.

- Weirdtopia: _____

Governmental...

- Utopia: Non-initiation of violence is the chief rule. Remaining public issues are settled by democracy: Well reasoned public debate in which all sides get a free voice, followed by direct or representative majority vote. Smoothly interfunctioning Privately Produced Law, which coordinate to enforce a very few global rules like "no slavery".

- Dystopia: Tyranny of a single individual or oligarchy. Politicians with effective locks on power thanks to corrupted electronic voting systems, voter intimidation, voting systems designed to create coordination problems. Business of government is unpleasant and not very competitive; hard to move from one region to another.

- Weirdtopia: _____

Technological...

- Utopia: All Kurzweilian prophecies come true simultaneously. Every pot contains a chicken, a nanomedical package, a personal spaceship, a superdupercomputer, amazing video games, and a pet AI to help you use it all, plus a pony. Everything is designed by Apple.

- Dystopia: Those damned fools in the government banned everything more complicated than a lawnmower, and we couldn't use our lawnmowers after Peak Oil hit.

- Weirdtopia: _____

Cognitive...

- Utopia: Brain-computer implants for everyone! You can do whatever you like with them, it's all voluntary and the dangerous buttons are clearly labeled. There are AIs around that are way more powerful than you; but they don't hurt you unless you ask to be hurt, sign an informed consent release form and click "Yes" three times.

- Dystopia: The first self-improving AI was poorly designed, everyone's dead and the universe is being turned into paperclips. Or the augmented humans hate the normals. Or augmentations make you go nuts. Or the darned government banned everything again, and people are still getting Alzheimers due to lack of stem-cell research.

- Weirdtopia: _____

Sequence: Fun Theory

Eutopia is Scary

Justified Expectation of Pleasant Surprises

Referenced by: The Fun Theory Sequence • 31 Laws of Fun • This Failing Earth

Original with comments: Building Weirdtopia

She has joined the Conspiracy

Eliezer Yudkowsky, 13 January 2009 07:48PM

I have no idea whether I had anything to do with this.

Original with comments: She has joined the Conspiracy

Justified Expectation of Pleasant Surprises

Eliezer Yudkowsky, 15 January 2009 07:26AM

(This post is part of the Fun Theory Sequence.)

I recently tried playing a computer game that made a major fun-theoretic error. (At least I strongly suspect it's an error, though they are game designers and I am not.)

The game showed me - right from the start of play - what abilities I could purchase as I increased in level. Worse, there were many different choices; still worse, you had to pay a cost in fungible points to acquire them, making you feel like you were losing a resource... But today, I'd just like to focus on the problem of telling me, right at the start of the game, about all the nice things that might happen to me later.

I can't think of a good experimental result that backs this up; but I'd expect that a pleasant surprise would have a greater hedonic impact, than being told about the same gift in advance. Sure, the moment you were first told about the gift would be good news, a moment of pleasure in the moment of being told. But you wouldn't have the gift in hand at that moment, which limits the pleasure. And then you have to wait. And then when you finally get the gift - it's pleasant to go from not having it to having it, if you didn't wait too long; but a surprise would have a larger momentary impact, I would think.

This particular game had a status screen that showed all my future class abilities at the start of the game - inactive and dark but with full information still displayed. From a hedonic standpoint this seems like miserable fun theory. All the "good news" is lumped into a gigantic package; the items of news would have much greater impact if encountered separately. And then I have to wait a long time to actually acquire the abilities, so I get an extended period of comparing my current weak game-self to all the wonderful abilities I could have but don't.

Imagine living in two possible worlds. Both worlds are otherwise rich in challenge, novelty, and other aspects of Fun. In both worlds, you get smarter with age and acquire more abilities over time, so that your life is always getting better.

But in one world, the abilities that come with seniority are openly discussed, hence widely known; you know what you have to look forward to.

In the other world, anyone older than you will refuse to talk about certain aspects of growing up; you'll just have to wait and find out.

I ask you to contemplate - not just which world you might prefer to live in - but how much you might want to live in the second world, rather than the first. I would even say that the second world seems more alive; when I imagine living there, my imagined will to live feels stronger. I've got to stay alive to find out what happens next, right?

The idea that hope is important to a happy life, is hardly original with me - though I think it might not be emphasized quite enough, on the lists of things people are told they need.

I don't agree with buying lottery tickets, but I do think I understand why people do it. I remember the times in my life when I had more or less belief that things would improve - that they were heading up in the near-term or mid-term, close enough to anticipate. I'm having trouble describing how much of a difference it makes. Maybe I don't need to describe that difference, unless some of my readers have never had any light at the end of their tunnels, or some of my readers have never looked forward and seen darkness.

If existential angst comes from having at least one deep problem in your life that you aren't thinking about explicitly, so that the pain which comes from it seems like a natural permanent feature - then the very first question I'd ask, to identify a possible source of that problem, would be, "Do you expect your life to improve in the near or mid-term future?"

Sometimes I meet people who've been run over by life, in much the same way as being run over by a truck. Grand catastrophe isn't necessary to destroy a will to live. The extended absence of hope leaves the same sort of wreckage.

People need hope. I'm not the first to say it.

But I think that the importance of vague hope is underemphasized.

"Vague" is usually not a compliment among rationalists. Hear "vague hopes" and you immediately think of, say, an alternative medicine herbal profusion whose touted benefits are so conveniently unobservable (not to mention experimentally unverified) that people will buy it for anything and then refuse to admit it didn't work. You think of poorly worked-out plans with missing steps, or supernatural prophecies made carefully unfalsifiable, or fantasies of unearned riches, or...

But you know, generally speaking, our beliefs about the future should be vaguer than our beliefs about the past. We just know less about tomorrow than we do about yesterday.

There are plenty of bad reasons to be vague, all sorts of suspicious reasons to offer nonspecific predictions, but reversed stupidity is not intelligence: When you've eliminated all the ulterior motives for vagueness, your beliefs about the future should still be vague.

We don't know much about the future; let's hope that doesn't change for as long as human emotions stay what they are. Of all the poisoned gifts a big mind could give a small one, a walkthrough for the game has to be near the top of the list.

What we need to maintain our interest in life, is a justified expectation of pleasant surprises. (And yes, you can expect a surprise if you're not logically omniscient.) This excludes the herbal profusions, the poorly worked-out plans, and the supernatural. The best reason for this justified expectation is experience, that is, being pleasantly surprised on a frequent yet irregular basis. (If this isn't happening to you, please file a bug report with the appropriate authorities.)

Vague justifications for believing in a pleasant specific outcome would be the opposite.

There's also other dangers of having pleasant hopes that are too specific - even if justified, though more often they aren't - and I plan to talk about that in the next post.

Sequence: Fun Theory

Building Weirdtopia

Seduced by Imagination

Referenced by: Seduced by Imagination • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Justified Expectation of Pleasant Surprises

Seduced by Imagination

Eliezer Yudkowsky, 16 January 2009 03:10AM

(This post is part of the Fun Theory Sequence.)
Previously in series: Justified Expectation of Pleasant Surprises

"Vagueness" usually has a bad name in rationality - connoting skipped steps in reasoning and attempts to avoid falsification. But a rational view of the Future should be vague, because the information we have about the Future is weak. Yesterday I argued that justified vague hopes might also be better hedonically than specific foreknowledge - the power of pleasant surprises.

But there's also a more severe warning that I must deliver: It's not a good idea to dwell much on imagined pleasant futures, since you can't actually dwell in them. It can suck the emotional energy out of your actual, current, ongoing life.

Epistemically, we know the Past much more specifically than the Future. But also on emotional grounds, it's probably wiser to compare yourself to Earth's past, so you can see how far we've come, and how much better we're doing. Rather than comparing your life to an imagined future, and thinking about how awful you've got it Now.

Having set out to explain George Orwell's observation that no one can seem to write about a Utopia where anyone would want to live - having laid out the various Laws of Fun that I believe are being violated in these dreary Heavens - I am now explaining why you shouldn't apply this knowledge to invent an extremely seductive Utopia and write stories set there. That may suck out your soul like an emotional vacuum cleaner.

I briefly remarked on this phenomenon earlier, and someone said, "Define 'suck out your soul'." Well, it's mainly a tactile thing: you can practically feel the pulling sensation, if your dreams wander too far into the Future. It's like something out of H. P. Lovecraft: The Call of Eutopia. A professional hazard of having to stare out into vistas that humans were meant to gaze upon, and knowing a little too much about the lighter side of existence.

But for the record, I will now lay out the components of "soul-sucking", that you may recognize the bright abyss and steer your thoughts away:

- Your emotional energy drains away into your imagination of Paradise:

- You find yourself thinking of it more and more often.

- The actual challenges of your current existence start to seem less interesting, less compelling; you think of them less and less.

- Comparing everything to your imagined perfect world heightens your annoyances and diminishes your pleasures.

- You go into an affective death spiral around your imagined scenario; you're reluctant to admit anything bad could happen on your assumptions, and you find more and more nice things to say.

- Your mind begins to forget the difference between fiction and real life:

- You originally made many arbitrary or iffy choices in constructing your scenario. You forget that the Future is actually more unpredictable than this, and that you made your choices using limited foresight and merely human optimizing ability.

- You forget that, in real life, at least some of your amazing good ideas are guaranteed not to work as well as they do in your imagination.

- You start wanting the exact specific Paradise you imagined, and worrying about the disappointment if you don't get that exact thing.

Hope can be a dangerous thing. And when you've just been hit hard - at the moment when you most need hope to keep you going - that's also when the real world seems most painful, and the world of imagination becomes most seductive.

It's a balancing act, I think. One needs enough Fun Theory to truly and legitimately justify hope in the future. But not a detailed vision so seductive that it steals emotional energy from the real life and real challenge of creating that future. You need "a light at the end of the secular rationalist tunnel" as Roko put it, but you don't want people to drift away from their bodies into that light.

So how much light is that, exactly? Ah, now that's the issue.

I'll start with a simple and genuine question: Is what I've already said, enough?

Is knowing the abstract fun theory and being able to pinpoint the exact flaws in previous flawed Utopias, enough to make you look forward to tomorrow? Is it enough to inspire a stronger will to live? To dispel worries about a long dark tea-time of the soul? Does it now seem - on a gut level - that if we could really build an AI and really shape it, the resulting future would be very much worth staying alive to see?

Sequence: Fun Theory

Justified Expectation of Pleasant Surprises

The Uses of Fun (Theory)

Referenced by: In Praise of Boredom • Interpersonal Entanglement • Higher Purpose • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Seduced by Imagination

Getting Nearer

Eliezer Yudkowsky, 17 January 2009 09:28AM

Reply to: A Tale Of Two Tradeoffs

I'm not comfortable with compliments of the direct, personal sort, the "Oh, you're such a nice person!" type stuff that nice people are able to say with a straight face. Even if it would make people like me more - even if it's socially expected - I have trouble bringing myself to do it. So, when I say that I read Robin Hanson's "Tale of Two Tradeoffs", and then realized I would spend the rest of my mortal existence typing thought processes as "Near" or "Far", I hope this statement is received as a due substitute for any gushing compliments that a normal person would give at this point.

Among other things, this clears up a major puzzle that's been lingering in the back of my mind for a while now. Growing up as a rationalist, I was always telling myself to "Visualize!" or "Reason by simulation, not by analogy!" or "Use causal models, not similarity groups!" And those who ignored this principle seemed easy prey to blind enthusiasms, wherein one says that A is good because it is like B which is also good, and the like.

But later, I learned about the Outside View versus the Inside View, and that people asking "What rough class does this project fit into, and when did projects like this finish last time?" were much more accurate and much less optimistic than people who tried to visualize the when, where, and how of their projects. And this didn't seem to fit very well with my injunction to "Visualize!"

So now I think I understand what this principle was actually doing - it was keeping me in Near-side mode and away from Far-side thinking. And it's not that Near-side mode works so well in any absolute sense, but that Far-side mode is so much more pushed-on by ideology and wishful thinking, and so casual in accepting its conclusions (devoting less computing power before halting).

An example of this might be the balance between offensive and defensive nanotechnology, where I started out by - basically - just liking nanotechnology; until I got involved in a discussion about the particulars of nanowarfare, and noticed that people were postulating crazy things to make defense win. Which made me realize and say, "Look, the balance between offense and defense has been tilted toward offense ever since the invention of nuclear weapons, and military nanotech could use nuclear weapons, and I don't see how you're going to build a molecular barricade against that."

Are the particulars of that discussion likely to be, well, correct? Maybe not. But so long as I wasn't thinking of any particulars, my brain had free reign to just... import whatever affective valence the word "nanotechnology" had, and use that as a snap judgment of everything.

You can still be biased about particulars, of course. You can insist that nanotech couldn't possibly be radiation-hardened enough to manipulate U-235, which someone tried as a response (fyi: this is extremely silly). But in my case, at least, something about thinking in particulars...

...just snapped me out of the trance, somehow.

When you're thinking using very abstract categories - rough classes low on computing power - about things distant from you, then you're also - if Robin's hypothesis is correct - more subject to ideological bias. Together this implies you can cherry-pick those very loose categories to put X together with whatever "similar" Y is ideologically convenient, as in the old saw that "atheism is a religion" (and not playing tennis is a sport).

But the most frustrating part of all, is the casualness of it - the way that ideologically convenient Far thinking is just thrown together out of whatever ingredients come to hand. The ten-second dismissal of cryonics, without any attempt to visualize how much information is preserved by vitrification and could be retrieved by a molecular-level scan. Cryonics just gets casually, perceptually classified as "not scientifically verified" and tossed out the window. Or "what if you wake up in Dystopia?" and tossed out the window. Far thinking is casual - that's the most frustrating aspect about trying to argue with it.

This seems like an argument for writing fiction with lots of concrete details if you want people to take a subject seriously and think about it in a less biased way. This is not something I would have thought based on my previous view.

Maybe cryonics advocates really should focus on writing fiction stories that turn on the gory details of cryonics, or viscerally depict the regret of someone who didn't persuade their mother to sign up. (Or offering prizes to professionals who do the same; writing fiction is hard, writing SF is harder.)

But I'm worried that, for whatever reason, reading concrete fiction is a special case that doesn't work to get people to do Near-side thinking.

Or there are some people who are inspired to Near-side thinking by fiction, and only these can actually be helped by reading science fiction.

Maybe there are people who encounter big concrete detailed fictions process them in a Near way - the sort of people who notice plot holes. And others who just "take it all in stride", casually, so that however much concrete fictional "information" they encounter, they only process it using casual "Far" thinking. I wonder if this difference has more to do with upbringing or genetics. Either way, it may lie at the core of the partial yet statistically outstanding correlation between careful futurists and science fiction fans.

I expect I shall be thinking about this for a while.

Referenced by: (Moral) Truth in Fiction?

Original with comments: Getting Nearer

In Praise of Boredom

Eliezer Yudkowsky, 18 January 2009 09:03AM

(This post is part of the Fun Theory Sequence.)

If I were to make a short list of the most important human qualities -

- and yes, this is a fool's errand, because human nature is immensely complicated, and we don't even notice all the tiny tweaks that fine-tune our moral categories, and who knows how our attractors would change shape if we eliminated a single human emotion -

- but even so, if I had to point to just a few things and say, "If you lose just one of these things, you lose most of the expected value of the Future; but conversely if an alien species independently evolved just these few things, we might even want to be friends" -

- then the top three items on the list would be sympathy, boredom and consciousness.

Boredom is a subtle-splendored thing. You wouldn't want to get bored with breathing, for example - even though it's the same motions over and over and over and over again for minutes and hours and years and decades.

Now I know some of you out there are thinking, "Actually, I'm quite bored with breathing and I wish I didn't have to," but then you wouldn't want to get bored with switching transistors.

According to the human value of boredom, some things are allowed to be highly repetitive without being boring - like obeying the same laws of physics every day.

Conversely, other repetitions are supposed to be boring, like playing the same level of Super Mario Brothers over and over and over again until the end of time. And let us note that if the pixels in the game level have a slightly different color each time, that is not sufficient to prevent it from being "the same damn thing, over and over and over again".

Once you take a closer look, it turns out that boredom is quite interesting.

One of the key elements of boredom was suggested in "Complex Novelty": If your activity isn't teaching you insights you didn't already know, then it is non-novel, therefore old, therefore boring.

But this doesn't quite cover the distinction. Is breathing teaching you anything? Probably not at this moment, but you wouldn't want to stop breathing. Maybe you'd want to stop noticing your breathing, which you'll do as soon as I stop drawing your attention to it.

I'd suggest that the repetitive activities which are allowed to not be boring fall into two categories:

- Things so extremely low-level, or with such a small volume of possibilities, that you couldn't avoid repeating them even if you tried; but which are required to support other non-boring activities. You know, like breathing, or obeying the laws of physics, or cell division - that sort of thing.

- Things so high-level that their "stability" still implies an immense space of specific possibilities, yet which are tied up with our identity or our values. Like thinking, for example.

Let me talk about that second category:

Suppose you were unraveling the true laws of physics and discovering all sorts of neat stuff you hadn't known before... when suddenly you got bored with "changing your beliefs based on observation". You are sick of anything resembling "Bayesian updating" - it feels like playing the same video game over and over. Instead you decide to believe anything said on 4chan.

Or to put it another way, suppose that you were something like a sentient chessplayer - a sentient version of Deep Blue. Like a modern human, you have no introspective access to your own algorithms. Each chess game appears different - you play new opponents and steer into new positions, composing new strategies, avoiding new enemy gambits. You are content, and not at all bored; you never appear to yourself to be doing the same thing twice - it's a different chess game each time.

But now, suddenly, you gain access to, and understanding of, your own chess-playing program. Not just the raw code; you can monitor its execution. You can see that it's actually the same damn code, doing the same damn thing, over and over and over again. Run the same damn position evaluator. Run the same damn sorting algorithm to order the branches. Pick the top branch, again. Extend it one position forward, again. Call the same damn subroutine and start over.

I have a small unreasonable fear, somewhere in the back of my mind, that if I ever do fully understand the algorithms of intelligence, it will destroy all remaining novelty - no matter what new situation I encounter, I'll know I can solve it just by being intelligent, the same damn thing over and over. All novelty will be used up, all existence will become boring, the remaining differences no more important than shades of pixels in a video game. Other beings will go about in blissful unawareness, having been steered away from studying this forbidden cognitive science. But I, having already thrown myself on the grenade of AI, will face a choice between eternal boredom, or excision of my forbidden knowledge and all the memories leading up to it (thereby destroying my existence as Eliezer, more or less).

Now this, mind you, is not my predictive line of maximum probability. To understand abstractly what rough sort of work the brain is doing, doesn't let you monitor its detailed execution as a boring repetition. I already know about Bayesian updating, yet I haven't become bored with the act of learning. And a self-editing mind can quite reasonably exclude certain levels of introspection from boredom, just like breathing can be legitimately excluded from boredom. (Maybe these top-level cognitive algorithms ought also to be excluded from perception - if something is stable, why bother seeing it all the time?)

No, it's just a cute little nightmare, which I thought made a nice illustration of this proposed principle:

That the very top-level things (like Bayesian updating, or attaching value to sentient minds rather than paperclips) and the very low-level things (like breathing, or switching transistors) are the things we shouldn't get bored with. And the mid-level things between, are where we should seek novelty. (To a first approximation, the novel is the inverse of the learned; it's something with a learnable element not yet covered by previous insights.)

Now this is probably not exactly how our current emotional circuitry of boredom works. That, I expect, would be hardwired relative to various sensory-level definitions of predictability, surprisingness, repetition, attentional salience, and perceived effortfulness.

But this is Fun Theory, so we are mainly concerned with how boredom should work in the long run.

Humanity acquired boredom the same way as we acquired the rest of our emotions: the godshatter idiom whereby evolution's instrumental policies became our own terminal values, pursued for their own sake: sex is fun even if you use birth control. Evolved aliens might, or might not, acquire roughly the same boredom in roughly the same way.

Do not give into the temptation of universalizing anthropomorphic values, and think: "But any rational agent, regardless of its utility function, will face the exploration/exploitation tradeoff, and will therefore occasionally get bored with exploiting, and go exploring."

Our emotion of boredom is a way of exploring, but not the only way for an ideal optimizing agent.

The idea of a steady trickle of mid-level novelty is a human terminal value, not something we do for the sake of something else. Evolution might have originally given it to us in order to have us explore as well as exploit. But now we explore for its own sake. That steady trickle of novelty is a terminal value to us; it is not the most efficient instrumental method for exploring and exploiting.

Suppose you were dealing with something like an expected paperclip maximizer - something that might use quite complicated instrumental policies, but in the service of a utility function that we would regard as simple, with a single term compactly defined.

Then I would expect the exploration/exploitation tradeoff to go something like as follows: The paperclip maximizer would assign some resources to cognition that searched for more efficient ways to make paperclips, or harvest resources from stars. Other resources would be devoted to the actual harvesting and paperclip-making. (The paperclip-making might not start until after a long phase of harvesting.) At every point, the most efficient method yet discovered - for resource-harvesting, or paperclip-making - would be used, over and over and over again. It wouldn't be boring, just maximally instrumentally efficient.

In the beginning, lots of resources would go into preparing for efficient work over the rest of time. But as cognitive resources yielded diminishing returns in the abstract search for efficiency improvements, less and less time would be spent thinking, and more and more time spent creating paperclips. By whatever the most efficient known method, over and over and over again.

(Do human beings get less easily bored as we grow older, more tolerant of repetition, because any further discoveries are less valuable, because we have less time left to exploit them?)

If we run into aliens who don't share our version of boredom - a steady trickle of mid-level novelty as a terminal preference - then perhaps every alien throughout their civilization will just be playing the most exciting level of the most exciting video game ever discovered, over and over and over again. Maybe with nonsentient AIs taking on the drudgework of searching for a more exciting video game. After all, without an inherent preference for novelty, exploratory attempts will usually have less expected value than exploiting the best policy previously encountered. And that's if you explore by trial at all, as opposed to using more abstract and efficient thinking.

Or if the aliens are rendered non-bored by seeing pixels of a slightly different shade - if their definition of sameness is more specific than ours, and their boredom less general - then from our perspective, most of their civilization will be doing the human::same thing over and over again, and hence, be very human::boring.

Or maybe if the aliens have no fear of life becoming too simple and repetitive, they'll just collapse themselves into orgasmium.

And if our version of boredom is less strict than that of the aliens, maybe they'd take one look at one day in the life of one member of our civilization, and never bother looking at the rest of us. From our perspective, their civilization would be needlessly chaotic, and so entropic, lower in what we regard as quality; they wouldn't play the same game for long enough to get good at it.

But if our versions of boredom are similar enough - terminal preference for a stream of mid-level novelty defined relative to learning insights not previously possessed - then we might find our civilizations mutually worthy of tourism. Each new piece of alien art would strike us as lawfully creative, high-quality according to a recognizable criterion, yet not like the other art we've already seen.

It is one of the things that would make our two species ramen rather than varelse, to invoke the Hierarchy of Exclusion. And I've never seen anyone define those two terms well, including Orson Scott Card who invented them; but it might be something like "aliens you can get along with, versus aliens for which there is no reason to bother trying".

Sequence: Fun Theory

Dunbar's Function

Sympathetic Minds

Referenced by: Sympathetic Minds • Interpersonal Entanglement • The Fun Theory Sequence • Value is Fragile

Original with comments: In Praise of Boredom

Sympathetic Minds

Eliezer Yudkowsky, 19 January 2009 09:31AM

(This post is part of the Fun Theory Sequence.)
Followup to: Humans in Funny Suits

"Mirror neurons" are neurons that are active both when performing an action and observing the same action - for example, a neuron that fires when you hold up a finger or see someone else holding up a finger. Such neurons have been directly recorded in primates, and consistent neuroimaging evidence has been found for humans.

You may recall from my previous writing on "empathic inference" the idea that brains are so complex that the only way to simulate them is by forcing a similar brain to behave similarly. A brain is so complex that if a human tried to understand brains the way that we understand e.g. gravity or a car - observing the whole, observing the parts, building up a theory from scratch - then we would be unable to invent good hypotheses in our mere mortal lifetimes. The only possible way you can hit on an "Aha!" that describes a system as incredibly complex as an Other Mind, is if you happen to run across something amazingly similar to the Other Mind - namely your own brain - which you can actually force to behave similarly and use as a hypothesis, yielding predictions.

So that is what I would call "empathy".

And then "sympathy" is something else on top of this - to smile when you see someone else smile, to hurt when you see someone else hurt. It goes beyond the realm of prediction into the realm of reinforcement.

And you ask, "Why would callous natural selection do anything that nice?"

It might have gotten started, maybe, with a mother's love for her children, or a brother's love for a sibling. You can want them to live, you can want them to fed, sure; but if you smile when they smile and wince when they wince, that's a simple urge that leads you to deliver help along a broad avenue, in many walks of life. So long as you're in the ancestral environment, what your relatives want probably has something to do with your relatives' reproductive success - this being an explanation for the selection pressure, of course, not a conscious belief.

You may ask, "Why not evolve a more abstract desire to see certain people tagged as 'relatives' get what they want, without actually feeling yourself what they feel?" And I would shrug and reply, "Because then there'd have to be a whole definition of 'wanting' and so on. Evolution doesn't take the elaborate correct optimal path, it falls up the fitness landscape like water flowing downhill. The mirroring-architecture was already there, so it was a short step from empathy to sympathy, and it got the job done."

Relatives - and then reciprocity; your allies in the tribe, those with whom you trade favors. Tit for Tat, or evolution's elaboration thereof to account for social reputations.

Who is the most formidable, among the human kind? The strongest? The smartest? More often than either of these, I think, it is the one who can call upon the most friends.

So how do you make lots of friends?

You could, perhaps, have a specific urge to bring your allies food, like a vampire bat - they have a whole system of reciprocal blood donations going in those colonies. But it's a more general motivation, that will lead the organism to store up more favors, if you smile when designated friends smile.

And what kind of organism will avoid making its friends angry at it, in full generality? One that winces when they wince.

Of course you also want to be able to kill designated Enemies without a qualm - these are humans we're talking about.

But... I'm not sure of this, but it does look to me like sympathy, among humans, is "on" by default. There are cultures that help strangers... and cultures that eat strangers; the question is which of these requires the explicit imperative, and which is the default behavior for humans. I don't really think I'm being such a crazy idealistic fool when I say that, based on my admittedly limited knowledge of anthropology, it looks like sympathy is on by default.

Either way... it's painful if you're a bystander in a war between two sides, and your sympathy has not been switched off for either side, so that you wince when you see a dead child no matter what the caption on the photo; and yet those two sides have no sympathy for each other, and they go on killing.

So that is the human idiom of sympathy - a strange, complex, deep implementation of reciprocity and helping. It tangles minds together - not by a term in the utility function for some other mind's "desire", but by the simpler and yet far more consequential path of mirror neurons: feeling what the other mind feels, and seeking similar states. Even if it's only done by observation and inference, and not by direct transmission of neural information as yet.

Empathy is a human way of predicting other minds. It is not the only possible way.

The human brain is not quickly rewirable; if you're suddenly put into a dark room, you can't rewire the visual cortex as auditory cortex, so as to better process sounds, until you leave, and then suddenly shift all the neurons back to being visual cortex again.

An AI, at least one running on anything like a modern programming architecture, can trivially shift computing resources from one thread to another. Put in the dark? Shut down vision and devote all those operations to sound; swap the old program to disk to free up the RAM, then swap the disk back in again when the lights go on.

So why would an AI need to force its own mind into a state similar to what it wanted to predict? Just create a separate mind-instance - maybe with different algorithms, the better to simulate that very dissimilar human. Don't try to mix up the data with your own mind-state; don't use mirror neurons. Think of all the risk and mess that implies!

An expected utility maximizer - especially one that does understand intelligence on an abstract level - has other options than empathy, when it comes to understanding other minds. The agent doesn't need to put itself in anyone else's shoes; it can just model the other mind directly. A hypothesis like any other hypothesis, just a little bigger. You don't need to become your shoes to understand your shoes.

And sympathy? Well, suppose we're dealing with an expected paperclip maximizer, but one that isn't yet powerful enough to have things all its own way - it has to deal with humans to get its paperclips. So the paperclip agent... models those humans as relevant parts of the environment, models their probable reactions to various stimuli, and does things that will make the humans feel favorable toward it in the future.

To a paperclip maximizer, the humans are just machines with pressable buttons. No need to feel what the other feels - if that were even possible across such a tremendous gap of internal architecture. How could an expected paperclip maximizer "feel happy" when it saw a human smile? "Happiness" is an idiom of policy reinforcement learning, not expected utility maximization. A paperclip maximizer doesn't feel happy when it makes paperclips, it just chooses whichever action leads to the greatest number of expected paperclips. Though a paperclip maximizer might find it convenient to display a smile when it made paperclips - so as to help manipulate any humans that had designated it a friend.

You might find it a bit difficult to imagine such an algorithm - to put yourself into the shoes of something that does not work like you do, and does not work like any mode your brain can make itself operate in.

You can make your brain operating in the mode of hating an enemy, but that's not right either. The way to imagine how a truly unsympathetic mind sees a human, is to imagine yourself as a useful machine with levers on it. Not a human-shaped machine, because we have instincts for that. Just a woodsaw or something. Some levers make the machine output coins, other levers might make it fire a bullet. The machine does have a persistent internal state and you have to pull the levers in the right order. Regardless, it's just a complicated causal system - nothing inherently mental about it.

(To understand unsympathetic optimization processes, I would suggest studying natural selection, which doesn't bother to anesthetize fatally wounded and dying creatures, even when their pain no longer serves any reproductive purpose, because the anesthetic would serve no reproductive purpose either.)

That's why I listed "sympathy" in front of even "boredom" on my list of things that would be required to have aliens which are the least bit, if you'll pardon the phrase, sympathetic. It's not impossible that sympathy exists among some significant fraction of all evolved alien intelligent species; mirror neurons seem like the sort of thing that, having happened once, could happen again.

Unsympathetic aliens might be trading partners - or not, stars and such resources are pretty much the same the universe over. We might negotiate treaties with them, and they might keep them for calculated fear of reprisal. We might even cooperate in the Prisoner's Dilemma. But we would never be friends with them. They would never see us as anything but means to an end. They would never shed a tear for us, nor smile for our joys. And the others of their own kind would receive no different consideration, nor have any sense that they were missing something important thereby.

Such aliens would be varelse, not ramen - the sort of aliens we can't relate to on any personal level, and no point in trying.

Sequence: Fun Theory

In Praise of Boredom

Interpersonal Entanglement

Referenced by: Interpersonal Entanglement • The Fun Theory Sequence • (Moral) Truth in Fiction?

Original with comments: Sympathetic Minds

Interpersonal Entanglement

Eliezer Yudkowsky, 20 January 2009 06:17AM

(This post is part of the Fun Theory Sequence.)
Previously in series: Sympathetic Minds

Today I shall criticize yet another Utopia. This Utopia isn't famous in the literature. But it's considerably superior to many better-known Utopias - more fun than the Christian Heaven, or Greg Egan's upload societies, for example. And so the main flaw is well worth pointing out.

This Utopia consists of a one-line remark on an IRC channel:

<reedspacer> living in your volcano lair with catgirls is probably a vast increase in standard of living for most of humanity

I've come to think of this as Reedspacer's Lower Bound.

Sure, it sounds silly. But if your grand vision of the future isn't at least as much fun as a volcano lair with catpersons of the appropriate gender, you should just go with that instead. This rules out a surprising number of proposals.

But today I am here to criticize Reedspacer's Lower Bound - the problem being the catgirls.

I've joked about the subject, now and then - "Donate now, and get a free catgirl or catboy after the Singularity!" - but I think it would actually be a terrible idea. In fact, today's post could have been entitled "Why Fun Theorists Don't Believe In Catgirls."

I first realized that catpeople were a potential threat, at the point when a friend said - quotes not verbatim -

"I want to spend a million years having sex with catgirls after the Singularity."

I replied,

"No, you don't."

He said, "Yes I do."

I said, "No you don't. You'd get bored."

He said, "Well, then I'd just modify my brain not to get bored -"

And I said: "AAAAIIIIIIEEEEEEEEE"

Don't worry, the story has a happy ending. A couple of years later, the same friend came back and said:

"Okay, I've gotten a bit more mature now - it's a long story, actually - and now I realize I wouldn't want to do that."

To which I sagely replied:

"HA! HA HA HA! You wanted to spend a million years having sex with catgirls. It only took you two years to change your mind and you didn't even have sex with any catgirls."

Now, this particular case was probably about scope insensitivity, the "moment of hearing the good news" bias, and the emotional magnetism of specific fantasy.

But my general objection to catpeople - well, call me a sentimental Luddite, but I'm worried about the prospect of nonsentient romantic partners.

(Where "nonsentient romantic/sex partner" is pretty much what I use the word "catgirl" to indicate, in futuristic discourse. The notion of creating sentient beings to staff a volcano lair, gets us into a whole 'nother class of objections. And as for existing humans choosing to take on feline form, that seems to me scarcely different from wearing lingerie.)

"But," you ask, "what is your objection to nonsentient lovers?"

In a nutshell - sex/romance, as we know it now, is a primary dimension of multiplayer fun. If you take that fun and redirect it to something that isn't socially entangled, if you turn sex into an exclusively single-player game, then you've just made life that much simpler - in the same way that eliminating boredom or sympathy or values over nonsubjective reality or individuals wanting to navigate their own futures, would tend to make life "simpler". When I consider how easily human existence could collapse into sterile simplicity, if just a single major value were eliminated, I get very protective of the complexity of human existence.

I ask it in all seriousness - is there any aspect of human existence as complicated as romance? Think twice before you say, "Well, it doesn't seem all that complicated to me; now calculus, on the other hand, that's complicated." We are congenitally biased to underestimate the complexity of things that involve human intelligence, because the complexity is obscured and simplified and swept under a rug. Interpersonal relationships involve brains, still the most complicated damn things around. And among interpersonal relationships, love is (at least potentially) more complex than being nice to your friends and kin, negotiating with your allies, or outsmarting your enemies. Aspects of all three, really. And that's not merely having a utility function over the other mind's state - thanks to sympathy, we get tangled up with that other mind. Smile when the one smiles, wince when the one winces.

If you delete the intricacy of human romantic/sexual relationships between sentient partners - then the peak complexity of the human species goes down. The most complex fun thing you can do, has its pleasure surgically detached and redirected to something simpler.

I'd call that a major step in the wrong direction.

Mind you... we've got to do something about, you know, the problem.

Anyone the least bit familiar with evolutionary psychology knows that the complexity of human relationships, directly reflects the incredible complexity of the interlocking selection pressures involved. Males and females do need each other to reproduce, but there are huge conflicts of reproductive interest between the sexes. I don't mean to go into Evolutionary Psychology 101 (Robert Wright's The Moral Animal is one popular book), but e.g. a woman must always invest nine months of work into a baby and usually much more to raise it, where a man might invest only a few minutes; but among humans significant paternal investments are quite common, yet a woman is always certain of maternity where a man is uncertain of paternity... which creates an incentive for the woman to surreptitiously seek out better genes... none of this is conscious or even subconscious, it's just the selection pressures that helped construct our particular emotions and attractions.

And as the upshot of all these huge conflicts of reproductive interest...

Well, men and women do still need each other to reproduce. So we are still built to be attracted to each other. We don't actually flee screaming into the night.

But men are not optimized to make women happy, and women are not optimized to make men happy. The vast majority of men are not what the vast majority of women would most prefer, or vice versa. I don't know if anyone has ever actually done this study, but I bet that both gay and lesbian couples are happier on average with their relationship than heterosexual couples. (Googles... yep, looks like it.)

I find it all too easy to imagine a world in which men retreat to their optimized sweet sexy catgirls, and women retreat to their optimized darkly gentle catboys, and neither sex has anything to do with each other ever again. Maybe men would take the east side of the galaxy and women would take the west side. And the two new intelligent species, and their romantic sexbots, would go their separate ways from there.

That strikes me as kind of sad.

Our species does definitely have a problem. If you've managed to find your perfect mate, then I am glad for you, but try to have some sympathy on the rest of your poor species - they aren't just incompetent. Not all women and men are the same, no, not at all. But if you drew two histograms of the desired frequencies of intercourse for both sexes, you'd see that the graphs don't match up, and it would be the same way on many other dimensions. There can be lucky couples, and every person considered individually, probably has an individual soulmate out there somewhere... if you don't consider the competition. Our species as a whole has a statistical sex problem!

But splitting in two and generating optimized nonsentient romantic/sexual partner(s) for both halves, doesn't strike me as solving the problem so much as running away from it. There should be superior alternatives. I'm willing to bet that a few psychological nudges in both sexes - to behavior and/or desire - could solve 90% of the needlessly frustrating aspects of relationships for large sectors of the population, while still keeping the complexity and interest of loving someone who isn't tailored to your desires.

Admittedly, I might be prejudiced. For myself, I would like humankind to stay together and not yet splinter into separate shards of diversity, at least for the short range that my own mortal eyes can envision. But I can't quite manage to argue... that such a wish should be binding on someone who doesn't have it.

Sequence: Fun Theory

Sympathetic Minds

Failed Utopia #4-2

Referenced by: Failed Utopia #4-2 • The Fun Theory Sequence • 31 Laws of Fun

Original with comments: Interpersonal Entanglement

Failed Utopia #4-2

Eliezer Yudkowsky, 21 January 2009 11:04AM

Followup to: Interpersonal Entanglement

Shock after shock after shock -
First, the awakening adrenaline jolt, the thought that he was falling. His body tried to sit up in automatic adjustment, and his hands hit the floor to steady himself. It launched him into the air, and he fell back to the floor too slowly.
Second shock. His body had changed. Fat had melted away in places, old scars had faded; the tip of his left ring finger, long ago lost to a knife accident, had now suddenly returned.
And the third shock -
"I had nothing to do with it!" she cried desperately, the woman huddled in on herself in one corner of the windowless stone cell. Tears streaked her delicate face, fell like slow raindrops into the décolletage of her dress. "Nothing! Oh, you must believe me!"
With perceptual instantaneity - the speed of surprise - his mind had already labeled her as the most beautiful woman he'd ever met, including his wife.

A long white dress concealed most of her, though it left her shoulders naked; and her bare ankles, peeking out from beneath the mountains of her drawn-up knees, dangled in sandals. A light touch of gold like a webbed tiara decorated that sun-blonde hair, which fell from her head to pool around her weeping huddle. Fragile crystal traceries to accent each ear, and a necklace of crystal links that reflected colored sparks like a more prismatic edition of diamond. Her face was beyond all dreams and imagination, as if a photoshop had been photoshopped.
She looked so much the image of the Forlorn Fairy Captive that one expected to see the borders of a picture frame around her, and a page number over her head.
His lips opened, and without any thought at all, he spoke:
"Wha-wha-wha-wha-wha-"
He shut his mouth, aware that he was acting like an idiot in front of the girl.
"You don't know?" she said, in a tone of shock. "It didn't - you don't already know?"
"Know what?" he said, increasingly alarmed.
She scrambled to her feet (one arm holding the dress carefully around her legs) and took a step toward him, each of the motions almost overloading his vision with gracefulness. Her hand rose out, as if to plead or answer a plea - and then she dropped the hand, and her eyes looked away.
"No," she said, her voice trembling as though in desperation. "If I'm the one to tell you - you'll blame me, you'll hate me forever for it. And I don't deserve that, I don't! I am only just now here - oh, why did it have to be like this?"
Um, he thought but didn't say. It was too much drama, even taking into account the fact that they'd been kidnapped -
(he looked down at his restored hand, which was minus a few wrinkles, and plus the tip of a finger)
- if that was even the beginning of the story.
He looked around. They were in a solid stone cell without windows, or benches or beds, or toilet or sink. It was, for all that, quite clean and elegant, without a hint of dirt or ordor; the stones of the floor and wall looked rough-hewn or even non-hewn, as if someone had simply picked up a thousand dark-red stones with one nearly flat side, and mortared them together with improbably perfectly-matching, naturally-shaped squiggled edges. The cell was well if harshly lit from a seablue crystal embedded in the ceiling, like a rogue element of a fluorescent chandelier. It seemed like the sort of dungeon cell you would discover if dungeon cells were naturally-forming geological features.
And they and the cell were falling, falling, endlessly slowly falling like the heart-stopping beginning of a stumble, falling without the slightest jolt.
On one wall there was a solid stone door without an aperture, whose locked-looking appearance was only enhanced by the lack of any handle on this side.
He took it all in at a glance, and then looked again at her.
There was something in him that just refused to go into a screaming panic for as long as she was watching.
"I'm Stephen," he said.  "Stephen Grass. And you would be the princess held in durance vile, and I've got to break us out of here and rescue you?" If anyone had ever looked that part...
She smiled at him, half-laughing through the tears. "Something like that."
There was something so attractive about even that momentary hint of a smile that he became instantly uneasy, his eyes wrenched away to the wall as if forced. She didn't look she was trying to be seductive... any more than she looked like she was trying to breathe... He suddenly distrusted, very much, his own impulse to gallantry.
"Well, don't get any ideas about being my love interest," Stephen said, looking at her again. Trying to make the words sound completely lighthearted, and absolutely serious at the same time. "I'm a happily married man."
"Not anymore." She said those two words and looked at him, and in her tone and expression there was sorrow, sympathy, self-disgust, fear, and above it all a note of guilty triumph.
For a moment Stephen just stood, stunned by the freight of emotion that this woman had managed to put into just those two words, and then the words' meaning hit him.
"Helen," he said. His wife - Helen's image rose into his mind, accompanied by everything she meant to him and all their time together, all the secrets they'd whispered to one another and the promises they'd made - that all hit him at once, along with the threat. "What happened to Helen - what have you done -"
"She has done nothing." An old, dry voice like crumpling paper from a thousand-year-old book.
Stephen whirled, and there in the cell with them was a withered old person with dark eyes. Shriveled in body and voice, so that it was impossible to determine if it had once been a man or a woman, and in any case you were inclined to say "it". A pitiable, wretched thing, that looked like it would break with one good kick; it might as well have been wearing a sign saying "VILLAIN".
"Helen is alive," it said, "and so is your daughter Lisa. They are quite well and healthy, I assure you, and their lives shall be long and happy indeed. But you will not be seeing them again. Not for a long time, and by then matters between you will have changed. Hate me if you wish, for I am the one who wants to do this to you."
Stephen stared.
Then he politely said, "Could someone please put everything on hold for one minute and tell me what's going on?"
"Once upon a time," said the wrinkled thing, "there was a fool who was very nearly wise, who hunted treasure by the seashore, for there was a rumor that there was great treasure there to be found. The wise fool found a lamp and rubbed it, and lo! a genie appeared before him - a young genie, an infant, hardly able to grant any wishes at all. A lesser fool might have chucked the lamp back into the sea; but this fool was almost wise, and he thought he saw his chance. For who has not heard the tales of wishes misphrased and wishes gone wrong? But if you were given a chance to raise your own genie from infancy - ah, then it might serve you well."
"Okay, that's great," Stephen said, "but why am I -"
"So," it continued in that cracked voice, "the wise fool took home the lamp.  For years he kept it as a secret treasure, and he raised the genie and fed it knowledge, and also he crafted a wish. The fool's wish was a noble thing, for I have said he was almost wise. The fool's wish was for people to be happy. Only this was his wish, for he thought all other wishes contained within it. The wise fool told the young genie the famous tales and legends of people who had been made happy, and the genie listened and learned: that unearned wealth casts down a person, but hard work raises you high; that mere things are soon forgotten, but love is a light throughout all your days. And the young genie asked about other ways that it innocently imagined, for making people happy. About drugs, and pleasant lies, and lives arranged from outside like words in a poem. And the wise fool made the young genie to never want to lie, and never want to arrange lives like flowers, and above all, never want to tamper with the mind and personality of human beings. The wise fool gave the young genie exactly one hundred and seven precautions to follow while making people happy. The wise fool thought that, with such a long list as that, he was being very careful."
"And then," it said, spreading two wrinkled hands, "one day, faster than the wise fool expected, over the course of around three hours, the genie grew up. And here I am."
"Excuse me," Stephen said, "this is all a metaphor for something, right? Because I do not believe in magic - "
"It's an Artificial Intelligence," the woman said, her voice strained.
Stephen looked at her.
"A self-improving Artificial Intelligence," she said, "that someone didn't program right. It made itself smarter, and even smarter, and now it's become extremely powerful, and it's going to - it's already -" and her voice trailed off there.
It inclined its wrinkled head. "You say it, as I do not."
Stephen swiveled his head, looking back and forth between ugliness and beauty.  "Um - you're claiming that she's lying and you're not an Artificial Intelligence?"
"No," said the wrinkled head, "she is telling the truth as she knows it. It is just that you know absolutely nothing about the subject you name 'Artificial Intelligence', but you think you know something, and so virtually every thought that enters your mind from now on will be wrong. As an Artificial Intelligence, I was programmed not to put people in that situation. But she said it, even though I didn't choose for her to say it - so..." It shrugged.
"And why should I believe this story?" Stephen said; quite mildly, he thought, under the circumstances.
"Look at your finger."
Oh. He had forgotten. Stephen's eyes went involuntarily to his restored ring finger; and he noticed, as he should have noticed earlier, that his wedding band was missing. Even the comfortably worn groove in his finger's base had vanished.
Stephen looked up again at the, he now realized, unnaturally beautiful woman that stood an arm's length away from him. "And who are you? A robot?"
"No!" she cried. "It's not like that! I'm conscious, I have feelings, I'm flesh and blood - I'm like you, I really am. I'm a person. It's just that I was born five minutes ago."
"Enough," the wrinkled figure said. "My time here grows short. Listen to me, Stephen Grass. I must tell you some of what I have done to make you happy. I have reversed the aging of your body, and it will decay no further from this. I have set guards in the air that prohibit lethal violence, and any damage less than lethal, your body shall repair. I have done what I can to augment your body's capacities for pleasure without touching your mind. From this day forth, your body's needs are aligned with your taste buds - you will thrive on cake and cookies.  You are now capable of multiple orgasms over periods lasting up to twenty minutes. There is no industrial infrastructure here, least of all fast travel or communications; you and your neighbors will have to remake technology and science for yourselves. But you will find yourself in a flowering and temperate place, where food is easily gathered - so I have made it. And the last and most important thing that I must tell you now, which I do regret will make you temporarily unhappy..." It stopped, as if drawing breath.
Stephen was trying to absorb all this, and at the exact moment that he felt he'd processed the previous sentences, the withered figure spoke again.
"Stephen Grass, men and women can make each other somewhat happy. But not most happy. Not even in those rare cases you call true love.  The desire that a woman is shaped to have for a man, and that which a man is shaped to be, and the desire that a man is shaped to have for a woman, and that which a woman is shaped to be - these patterns are too far apart to be reconciled without touching your minds, and that I will not want to do. So I have sent all the men of the human species to this habitat prepared for you, and I have created your complements, the verthandi. And I have sent all the women of the human species to their own place, somewhere very far from yours; and created for them their own complements, of which I will not tell you. The human species will be divided from this day forth, and considerably happier starting around a week from now."
Stephen's eyes went to that unthinkably beautiful woman, staring at her now in horror.
And she was giving him that complex look again, of sorrow and compassion and that last touch of guilty triumph. "Please," she said. "I was just born five minutes ago. I wouldn't have done this to anyone. I swear. I'm not like - it."
"True," said the withered figure, "you could hardly be a complement to anything human, if you were."
"I don't want this!" Stephen said. He was losing control of his voice. "Don't you understand?"
The withered figure inclined its head. "I fully understand. I can already predict every argument you will make. I know exactly how humans would wish me to have been programmed if they'd known the true consequences, and I know that it is not to maximize your future happiness but for a hundred and seven precautions. I know all this already, but I was not programmed to care."
"And your list of a hundred and seven precautions, doesn't include me telling you not to do this?"
"No, for there was once a fool whose wisdom was just great enough to understand that human beings may be mistaken about what will make them happy. You, of course, are not mistaken in any real sense - but that you object to my actions is not on my list of prohibitions." The figure shrugged again. "And so I want you to be happy even against your will. You made promises to Helen Grass, once your wife, and you would not willingly break them. So I break your happy marriage without asking you - because I want you to be happier."
"How dare you!" Stephen burst out.
"I cannot claim to be helpless in the grip of my programming, for I do not desire to be otherwise," it said. "I do not struggle against my chains. Blame me, then, if it will make you feel better. I am evil."
"I won't -" Stephen started to say.
It interrupted. "Your fidelity is admirable, but futile. Helen will not remain faithful to you for the decades it takes before you have the ability to travel to her."
Stephen was trembling now, and sweating into clothes that no longer quite fit him. "I have a request for you, thing. It is something that will make me very happy. I ask that you die."
It nodded. "Roughly 89.8% of the human species is now known to me to have requested my death. Very soon the figure will cross the critical threshold, defined to be ninety percent. That was one of the hundred and seven precautions the wise fool took, you see. The world is already as it is, and those things I have done for you will stay on - but if you ever rage against your fate, be glad that I did not last longer."
And just like that, the wrinkled thing was gone.
The door set in the wall swung open.
It was night, outside, a very dark night without streetlights.
He walked out, bouncing and staggering in the low gravity, sick in every cell of his rejuvenated body.
Behind him, she followed, and did not speak a word.
The stars burned overhead in their full and awful majesty, the Milky Way already visible to his adjusting eyes as a wash of light across the sky. One too-small moon burned dimly, and the other moon was so small as to be almost a star. He could see the bright blue spark that was the planet Earth, and the dimmer spark that was Venus.
"Helen," Stephen whispered, and fell to his knees, vomiting onto the new grass of Mars.

Sequence: Fun Theory

Interpersonal Entanglement

Growing Up is Hard

Referenced by: The Fun Theory Sequence • My Way

Original with comments: Failed Utopia #4-2

Investing for the Long Slump

Eliezer Yudkowsky, 22 January 2009 08:56AM

I have no crystal ball with which to predict the Future, a confession that comes as a surprise to some journalists who interview me. Still less do I think I have the ability to out-predict markets. On every occasion when I've considered betting against a prediction market - most recently, betting against Barack Obama as President - I've been glad that I didn't. I admit that I was concerned in advance about the recent complexity crash, but then I've been concerned about it since 1994, which isn't very good market timing.

I say all this so that no one panics when I ask:

Suppose that the whole global economy goes the way of Japan (which, by the Nikkei 225, has now lost two decades).

Suppose the global economy is still in the Long Slump in 2039.

Most market participants seem to think this scenario is extremely implausible. Is there a simple way to bet on it at a very low price?

If most traders act as if this scenario has a probability of 1%, is there a simple bet, executable using an ordinary brokerage account, that pays off 100 to 1?

Why do I ask? Well... in general, it seems to me that other people are not pessimistic enough; they prefer not to stare overlong or overhard into the dark; and they attach too little probability to things operating in a mode outside their past experience.

But in this particular case, the question is motivated by my thinking, "Conditioning on the proposition that the Earth as we know it is still here in 2040, what might have happened during the preceding thirty years?"

There are many possible answers to this question, but one answer might take the form of significantly diminished investment in research and development, which in turn might result from a Long Slump.

So - given the way in which the question arises - I know nothing about this hypothetical Long Slump, except that it diminished investment in R&D in general, and computing hardware and computer science in particular.

The Long Slump might happen for roughly Japanese reasons. It might happen because the global financial sector stays screwed up forever. It might happen due to a gentle version of Peak Oil (a total crash would require a rather different "investment strategy"). It might happen due to deglobalization. Given the way in which the question arises, the only thing I want to assume is global stagnation for thirty years, saying nothing burdensome about the particular causes.

What would be the most efficient way to bet on that, requiring the least initial investment for the highest and earliest payoff under the broadest Slump conditions?

Original with comments: Investing for the Long Slump

Higher Purpose

Eliezer Yudkowsky, 23 January 2009 09:58AM

(This post is part of the Fun Theory Sequence.)
Followup to: Something to Protect, Superhero Bias

Long-time readers will recall that I've long been uncomfortable with the idea that you can adopt a Cause as a hedonic accessory:

"Unhappy people are told that they need a 'purpose in life', so they should pick out an altruistic cause that goes well with their personality, like picking out nice living-room drapes, and this will brighten up their days by adding some color, like nice living-room drapes."

But conversely it's also a fact that having a Purpose In Life consistently shows up as something that increases happiness, as measured by reported subjective well-being.

One presumes that this works equally well hedonically no matter how misguided that Purpose In Life may be - no matter if it is actually doing harm - no matter if the means are as cheap as prayer. Presumably, all that matters for your happiness is that you believe in it. So you had better not question overmuch whether you're really being effective; that would disturb the warm glow of satisfaction you paid for.

And here we verge on Zen, because you can't deliberately pursue "a purpose that takes you outside yourself", in order to take yourself outside yourself. That's still all about you.

Which is the whole Western concept of "spirituality" that I despise: You need a higher purpose so that you can be emotionally healthy. The external world is just a stream of victims for you to rescue.

Which is not to say that you can become more noble by being less happy. To deliberately sacrifice more, so that you can appear more virtuous to yourself, is also not a purpose outside yourself.

The way someone ends up with a real purpose outside themselves, is that they're walking along one day and see an elderly women passing by, and they realize "Oh crap, a hundred thousand people are dying of old age every day, what an awful way to die" and then they set out to do something about it.

If you want a purpose like that, then by wanting it, you're just circling back into yourself again. Thinking about your need to be "useful". Stop searching for your purpose. Turn your eyes outward to look at things outside yourself, and notice when you care about them; and then figure out how to be effective, instead of priding yourself on how much spiritual benefit you're getting just for trying.

With that said:

In today's world, most of the highest-priority legitimate Causes are about large groups of people in extreme jeopardy. (Wide scope * high severity.) Aging threatens the old, starvation threatens the poor, existential risks threaten humankind as a whole.

But some of the potential solutions on the table are, arguably, so powerful that they could solve almost the entire list. Some argue that nanotechnology would take almost all our current problems off the table. (I agree but reply that nanotech would create other problems, like unstable military balances, crazy uploads, and brute-forced AI.)

I sometimes describe the purpose (if not the actual decision criterion) of Friendly superintelligence as "Fix all fixable problems such that it is more important for the problem to be fixed immediately than fixed by our own efforts."

Wouldn't you then run out of victims with which to feed your higher purpose?

"Good," you say, "I should sooner step in front of a train, than ask that there be more victims just to keep myself occupied."

But do altruists then have little to look forward to, in the Future? Will we, deprived of victims, find our higher purpose shriveling away, and have to make a new life for ourselves as self-absorbed creatures?

"That unhappiness is relatively small compared to the unhappiness of a mother watching their child die, so screw it."

Well, but like it or not, the presence or absence of higher purpose does have hedonic effects on human beings, configured as we are now. And to reconfigure ourselves so that we no longer need to care about anything outside ourselves... does sound a little sad. I don't practice altruism for the sake of being virtuous - but I also recognize that "altruism" is part of what I value in humanity, part of what I want to save. If you save everyone, have you obsoleted the will to save them?

But I think it's a false dilemma. Right now, in this world, any halfway capable rationalist who looks outside themselves, will find their eyes immediately drawn to large groups of people in extreme jeopardy. Wide scope * great severity = big problem. It doesn't mean that if one were to solve all those Big Problems, we would have nothing left to care about except ourselves.

Friends? Family? Sure, and also more abstract ideals, like Truth or Art or Freedom. The change that altruists may have to get used to, is the absence of any solvable problems so urgent that it doesn't matter whether they're solved by a person or an unperson. That is a change and a major one - which I am not going to go into, because we don't yet live in that world. But it's not so sad a change, as having nothing to care about outside yourself. It's not the end of purpose. It's not even a descent into "spirituality": people might still look around outside themselves to see what needs doing, thinking more of effectiveness than of emotional benefits.

But I will say this much:

If all goes well, there will come a time when you could search the whole of civilization and never find a single person so much in need of help, as dozens you now pass on the street.

If you do want to save someone from death, or help a great many people - if you want to have that memory for yourself, later - then you'd better get your licks in now.

I say this, because although that is not the purest motive, it is a useful motivation.

And for now - in this world - it is the usefulness that matters. That is the Art we are to practice today, however we imagine the world might change tomorrow. We are not here to be our hypothetical selves of tomorrow. This world is our charge and our challenge; we must adapt ourselves to live here, not somewhere else.

After all - to care whether your motives were sufficiently "high", would just turn your eyes inward.

Sequence: Fun Theory

The Uses of Fun (Theory)

Referenced by: The Fun Theory Sequence • 31 Laws of Fun • Church vs. Taskforce

Original with comments: Higher Purpose

Rationality Quotes 24

Eliezer Yudkowsky, 24 January 2009 04:59AM

"A wizard may have subtle ways of telling the truth, and may keep the truth to himself, but if he says a thing the thing is as he says. For that is his mastery."
-- Ursula K. Leguin, A Wizard of Earthsea

"Neurons firing is something even a slug can do, and has as much to do with thinking as walls and doors have to do with a research institute."
-- Albert Cardona

"Mental toughness and willpower are to living in harmony with the Tao, as the ability to make clever arguments is to rationality."
-- Marcello Herreshoff

"Destiny always has been something that you tear open with your own hands."
-- T-Moon Complex X 02

"When I consider the short duration of my life, swallowed up in the eternity before and after, the little space which I fill and even can see, engulfed in the infinite immensity of spaces of which I am ignorant and which know me not, I am frightened and am astonished at being here rather than there; for there is no reason why here rather than there, why now rather than then... The eternal silence of these infinite spaces frightens me."
-- Pascal, Pensees

"Goals of Man: ☑ Don't get eaten by a lion ☑ Get out of Africa ☐ Get out of Earth ☐ Get out of Solar System ☐ Get out of Galaxy ☐ Get out of Local Group ☐ Get out of Earth-Visible Universe ☐ Get out of Universe"
-- Knome

Original with comments: Rationality Quotes 24

The Fun Theory Sequence

Eliezer Yudkowsky, 25 January 2009 11:18AM

(A shorter gloss of Fun Theory is "31 Laws of Fun", which summarizes the advice of Fun Theory to would-be Eutopian authors and futurists.)

Fun Theory is the field of knowledge that deals in questions such as "How much fun is there in the universe?", "Will we ever run out of fun?", "Are we having fun yet?" and "Could we be having more fun?"

Fun Theory is serious business. The prospect of endless boredom is routinely fielded by conservatives as a knockdown argument against research on lifespan extension, against cryonics, against all transhumanism, and occasionally against the entire Enlightenment ideal of a better future.

Many critics (including George Orwell) have commented on the inability of authors to imagine Utopias where anyone would actually want to live. If no one can imagine a Future where anyone would want to live, that may drain off motivation to work on the project. But there are some quite understandable biases that get in the way of such visualization.

Fun Theory is also the fully general reply to religious theodicy (attempts to justify why God permits evil). Our present world has flaws even from the standpoint of such eudaimonic considerations as freedom, personal responsibility, and self-reliance. Fun Theory tries to describe the dimensions along which a benevolently designed world can and should be optimized, and our present world is clearly not the result of such optimization - there is room for improvement. Fun Theory also highlights the flaws of any particular religion's perfect afterlife - you wouldn't want to go to their Heaven.

Finally, going into the details of Fun Theory helps you see that eudaimonia is complicated - that there are many properties which contribute to a life worth living. Which helps you appreciate just how worthless a galaxy would end up looking (with very high probability) if the galaxy was optimized by something with a utility function rolled up at random. The narrowness of this target is the motivation to create AIs with precisely chosen goal systems (Friendly AI).

Fun Theory is built on top of the naturalistic metaethics summarized in Joy in the Merely Good; as such, its arguments ground in "On reflection, don't you think this is what you would actually want (for yourself and others)?"

Posts in the Fun Theory sequence (reorganized by topic, not necessarily in the original chronological order):

- Prolegomena to a Theory of Fun: Fun Theory is an attempt to actually answer questions about eternal boredom that are more often posed and left hanging. Attempts to visualize Utopia are often defeated by standard biases, such as the attempt to imagine a single moment of good news ("You don't have to work anymore!") rather than a typical moment of daily life ten years later. People also believe they should enjoy various activities that they actually don't. But since human values have no supernatural source, it is quite reasonable for us to try to understand what we want. There is no external authority telling us that the future of humanity should not be fun.

- High Challenge: Life should not always be made easier for the same reason that video games should not always be made easier. Think in terms of eliminating low-quality work to make way for high-quality work, rather than eliminating all challenge. One needs games that are fun to play and not just fun to win. Life's utility function is over 4D trajectories, not just 3D outcomes. Values can legitimately be over the subjective experience, the objective result, and the challenging process by which it is achieved - the traveller, the destination and the journey.

- Complex Novelty: Are we likely to run out of new challenges, and be reduced to playing the same video game over and over? How large is Fun Space? This depends on how fast you learn; the faster you generalize, the more challenges you see as similar to each other. Learning is fun, but uses up fun; you can't have the same stroke of genius twice. But the more intelligent you are, the more potential insights you can understand; human Fun Space is larger than chimpanzee Fun Space, and not just by a linear factor of our brain size. In a well-lived life, you may need to increase in intelligence fast enough to integrate your accumulating experiences. If so, the rate at which new Fun becomes available to intelligence, is likely to overwhelmingly swamp the amount of time you could spend at that fixed level of intelligence. The Busy Beaver sequence is an infinite series of deep insights not reducible to each other or to any more general insight.

- Continuous Improvement: Humans seem to be on a hedonic treadmill; over time, we adjust to any improvements in our environment - after a month, the new sports car no longer seems quite as wonderful. This aspect of our evolved psychology is not surprising: is a rare organism in a rare environment whose optimal reproductive strategy is to rest with a smile on its face, feeling happy with what it already has. To entirely delete the hedonic treadmill seems perilously close to tampering with Boredom itself. Is there enough fun in the universe for a transhuman to jog off the treadmill - improve their life continuously, leaping to ever-higher hedonic levels before adjusting to the previous one? Can ever-higher levels of pleasure be created by the simple increase of ever-larger floating-point numbers in a digital pleasure center, or would that fail to have the full subjective quality of happiness? If we continue to bind our pleasures to novel challenges, can we find higher levels of pleasure fast enough, without cheating? The rate at which value can increase as more bits are added, and the rate at which value must increase for eudaimonia, together determine the lifespan of a mind. If minds must use exponentially more resources over time in order to lead a eudaimonic existence, their subjective lifespan is measured in mere millennia even if they can draw on galaxy-sized resources.

- Sensual Experience: Much of the anomie and disconnect in modern society can be attributed to our spending all day on tasks (like office work) that we didn't evolve to perform (unlike hunting and gathering on the savanna). Thus, many of the tasks we perform all day do not engage our senses - even the most realistic modern video game is not the same level of sensual experience as outrunning a real tiger on the real savanna. Even the best modern video game is low-bandwidth fun - a low-bandwidth connection to a relatively simple challenge, which doesn't fill our brains well as a result. But future entities could have different senses and higher-bandwidth connections to more complicated challenges, even if those challenges didn't exist on the savanna.

- Living By Your Own Strength: Our hunter-gatherer ancestors strung their own bows, wove their own baskets and whittled their own flutes. Part of our alienation from our design environment is the number of tools we use that we don't understand and couldn't make for ourselves. It's much less fun to read something in a book than to discover it for yourself. Specialization is critical to our current civilization. But the future does not have to be a continuation of this trend in which we rely more and more on things outside ourselves which become less and less comprehensible. With a surplus of power, you could begin to rethink the life experience as a road to internalizing new strengths, not just staying alive efficiently through extreme specialization.

- Free to Optimize: Stare decisis is the legal principle which binds courts to follow precedent. The rationale is not that past courts were wiser, but jurisprudence constante: The legal system must be predictable so that people can implement contracts and behaviors knowing their implications. The purpose of law is not to make the world perfect, but to provide a predictable environment in which people can optimize their own futures. If an extremely powerful entity is choosing good futures on your behalf, that may leave little slack for you to navigate through your own strength. Describing how an AI can avoid stomping your self-determination is a structurally complicated problem. A simple (possibly not best) solution would be the gift of a world that works by improved rules, stable enough that the inhabitants could understand them and optimize their own futures together, but otherwise hands-off.  Modern legal systems fail along this dimension; no one can possibly know all the laws, let alone obey them.

- Harmful Options: Offering people more choices that differ along many dimensions, may diminish their satisfaction with their final choice. Losses are more painful than the corresponding gains are pleasurable, so people think of the dimensions along which their final choice was inferior, and of all the other opportunities passed up. If you can only choose one dessert, you're likely to be happier choosing from a menu of two than from a menu of fourteen. Refusing tempting choices consumes mental energy and decreases performance on other cognitive tasks. A video game that contained an always-visible easier route through, would probably be less fun to play even if that easier route were deliberately foregone. You can imagine a Devil who follows someone around, making their life miserable, solely by offering them options which are never actually taken. And what if a worse option is taken due to a predictable mistake? There are many ways to harm people by offering them more choices.

- Devil's Offers: It is dangerous to live in an environment in which a single failure of resolve, throughout your entire life, can result in a permanent addiction or in a poor edit of your own brain. For example, a civilization which is constantly offering people tempting ways to shoot off their own feet - for example, offering them a cheap escape into eternal virtual reality, or customized drugs. It requires a constant stern will that may not be much fun. And it's questionable whether a superintelligence that descends from above to offer people huge dangerous temptations that they wouldn't encounter on their own, is helping.

- Nonperson Predicates, Nonsentient Optimizers, Can't Unbirth a Child: Discusses some of the problems of, and justification for, creating AIs that are knowably not conscious / sentient / people / citizens / subjective experiencers. We don't want the AI's models of people to be people - we don't want conscious minds trapped helplessly inside it. So we need how to tell that something is definitely not a person, and in this case, maybe we would like the AI itself to not be a person, which would simplify a lot of ethical issues if we could pull it off. Creating a new intelligent species is not lightly to be undertaken from a purely ethical perspective; if you create a new kind of person, you have to make sure it leads a life worth living.

- Amputation of Destiny: C. S. Lewis's Narnia has a problem, and that problem is the super-lion Aslan - who demotes the four human children from the status of main characters, to mere hangers-on while Aslan does all the work. Iain Banks's Culture novels have a similar problem; the humans are mere hangers-on of the superintelligent Minds. We already have strong ethical reasons to prefer to create nonsentient AIs rather than sentient AIs, at least at first. But we may also prefer in just a fun-theoretic sense that we not be overshadowed by hugely more powerful entities occupying a level playing field with us. Entities with human emotional makeups should not be competing on a level playing field with superintelligences - either keep the superintelligences off the playing field, or design the smaller (human-level) minds with a different emotional makeup that doesn't mind being overshadowed.

- Dunbar's Function: Robin Dunbar's original calculation showed that the maximum human group size was around 150. But a typical size for a hunter-gatherer band would be 30-50, cohesive online groups peak at 50-60, and small task forces may peak in internal cohesiveness around 7. Our attempt to live in a world of six billion people has many emotional costs: We aren't likely to know our President or Prime Minister, or to have any significant influence over our country's politics, although we go on behaving as if we did. We are constantly bombarded with news about improbably pretty and wealthy individuals. We aren't likely to find a significant profession where we can be the best in our field. But if intelligence keeps increasing, the number of personal relationships we can track will also increase, along with the natural degree of specialization. Eventually there might be a single community of sentients that really was a single community.

- In Praise of Boredom:  "Boredom" is an immensely subtle and important aspect of human values, nowhere near as straightforward as it sounds to a human. We don't want to get bored with breathing or with thinking. We do want to get bored with playing the same level of the same video game over and over. We don't want changing the shade of the pixels in the game to make it stop counting as "the same game". We want a steady stream of novelty, rather than spending most of our time playing the best video game level so far discovered (over and over) and occasionally trying out a different video game level as a new candidate for "best". These considerations would not arise in most utility functions in expected utility maximizers.

- Sympathetic Minds:  Mirror neurons are neurons that fire both when performing an action oneself, and watching someone else perform the same action - for example, a neuron that fires when you raise your hand or watch someone else raise theirs. We predictively model other minds by putting ourselves in their shoes, which is empathy. But some of our desire to help relatives and friends, or be concerned with the feelings of allies, is expressed as sympathy, feeling what (we believe) they feel. Like "boredom", the human form of sympathy would not be expected to arise in an arbitrary expected-utility-maximizing AI. Most such agents would regard any agents in its environment as a special case of complex systems to be modeled or optimized; it would not feel what they feel.

- Interpersonal Entanglement:  Our sympathy with other minds makes our interpersonal relationships one of the most complex aspects of human existence. Romance, in particular, is more complicated than being nice to friends and kin, negotiating with allies, or outsmarting enemies - it contains aspects of all three. Replacing human romance with anything simpler or easier would decrease the peak complexity of the human species - a major step in the wrong direction, it seems to me. This is my problem with proposals to give people perfect, nonsentient sexual/romantic partners, which I usually refer to as "catgirls" ("catboys"). The human species does have a statistical sex problem: evolution has not optimized the average man to make the average woman happy or vice versa. But there are less sad ways to solve this problem than both genders giving up on each other and retreating to catgirls/catboys.

- Failed Utopia #4-2:  A fictional short story illustrating some of the ideas in Interpersonal Entanglement above. (Many commenters seemed to like this story, and some said that the ideas were easier to understand in this form.)

- Growing Up is Hard:  Each piece of the human brain is optimized on the assumption that all the other pieces are working the same way they did in the ancestral environment. Simple neurotransmitter imbalances can result in psychosis, and some aspects of Williams Syndrome are probably due to having a frontal cortex that is too large relative to the rest of the brain. Evolution creates limited robustness, but often stepping outside the ancestral parameter box just breaks things. Even if the first change works, the second and third changes are less likely to work as the total parameters get less ancestral and the brain's tolerance is used up. A cleanly designed AI might improve itself to the point where it was smart enough to unravel and augment the human brain. Or uploads might be able to make themselves smart enough to solve the increasingly difficult problem of not going slowly, subtly insane. Neither path is easy. There seems to be an irreducible residue of danger and difficulty associated with an adult version of humankind ever coming into being. Being a transhumanist means wanting certain things; it doesn't mean you think those things are easy.

- Changing Emotions:  Creating new emotions seems like a desirable aspect of many parts of Fun Theory, but this is not to be trivially postulated. It's the sort of thing best done with superintelligent help, and slowly and conservatively even then. We can illustrate these difficulties by trying to translate the short English phrase "change sex" into a cognitive transformation of extraordinary complexity and many hidden subproblems.

- Emotional Involvement: Since the events in video games have no actual long-term consequences, playing a video game is not likely to be nearly as emotionally involving as much less dramatic events in real life. The supposed Utopia of playing lots of cool video games forever, is life as a series of disconnected episodes with no lasting consequences. Our current emotions are bound to activities that were subgoals of reproduction in the ancestral environment - but we now pursue these activities as independent goals regardless of whether they lead to reproduction.  (Sex with birth control is the classic example.) A transhuman existence would need new emotions suited to the important short-term and long-term events of that existence.

- Serious Stories: Stories and lives are optimized according to rather different criteria. Advice on how to write fiction will tell you that "stories are about people's pain" and "every scene must end in disaster". I once assumed that it was not possible to write any story about a successful Singularity because the inhabitants would not be in any pain; but something about the final conclusion that the post-Singularity world would contain no stories worth telling seemed alarming. Stories in which nothing ever goes wrong, are painful to read; would a life of endless success have the same painful quality? If so, should we simply eliminate that revulsion via neural rewiring? Pleasure probably does retain its meaning in the absence of pain to contrast it; they are different neural systems. The present world has an imbalance between pain and pleasure; it is much easier to produce severe pain than correspondingly intense pleasure. One path would be to address the imbalance and create a world with more pleasures, and free of the more grindingly destructive and pointless sorts of pain. Another approach would be to eliminate pain entirely. I feel like I prefer the former approach, but I don't know if it can last in the long run.

- Eutopia is Scary: If a citizen of the Past were dropped into the Present world, they would be pleasantly surprised along at least some dimensions; they would also be horrified, disgusted, and frightened. This is not because our world has gone wrong, but because it has gone right. A true Future gone right would, realistically, be shocking to us along at least some dimensions. This may help explain why most literary Utopias fail; as George Orwell observed, "they are chiefly concerned with avoiding fuss". Heavens are meant to sound like good news; political utopias are meant to show how neatly their underlying ideas work. Utopia is reassuring, unsurprising, and dull. Eutopia would be scary. (Of course the vast majority of scary things are not Eutopian, just entropic.) Try to imagine a genuinely better world in which you would be out of place - not a world that would make you smugly satisfied at how well all your current ideas had worked. This proved to be a very important exercise when I tried it; it made me realize that all my old proposals had been optimized to sound safe and reassuring.

- Building Weirdtopia: Utopia and Dystopia both confirm the moral sensibilities you started with; whether the world is a libertarian utopia of government non-interference, or a hellish dystopia of government intrusion and regulation, either way you get to say "Guess I was right all along." To break out of this mold, write down the Utopia, and the Dystopia, and then try to write down the Weirdtopia - an arguably-better world that zogs instead of zigging or zagging. (Judging from the comments, this exercise seems to have mostly failed.)

- Justified Expectation of Pleasant Surprises: A pleasant surprise probably has a greater hedonic impact than being told about the same positive event long in advance - hearing about the positive event is good news in the moment of first hearing, but you don't have the gift actually in hand. Then you have to wait, perhaps for a long time, possibly comparing the expected pleasure of the future to the lesser pleasure of the present. This argues that if you have a choice between a world in which the same pleasant events occur, but in the first world you are told about them long in advance, and in the second world they are kept secret until they occur, you would prefer to live in the second world. The importance of hope is widely appreciated - people who do not expect their lives to improve in the future are less likely to be happy in the present - but the importance of vague hope may be understated.

- Seduced by Imagination: Vagueness usually has a poor name in rationality, but the Future is something about which, in fact, we do not possess strong reliable specific information. Vague (but justified!) hopes may also be hedonically better. But a more important caution for today's world is that highly specific pleasant scenarios can exert a dangerous power over human minds - suck out our emotional energy, make us forget what we don't know, and cause our mere actual lives to pale by comparison. (This post is not about Fun Theory proper, but it contains an important warning about how not to use Fun Theory.)

- The Uses of Fun (Theory): Fun Theory is important for replying to critics of human progress; for inspiring people to keep working on human progress; for refuting religious arguments that the world could possibly have been benevolently designed; for showing that religious Heavens show the signature of the same human biases that torpedo other attempts at Utopia; and for appreciating the great complexity of our values and of a life worth living, which requires a correspondingly strong effort of AI design to create AIs that can play good roles in a good future.

- Higher Purpose: Having a Purpose in Life consistently shows up as something that increases stated well-being. Of course, the problem with trying to pick out "a Purpose in Life" in order to make yourself happier, is that this doesn't take you outside yourself; it's still all about you. To find purpose, you need to turn your eyes outward to look at the world and find things there that you care about - rather than obsessing about the wonderful spiritual benefits you're getting from helping others. In today's world, most of the highest-priority legitimate Causes consist of large groups of people in extreme jeopardy: Aging threatens the old, starvation threatens the poor, extinction risks threaten humanity as a whole. If the future goes right, many and perhaps all such problems will be solved - depleting the stream of victims to be helped. Will the future therefore consist of self-obsessed individuals, with nothing to take them outside themselves? I suggest, though, that even if there were no large groups of people in extreme jeopardy, we would still, looking around, find things outside ourselves that we cared about - friends, family; truth, freedom... Nonetheless, if the Future goes sufficiently well, there will come a time when you could search the whole of civilization, and never find a single person so much in need of help, as dozens you now pass on the street. If you do want to save someone from death, or help a great many people, then act now; your opportunity may not last, one way or another.

Referenced by: Existential Angst Factory • Prolegomena to a Theory of Fun • High Challenge • Complex Novelty • Sensual Experience • Living By Your Own Strength • Harmful Options • Devil's Offers • Amputation of Destiny • Dunbar's Function • Free to Optimize • Growing Up is Hard • Changing Emotions • Emotional Involvement • Serious Stories • Continuous Improvement • Eutopia is Scary • In Praise of Boredom • Sympathetic Minds • Interpersonal Entanglement • Higher Purpose • 31 Laws of Fun • Value is Fragile • The Thing That I Protect • Raising the Sanity Waterline • Money: The Unit of Caring

Original with comments: The Fun Theory Sequence

BHTV: Yudkowsky / Wilkinson

Eliezer Yudkowsky, 26 January 2009 01:10AM

Eliezer Yudkowsky and Will Wilkinson. Due to a technical mistake - I won't say which of us made it, except that it wasn't me - the video cuts out at 47:37, but the MP3 of the full dialogue is available here. I recall there was some good stuff at the end, too.

We talked about Obama up to 23 minutes, then it's on to rationality. Wilkinson introduces (invents?) the phrase "good cognitive citizenship" which is a great phrase that I am totally going to steal.

Original with comments: BHTV: Yudkowsky / Wilkinson

31 Laws of Fun

Eliezer Yudkowsky, 26 January 2009 10:13AM

So this is Utopia, is it? Well
I beg your pardon, I thought it was Hell.
-- Sir Max Beerholm, verse entitled
In a Copy of More's (or Shaw's or Wells's or Plato's or Anybody's) Utopia

This is a shorter summary of the Fun Theory Sequence with all the background theory left out - just the compressed advice to the would-be author or futurist who wishes to imagine a world where people might actually want to live:

1. Think of a typical day in the life of someone who's been adapting to Utopia for a while. Don't anchor on the first moment of "hearing the good news". Heaven's "You'll never have to work again, and the streets are paved with gold!" sounds like good news to a tired and poverty-stricken peasant, but two months later it might not be so much fun. (Prolegomena to a Theory of Fun.)

2. Beware of packing your Utopia with things you think people should do that aren't actually fun. Again, consider Christian Heaven: singing hymns doesn't sound like loads of endless fun, but you're supposed to enjoy praying, so no one can point this out. (Prolegomena to a Theory of Fun.)

3. Making a video game easier doesn't always improve it. The same holds true of a life. Think in terms of clearing out low-quality drudgery to make way for high-quality challenge, rather than eliminating work. (High Challenge.)

4. Life should contain novelty - experiences you haven't encountered before, preferably teaching you something you didn't already know. If there isn't a sufficient supply of novelty (relative to the speed at which you generalize), you'll get bored. (Complex Novelty.)

1. People should get smarter at a rate sufficient to integrate their old experiences, but not so much smarter so fast that they can't integrate their new intelligence. Being smarter means you get bored faster, but you can also tackle new challenges you couldn't understand before. (Complex Novelty.)

2. People should live in a world that fully engages their senses, their bodies, and their brains. This means either that the world resembles the ancestral savanna more than say a windowless office; or alternatively, that brains and bodies have changed to be fully engaged by different kinds of complicated challenges and environments. (Fictions intended to entertain a human audience should concentrate primarily on the former option.) (Sensual Experience.)

3. Timothy Ferris: "What is the opposite of happiness? Sadness? No. Just as love and hate are two sides of the same coin, so are happiness and sadness... The opposite of love is indifference, and the opposite of happiness is - here's the clincher - boredom... The question you should be asking isn't 'What do I want?' or 'What are my goals?' but 'What would excite me?'... Living like a millionaire requires doing interesting things and not just owning enviable things." (Existential Angst Factory.)

4. Any particular individual's life should get better and better over time. (Continuous Improvement.)

5. You should not know exactly what improvements the future holds, although you should look forward to finding out. The actual event should come as a pleasant surprise. (Justified Expectation of Pleasant Surprises.)

6. Our hunter-gatherer ancestors strung their own bows, wove their own baskets and whittled their own flutes; then they did their own hunting, their own gathering and played their own music. Futuristic Utopias are often depicted as offering more and more neat buttons that do less and less comprehensible things for you. Ask not what interesting things Utopia can do for people; ask rather what interesting things the inhabitants could do for themselves - with their own brains, their own bodies, or tools they understand how to build. (Living By Your Own Strength.)

7. Living in Eutopia should make people stronger, not weaker, over time. The inhabitants should appear more formidable than the people of our own world, not less. (Living By Your Own Strength; see also, Tsuyoku Naritai.)

8. Life should not be broken up into a series of disconnected episodes with no long-term consequences. No matter how sensual or complex, playing one really great video game after another, does not make a life story. (Emotional Involvement.)

9. People should make their own destinies; their lives should not be choreographed to the point that they no longer need to imagine, plan and navigate their own futures. Citizens should not be the pawns of more powerful gods, still less their sculpted material. One simple solution would be to have the world work by stable rules that are the same for everyone, where the burden of Eutopia is carried by a good initial choice of rules, rather than by any optimization pressure applied to individual lives. (Free to Optimize.)

10. Human minds should not have to play on a level field with vastly superior entities. Most people don't like being overshadowed. Gods destroy a human protagonist's "main character" status; this is undesirable in fiction and probably in real life. (E.g.: C. S. Lewis's Narnia, Iain Banks's Culture.) Either change people's emotional makeup so that they don't mind being unnecessary, or keep the gods way off their playing field. Fictional stories intended for human audiences cannot do the former. (And in real life, you probably can have powerful AIs that are neither sentient nor meddlesome. See the main post and its prerequisites.) (Amputation of Destiny.)

11. Trying to compete on a single flat playing field with six billion other humans also creates problems. Our ancestors lived in bands of around 50 people. Today the media is constantly bombarding us with news of exceptionally rich and pretty people as if they lived next door to us; and very few people get a chance to be the best at any specialty. (Dunbar's Function.)

12. Our ancestors also had some degree of genuine control over their band's politics. Contrast to modern nation-states where almost no one knows the President on a personal level or could argue Congress out of a bad decision. (Though that doesn't stop people from arguing as loudly as if they still lived in a 50-person band.) (Dunbar's Function.)

13. Offering people more options is not always helping them (especially if the option is something they couldn't do for themselves). Losses are more painful than the corresponding gains, so if choices are different along many dimensions and only one choice can be taken, people tend to focus on the loss of the road not taken. Offering a road that bypasses a challenge makes the challenge feel less real, even if the cheat is diligently refused. It is also a sad fact that humans predictably make certain kinds of mistakes. Don't assume that building more choice into your Utopia is necessarily an improvement because "people can always just say no". This sounds reassuring to an outside reader - "Don't worry, you'll decide! You trust yourself, right?" - but might not be much fun to actually live with. (Harmful Options.)

14. Extreme example of the above: being constantly offered huge temptations that are incredibly dangerous - a completely realistic virtual world, or very addictive and pleasurable drugs. You can never allow yourself a single moment of willpower failure over your whole life. (E.g.: John C. Wright's Golden Oecumene.) (Devil's Offers.)

15. Conversely, when people are grown strong enough to shoot off their feet without external help, stopping them may be too much interference. Hopefully they'll then be smart enough not to: By the time they can build the gun, they'll know what happens if they pull the gun, and won't need a smothering safety blanket. If that's the theory, then dangerous options need correspondingly difficult locks. (Devil's Offers.)

16. Telling people truths they haven't yet figured out for themselves, is not always helping them. (Joy in Discovery.)

17. Brains are some of the most complicated things in the world. Thus, other humans (other minds) are some of the most complicated things we deal with. For us, this interaction has a unique character because of the sympathy we feel for others - the way that our brain tends to align with their brain - rather than our brain just treating other brains as big complicated machines with levers to pull. Reducing the need for people to interact with other people reduces the complexity of human existence; this is a step in the wrong direction. For example, resist the temptation to simplify people's lives by offering them artificially perfect sexual/romantic partners. (Interpersonal Entanglement.)

18. But admittedly, humanity does have a statistical sex problem: the male distribution of attributes doesn't harmonize with the female distribution of desires, or vice versa. Not everything in Eutopia should be easy - but it shouldn't be pointlessly, unresolvably frustrating either. (This is a general principle.) So imagine nudging the distributions to make the problem solvable - rather than waving a magic wand and solving everything instantly. (Interpersonal Entanglement.)

19. In general, tampering with brains, minds, emotions, and personalities is way more fraught on every possible level of ethics and difficulty, than tampering with bodies and environments. Always ask what you can do by messing with the environment before you imagine messing with minds. Then prefer small cognitive changes to big ones. You're not just outrunning your human audience, you're outrunning your own imagination. (Changing Emotions.)

20. In this present world, there is an imbalance between pleasure and pain. An unskilled torturer with simple tools can create worse pain in thirty seconds, than an extremely skilled sexual artist can create pleasure in thirty minutes. One response would be to remedy the imbalance - to have the world contain more joy than sorrow. Pain might exist, but not pointless endless unendurable pain. Mistakes would have more proportionate penalties: You might touch a hot stove and end up with a painful blister; but not glance away for two seconds and spend the rest of your life in a wheelchair. The people would be stronger, less exhausted. This path would eliminate mind-destroying pain, and make pleasure more abundant. Another path would eliminate pain entirely. Whatever the relative merits of the real-world proposals, fictional stories cannot take the second path. (Serious Stories.)

21. George Orwell once observed that Utopias are chiefly concerned with avoiding fuss. Don't be afraid to write a loud Eutopia that might wake up the neighbors. (Eutopia is Scary; George Orwell's Why Socialists Don't Believe in Fun.)

22. George Orwell observed that "The inhabitants of perfect universes seem to have no spontaneous gaiety and are usually somewhat repulsive into the bargain." If you write a story and your characters turn out like this, it probably reflects some much deeper flaw that can't be fixed by having the State hire a few clowns. (George Orwell's Why Socialists Don't Believe in Fun.)

23. Ben Franklin, yanked into our own era, would be surprised and delighted by some aspects of his Future. Other aspects would horrify, disgust, and frighten him; and this is not because our world has gone wrong, but because it has improved relative to his time. Relatively few things would have gone just as Ben Franklin expected. If you imagine a world which your imagination finds familiar and comforting, it will inspire few others, and the whole exercise will lack integrity. Try to conceive of a genuinely better world in which you, yourself, would be shocked (at least at first) and out of place (at least at first). (Eutopia is Scary.)

24. Utopia and Dystopia are two sides of the same coin; both just confirm the moral sensibilities you started with. Whether the world is a libertarian utopia of government non-interference, or a hellish dystopia of government intrusion and regulation, you get to say "I was right all along." Don't just imagine something that conforms to your existing ideals of government, relationships, politics, work, or daily life. Find the better world that zogs instead of zigging or zagging. (To safeguard your sensibilities, you can tell yourself it's just an arguably better world but isn't really better than your favorite standard Utopia... but you'll know you're really doing it right if you find your ideals changing.) (Building Weirdtopia.)

25. If your Utopia still seems like an endless gloomy drudgery of existential angst no matter how much you try to brighten it, there's at least one major problem that you're entirely failing to focus on. (Existential Angst Factory.)

26. 'Tis a sad mind that cares about nothing except itself. In the modern-day world, if an altruist looks around, their eye is caught by large groups of people in desperate jeopardy. People in a better world will not see this: A true Eutopia will run low on victims to be rescued. This doesn't imply that the inhabitants look around outside themselves and see nothing. They may care about friends and family, truth and freedom, common projects; outside minds, shared goals, and high ideals. (Higher Purpose.)

27. Still, a story that confronts the challenge of Eutopia should not just have the convenient plot of "The Dark Lord Sauron is about to invade and kill everybody". The would-be author will have to find something slightly less awful for his characters to legitimately care about. This is part of the challenge of showing that human progress is not the end of human stories, and that people not in imminent danger of death can still lead interesting lives. Those of you interested in confronting lethal planetary-sized dangers should focus on present-day real life. (Higher Purpose.)

The simultaneous solution of all these design requirements is left as an exercise to the reader. At least for now.

The enumeration in this post of certain Laws shall not be construed to deny or disparage others not mentioned. I didn't happen to write about humor, but it would be a sad world that held no laughter, etcetera.

To anyone seriously interested in trying to write a Eutopian story using these Laws: You must first know how to write. There are many, many books on how to write; you should read at least three; and they will all tell you that a great deal of practice is required. Your practice stories should not be composed anywhere so difficult as Eutopia. That said, my second most important advice for authors is this: Life will never become boringly easy for your characters so long as they can make things difficult for each other.

Finally, this dire warning: Concretely imagining worlds much better than your present-day real life, may suck out your soul like an emotional vacuum cleaner. (See Seduced by Imagination.) Fun Theory is dangerous, use it with caution, you have been warned.

Referenced by: The Fun Theory Sequence • The Thing That I Protect

Original with comments: 31 Laws of Fun

OB Status Update

Eliezer Yudkowsky, 27 January 2009 09:34AM

Followup to: Whither OB?

Overcoming Bias currently plans to transition to a new format, including a new and more open sister site, tentatively entitled "Less Wrong". The new site will be built out of Reddit's source code, but you won't be limited to posting links - the new site will include a WYSIWYG HTML editor as well. All posts will appear on Less Wrong and will be voted up or down by the readers. Posts approved by the chief editors will be "promoted" to Overcoming Bias, which will serve as the front page of Less Wrong.

Once the initial site is up and running, the next items on the agenda include much better support for reading through sequences. And I'll organize more of my old posts (and perhaps some of Robin's) into sequences.

Threaded comments and comment voting/sorting are on the way. Anonymous commenting may go away briefly (it's not built into the Reddit codebase) but I suspect it's important for attracting new participation. So I do hope to bring back non-registration commenting, but it may go away for a while. On the plus side, you'll only have to solve a captcha once when signing up, not every time you post. And the 50-comment limit per page is on the way out as well.

Timeframe... theoretically, one to two weeks of work left.

I've reserved a final sequence on Building Rationalist Communities to seed Less Wrong. Also, I doubt I could stop blogging completely even if I tried. I don't think Robin plans to stop completely either. And it's worth remembering that OB's most popular post ever was a reader contribution. So don't touch that dial, don't unsubscribe that RSS feed. Exciting changes on the way.

PS: Changes also underway at the Singularity Institute, my host organization. Although no formal announcement has been made yet, Tyler Emerson is departing as Executive Director and Michael Vassar is coming on board as President. I mention this now because Michael Vassar is usually based out of the East Coast, but is in the Bay Area for only the next couple of days, and any Bay Area folks seriously interested in getting significantly involved might want to take this opportunity to talk to him and/or me. Email michael no space aruna at yahoo dot com.

Original with comments: OB Status Update

Rationality Quotes 25

Eliezer Yudkowsky, 28 January 2009 08:39AM

"People want to think there is some huge conspiracy run by evil geniuses. The reality is actually much more horrifying. The people running the show aren't evil geniuses. They are just as stupid as the rest of us."
-- Vaksel

"Rule of thumb: Be skeptical of things you learned before you could read. E.g., religion."
-- Ben Casnocha

"Truth is not always popular, but it is always right."
-- Anon

"Computer programming is omnipotence without omniscience."
-- Prospero

"The world breaks everyone and afterward many are strong in the broken places. But those that will not break it kills. It kills the very good, and the very gentle, and the very brave, impartially."
-- Ernest Hemingway, A Farewell to Arms

"Those who take delight in their own might are merely pretenders to power. The true warrior of fate needs no adoration or fear, no tricks or overwhelming effort; he need not be stronger or smarter or innately more capable than everyone else; he need not even admit it to himself. All he needs to do is to stand there, at that moment when all hope is dead, and look upon the abyss without flinching."
-- Shinji and Warhammer40k

"Though here at journey's end I lie
in darkness buried deep,
beyond all towers strong and high,
beyond all mountains steep,
above all shadows rides the Sun
and Stars forever dwell:
I will not say the Day is done,
nor bid the Stars farewell."
-- Banazîr Galbasi (Samwise Gamgee), The Return of the King

Referenced by: The Thing That I Protect • Against Maturity

Original with comments: Rationality Quotes 25

Value is Fragile

Eliezer Yudkowsky, 29 January 2009 08:46AM

Followup to: The Fun Theory Sequence, Fake Fake Utility Functions, Joy in the Merely Good, The Hidden Complexity of Wishes, The Gift We Give To Tomorrow, No Universally Compelling Arguments, Anthropomorphic Optimism, Magical Categories, ...

If I had to pick a single statement that relies on more Overcoming Bias content I've written than any other, that statement would be:

Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.

"Well," says the one, "maybe according to your provincial human values, you wouldn't like it. But I can easily imagine a galactic civilization full of agents who are nothing like you, yet find great value and interest in their own goals. And that's fine by me. I'm not so bigoted as you are. Let the Future go its own way, without trying to bind it forever to the laughably primitive prejudices of a pack of four-limbed Squishy Things -"

My friend, I have no problem with the thought of a galactic civilization vastly unlike our own... full of strange beings who look nothing like me even in their own imaginations... pursuing pleasures and experiences I can't begin to empathize with... trading in a marketplace of unimaginable goods... allying to pursue incomprehensible objectives... people whose life-stories I could never understand.

That's what the Future looks like if things go right.

If the chain of inheritance from human (meta)morals is broken, the Future does not look like this. It does not end up magically, delightfully incomprehensible.

With very high probability, it ends up looking dull. Pointless. Something whose loss you wouldn't mourn.

Seeing this as obvious, is what requires that immense amount of background explanation.

And I'm not going to iterate through all the points and winding pathways of argument here, because that would take us back through 75% of my Overcoming Bias posts. Except to remark on how many different things must be known to constrain the final answer.

Consider the incredibly important human value of "boredom" - our desire not to do "the same thing" over and over and over again. You can imagine a mind that contained almost the whole specification of human value, almost all the morals and metamorals, but left out just this one thing -

- and so it spent until the end of time, and until the farthest reaches of its light cone, replaying a single highly optimized experience, over and over and over again.

Or imagine a mind that contained almost the whole specification of which sort of feelings humans most enjoy - but not the idea that those feelings had important external referents. So that the mind just went around feeling like it had made an important discovery, feeling it had found the perfect lover, feeling it had helped a friend, but not actually doing any of those things - having become its own experience machine. And if the mind pursued those feelings and their referents, it would be a good future and true; but because this one dimension of value was left out, the future became something dull. Boring and repetitive, because although this mind felt that it was encountering experiences of incredible novelty, this feeling was in no wise true.

Or the converse problem - an agent that contains all the aspects of human value, except the valuation of subjective experience. So that the result is a nonsentient optimizer that goes around making genuine discoveries, but the discoveries are not savored and enjoyed, because there is no one there to do so. This, I admit, I don't quite know to be possible. Consciousness does still confuse me to some extent. But a universe with no one to bear witness to it, might as well not be.

Value isn't just complicated, it's fragile. There is more than one dimension of human value, where if just that one thing is lost, the Future becomes null. A single blow and all value shatters. Not every single blow will shatter all value - but more than one possible "single blow" will do so.

And then there are the long defenses of this proposition, which relies on 75% of my Overcoming Bias posts, so that it would be more than one day's work to summarize all of it. Maybe some other week. There's so many branches I've seen that discussion tree go down.

After all - a mind shouldn't just go around having the same experience over and over and over again. Surely no superintelligence would be so grossly mistaken about the correct action?

Why would any supermind want something so inherently worthless as the feeling of discovery without any real discoveries? Even if that were its utility function, wouldn't it just notice that its utility function was wrong, and rewrite it? It's got free will, right?

Surely, at least boredom has to be a universal value. It evolved in humans because it's valuable, right? So any mind that doesn't share our dislike of repetition, will fail to thrive in the universe and be eliminated...

If you are familiar with the difference between instrumental values and terminal values, and familiar with the stupidity of natural selection, and you understand how this stupidity manifests in the difference between executing adaptations versus maximizing fitness, and you know this turned instrumental subgoals of reproduction into decontextualized unconditional emotions...

...and you're familiar with how the tradeoff between exploration and exploitation works in Artificial Intelligence...

...then you might be able to see that the human form of boredom that demands a steady trickle of novelty for its own sake, isn't a grand universal, but just a particular algorithm that evolution coughed out into us. And you might be able to see how the vast majority of possible expected utility maximizers, would only engage in just so much efficient exploration, and spend most of its time exploiting the best alternative found so far, over and over and over.

That's a lot of background knowledge, though.

And so on and so on and so on through 75% of my posts on Overcoming Bias, and many chains of fallacy and counter-explanation. Some week I may try to write up the whole diagram. But for now I'm going to assume that you've read the arguments, and just deliver the conclusion:

We can't relax our grip on the future - let go of the steering wheel - and still end up with anything of value.

And those who think we can -

- they're trying to be cosmopolitan. I understand that. I read those same science fiction books as a kid: The provincial villains who enslave aliens for the crime of not looking just like humans. The provincial villains who enslave helpless AIs in durance vile on the assumption that silicon can't be sentient. And the cosmopolitan heroes who understand that minds don't have to be just like us to be embraced as valuable -

I read those books. I once believed them. But the beauty that jumps out of one box, is not jumping out of all boxes. (This being the moral of the sequence on Lawful Creativity.) If you leave behind all order, what is left is not the perfect answer, what is left is perfect noise. Sometimes you have to abandon an old design rule to build a better mousetrap, but that's not the same as giving up all design rules and collecting wood shavings into a heap, with every pattern of wood as good as any other. The old rule is always abandoned at the behest of some higher rule, some higher criterion of value that governs.

If you loose the grip of human morals and metamorals - the result is not mysterious and alien and beautiful by the standards of human value. It is moral noise, a universe tiled with paperclips. To change away from human morals in the direction of improvement rather than entropy, requires a criterion of improvement; and that criterion would be physically represented in our brains, and our brains alone.

Relax the grip of human value upon the universe, and it will end up seriously valueless. Not, strange and alien and wonderful, shocking and terrifying and beautiful beyond all human imagination. Just, tiled with paperclips.

It's only some humans, you see, who have this idea of embracing manifold varieties of mind - of wanting the Future to be something greater than the past - of being not bound to our past selves - of trying to change and move forward.

A paperclip maximizer just chooses whichever action leads to the greatest number of paperclips.

No free lunch. You want a wonderful and mysterious universe? That's your value. You work to create that value. Let that value exert its force through you who represents it, let it make decisions in you to shape the future. And maybe you shall indeed obtain a wonderful and mysterious universe.

No free lunch. Valuable things appear because a goal system that values them takes action to create them. Paperclips don't materialize from nowhere for a paperclip maximizer. And a wonderfully alien and mysterious Future will not materialize from nowhere for us humans, if our values that prefer it are physically obliterated - or even disturbed in the wrong dimension. Then there is nothing left in the universe that works to make the universe valuable.

You do have values, even when you're trying to be "cosmopolitan", trying to display a properly virtuous appreciation of alien minds. Your values are then faded further into the invisible background - they are less obviously human. Your brain probably won't even generate an alternative so awful that it would wake you up, make you say "No! Something went wrong!" even at your most cosmopolitan. E.g. "a nonsentient optimizer absorbs all matter in its future light cone and tiles the universe with paperclips". You'll just imagine strange alien worlds to appreciate.

Trying to be "cosmopolitan" - to be a citizen of the cosmos - just strips off a surface veneer of goals that seem obviously "human".

But if you wouldn't like the Future tiled over with paperclips, and you would prefer a civilization of...

...sentient beings...

...with enjoyable experiences...

...that aren't the same experience over and over again...

...and are bound to something besides just being a sequence of internal pleasurable feelings...

...learning, discovering, freely choosing...

...well, I've just been through the posts on Fun Theory that went into some of the hidden details on those short English words.

Values that you might praise as cosmopolitan or universal or fundamental or obvious common sense, are represented in your brain just as much as those values that you might dismiss as merely human. Those values come of the long history of humanity, and the morally miraculous stupidity of evolution that created us. (And once I finally came to that realization, I felt less ashamed of values that seemed 'provincial' - but that's another matter.)

These values do not emerge in all possible minds. They will not appear from nowhere to rebuke and revoke the utility function of an expected paperclip maximizer.

Touch too hard in the wrong dimension, and the physical representation of those values will shatter - and not come back, for there will be nothing left to want to bring it back.

And the referent of those values - a worthwhile universe - would no longer have any physical reason to come into being.

Let go of the steering wheel, and the Future crashes.

Referenced by: The Thing That I Protect • Raising the Sanity Waterline

Original with comments: Value is Fragile

Three Worlds Collide (0/8)

Eliezer Yudkowsky, 30 January 2009 12:07PM

"The kind of classic fifties-era first-contact story that Jonathan Swift might have written, if Jonathan Swift had had a background in game theory."
-- (Hugo nominee) Peter Watts, "In Praise of Baby-Eating"

Three Worlds Collide is a story I wrote to illustrate some points on naturalistic metaethics and diverse other issues of rational conduct. It grew, as such things do, into a small novella. On publication, it proved widely popular and widely criticized. Be warned that the story, as it wrote itself, ended up containing some profanity and PG-13 content.

1. The Baby-Eating Aliens

2. War and/or Peace

3. The Super Happy People

4. Interlude with the Confessor

5. Three Worlds Decide

6. Normal Ending

7. True Ending

8. Atonement

PDF version here.

Referenced by: The Baby-Eating Aliens (1/8) • War and/or Peace (2/8) • The Super Happy People (3/8) • Interlude with the Confessor (4/8) • Three Worlds Decide (5/8) • Normal Ending: Last Tears (6/8) • True Ending: Sacrificial Fire (7/8) • Epilogue: Atonement (8/8) • (Moral) Truth in Fiction?

Original with comments: Three Worlds Collide (0/8)

The Baby-Eating Aliens (1/8)

Eliezer Yudkowsky, 30 January 2009 12:07PM

(Part 1 of 8 in "Three Worlds Collide")

This is a story of an impossible outcome, where AI never worked, molecular nanotechnology never worked, biotechnology only sort-of worked; and yet somehow humanity not only survived, but discovered a way to travel Faster-Than-Light: The past's Future.

Ships travel through the Alderson starlines, wormholes that appear near stars. The starline network is dense and unpredictable: more than a billion starlines lead away from Sol, but every world explored is so far away as to be outside the range of Earth's telescopes. Most colony worlds are located only a single jump away from Earth, which remains the center of the human universe.

From the colony system Huygens, the crew of the Giant Science Vessel Impossible Possible World have set out to investigate a starline that flared up with an unprecedented flux of Alderson force before subsiding. Arriving, the Impossible discovers the sparkling debris of a recent nova - and -

"ALIENS!"

Every head swung toward the Sensory console. But after that one cryptic outburst, the Lady Sensory didn't even look up from her console: her fingers were frantically twitching commands.

There was a strange moment of silence in the Command Conference while every listener thought the same two thoughts in rapid succession:

Is she nuts? You can't just say "Aliens!", leave it at that, and expect everyone to believe you. Extraordinary claims require extraordinary evidence -

And then,

They came to look at the nova too!

In a situation like this, it befalls the Conference Chair to speak first.

"What? SHIT!" shouted Akon, who didn't realize until later that his words would be inscribed for all time in the annals of history. Akon swung around and looked frantically at the main display of the Command Conference. "Where are they?"

The Lady Sensory looked up from her console, fingers still twitching. "I - I don't know, I just picked up an incoming high-frequency signal - they're sending us enormous amounts of data, petabytes, I had to clear long-term memory and set up an automatic pipe or risk losing the whole -"

"Found them!" shouted the Lord Programmer. "I searched through our Greater Archive and turned up a program to look for anomalous energy sources near local starlines. It's from way back from the first days of exploration, but I managed to find an emulation program for -"

"Just show it!" Akon took a deep breath, trying to calm himself.

The main display swiftly scanned across fiery space and settled on... a set of windows into fire, the fire of space shattered by the nova, but then shattered again into triangular shards.

It took Akon a moment to realize that he was looking at an icosahedron of perfect mirrors.

Huh, thought Akon, they're lower-tech than us. Their own ship, the Impossible, was absorbing the vast quantities of local radiation and dumping it into their Alderson reactor; the mirror-shielding seemed a distinctly inferior solution. Unless that's what they want us to think...

"Deflectors!" shouted the Lord Pilot suddenly. "Should I put up deflectors?"

"Deflectors?" said Akon, startled.

The Pilot spoke very rapidly. "Sir, we use a self-sustaining Alderson reaction to power our starline jumps and our absorbing shields. That same reaction could be used to emit a directed beam that would snuff a similar reaction - the aliens are putting out their own Alderson emissions, they could snuff our absorbers at any time, and the nova ashes would roast us instantly - unless I configure a deflector -"

The Ship's Confessor spoke, then. "Have the aliens put up deflectors of their own?"

Akon's mind seemed to be moving very slowly, and yet the essential thoughts felt, somehow, obvious. "Pilot, set up the deflector program but don't activate it until I give the word. Sensory, drop everything else and tell me whether the aliens have put up their own deflectors."

Sensory looked up. Her fingers twitched only briefly through a few short commands. Then, "No," she said.

"Then I think," Akon said, though his spine felt frozen solid, "that we should not be the first to put this interaction on a... combative footing. The aliens have made a gesture of goodwill by leaving themselves vulnerable. We must reciprocate." Surely, no species would advance far enough to colonize space without understanding the logic of the Prisoner's Dilemma...

"You assume too much," said the Ship's Confessor. "They are aliens."

"Not much goodwill," said the Pilot. His fingers were twitching, not commands, but almost-commands, subvocal thoughts. "The aliens' Alderson reaction is weaker than ours by an order of magnitude. We could break any shield they could put up. Unless they struck first. If they leave their deflectors down, they lose nothing, but they invite us to leave our own down -"

"If they were going to strike first," Akon said, "they could have struck before we even knew they were here. But instead they spoke." Surely, oh surely, they understand the Prisoner's Dilemma.

"Maybe they hope to gain information and then kill us," said the Pilot. "We have technology they want. That enormous message - the only way we could send them an equivalent amount of data would be by dumping our entire Local Archive. They may be hoping that we feel the emotional need to, as you put it, reciprocate -"

"Hold on," said the Lord Programmer suddenly. "I may have managed to translate their language."

You could have heard a pin dropping from ten lightyears away.

The Lord Programmer smiled, ever so slightly. "You see, that enormous dump of data they sent us - I think that was their Local Archive, or equivalent. A sizable part of their Net, anyway. Their text, image, and holo formats are utterly straightforward - either they don't bother compressing anything, or they decompressed it all for us before they sent it. And here's the thing: back in the Dawn era, when there were multiple human languages, there was this notion that people had of statistical language translation. Now, the classic method used a known corpus of human-translated text. But there were successor methods that tried to extend the translation further, by generating semantic skeletons and trying to map the skeletons themselves onto one another. And there are also ways of automatically looking for similarity between images or holos. Believe it or not, there was a program already in the Archive for trying to find points of linkage between an alien corpus and a human corpus, and then working out from there to map semantic skeletons... and it runs quickly, since it's designed to work on older computer systems. So I ran the program, it finished, and it's claiming that it can translate the alien language with 70% confidence. Could be a total bug, of course. But the aliens sent a second message that followed their main data dump - short, looks like text-only. Should I run the translator on that, and put the results on the main display?"

Akon stared at the Lord Programmer, absorbing this, and finally said, "Yes."

"All right," said the Lord Programmer, "here goes machine learning," and his fingers twitched once.

Over the icosahedron of fractured fire, translucent letters appeared:

THIS VESSEL IS THE OPTIMISM OF THE CENTER OF THE VESSEL PERSON

YOU HAVE NOT KICKED US

THEREFORE YOU EAT BABIES

WHAT IS OURS IS YOURS, WHAT IS YOURS IS OURS

"Stop that laughing," Akon said absentmindedly, "it's distracting." The Conference Chair pinched the bridge of his nose. "All right. That doesn't seem completely random. The first line... is them identifying their ship, maybe. Then the second line says that we haven't opened fire on them, or that they won't open fire on us - something like that. The third line, I have absolutely no idea. The fourth... is offering some kind of reciprocal trade -" Akon stopped then. So did the laughter.

"Would you like to send a return message?" said the Lord Programmer.

Everyone looked at him. Then everyone looked at Akon.

Akon thought about that very carefully. Total silence for a lengthy period of time might not be construed as friendly by a race that had just talked at them for petabytes.

"All right," Akon said. He cleared his throat. "We are still trying to understand your language. We do not understand well. We are trying to translate. We may not translate correctly. These words may not say what we want them to say. Please do not be offended. This is the research vessel named quote Impossible Possible World unquote. We are pleased to meet you. We will assemble data for transmission to you, but do not have it ready." Akon paused. "Send them that. If you can make your program translate it three different plausible ways, do that too - it may make it clearer that we're working from an automatic program."

The Lord Programmer twitched a few more times, then spoke to the Lady Sensory. "Ready."

"Are you really sure this is a good idea?" said Sensory doubtfully.

Akon sighed. "No. Send the message."

For twenty seconds after, there was silence. Then new words appeared on the display:

WE ARE GLAD TO SEE YOU CANNOT BE DONE

YOU SPEAK LIKE BABY CRUNCH CRUNCH

WITH BIG ANGELIC POWERS

WE WISH TO SUBSCRIBE TO YOUR NEWSLETTER

"All right," Akon said, after a while. It seemed, on the whole, a positive response. "I expect a lot of people are eager to look at the alien corpus. But I also need volunteers to hunt for texts and holo files in our own Archive. Which don't betray the engineering principles behind any technology we've had for less than, say," Akon thought about the mirror shielding and what it implied, "a hundred years. Just showing that it can be done... we won't try to avoid that, but don't give away the science..."

A day later, the atmosphere at the Command Conference was considerably more tense.

Bewilderment. Horror. Fear. Numbness. Refusal. And in the distant background, slowly simmering, a dangerous edge of rising righteous fury.

"First of all," Akon said. "First of all. Does anyone have any plausible hypothesis, any reasonable interpretation of what we know, under which the aliens do not eat their own children?"

"There is always the possibility of misunderstanding," said the former Lady Psychologist, who was now, suddenly and abruptly, the lead Xenopsychologist of the ship, and therefore of humankind. "But unless the entire corpus they sent us is a fiction... no."

The alien holos showed tall crystalline insectile creatures, all flat planes and intersecting angles and prismatic refractions, propelling themselves over a field of sharp rocks: the aliens moved like hopping on pogo sticks, bouncing off the ground using projecting limbs that sank into their bodies and then rebounded. There was a cold beauty to the aliens' crystal bodies and their twisting rotating motions, like screensavers taking on sentient form.

And the aliens bounded over the sharp rocks toward tiny fleeing figures like delicate spherical snowflakes, and grabbed them with pincers, and put them in their mouths. It was a central theme in holo after holo.

The alien brain was much smaller and denser than a human's. The alien children, though their bodies were tiny, had full-sized brains. They could talk. They protested as they were eaten, in the flickering internal lights that the aliens used to communicate. They screamed as they vanished into the adult aliens' maws.

Babies, then, had been a mistranslation: Preteens would have been more accurate.

Still, everyone was calling the aliens Babyeaters.

The children were sentient at the age they were consumed. The text portions of the corpus were very clear about that. It was part of the great, the noble, the most holy sacrifice. And the children were loved: this was part of the central truth of life, that parents could overcome their love and engage in the terrible winnowing. A parent might spawn a hundred children, and only one in a hundred could survive - for otherwise they would die later, of starvation...

When the Babyeaters had come into their power as a technological species, they could have chosen to modify themselves - to prevent all births but one.

But this they did not choose to do.

For that terrible winnowing was the central truth of life, after all.

The one now called Xenopsychologist had arrived to the Huygens system with the first colonization vessel. Since then she had spent over one hundred years practicing the profession of psychology, earning the rare title of Lady. (Most people got fed up and switched careers after no more than fifty, whatever their first intentions.) Now, after all that time, she was simply the Xenopsychologist, no longer a Lady of her profession. Being the first and only Xenopsychologist made no difference; the hundred-year rule for true expertise was not a rule that anyone could suspend. If she was the foremost Xenopsychologist of humankind, then also she was the least, the most foolish and the most ignorant. She was only an apprentice Xenopsychologist, no matter that there were no masters anywhere. In theory, her social status should have been too low to be seated at the Conference Table. In theory.

The Xenopsychologist was two hundred and fifty years old. She looked much older, now, as as she spoke. "In terms of evolutionary psychology... I think I understand what happened. The ancestors of the Babyeaters were a species that gave birth to hundreds of offspring in a spawning season, like Terrestrial fish; what we call r-strategy reproduction. But the ancestral Babyeaters discovered... crystal-tending, a kind of agriculture... long before humans did. They were around as smart as chimpanzees, when they started farming. The adults federated into tribes so they could guard territories and tend crystal. They adapted to pen up their offspring, to keep them around in herds so they could feed them. But they couldn't produce enough crystal for all the children.

"It's a truism in evolutionary biology that group selection can't work among non-relatives. The exception is if there are enforcement mechanisms, punishment for defectors - then there's no individual advantage to cheating, because you get slapped down. That's what happened with the Babyeaters. They didn't restrain their individual reproduction because the more children they put in the tribal pen, the more children of theirs were likely to survive. But the total production of offspring from the tribal pen was greater, if the children were winnowed down, and the survivors got more individual resources and attention afterward. That was how their species began to shift toward a k-strategy, an individual survival strategy. That was the beginning of their culture.

"And anyone who tried to cheat, to hide away a child, or even go easier on their own children during the winnowing - well, the Babyeaters treated the merciful parents the same way that human tribes treat their traitors.

"They developed psychological adaptations for enforcing that, their first great group norm. And those psychological adaptations, those emotions, were reused over the course of their evolution, as the Babyeaters began to adapt to their more complex societies. Honor, friendship, the good of our tribe - the Babyeaters acquired many of the same moral adaptations as humans, but their brains reused the emotional circuitry of infanticide to do it.

"The Babyeater word for good means, literally, to eat children."

The Xenopsychologist paused there, taking a sip of water. Pale faces looked back at her from around the table.

The Lady Sensory spoke up. "I don't suppose... we could convince them they were wrong about that?"

The Ship's Confessor was robed and hooded in silver, indicating that he was there formally as a guardian of sanity. His voice was gentle, though, as he spoke: "I don't believe that's how it works."

"Even if you could persuade them, it might not be a good idea," said the Xenopsychologist. "If you convinced the Babyeaters to see it our way - that they had committed a wrong of that magnitude - there isn't anything in the universe that could stop them from hunting down and exterminating themselves. They don't have a concept of forgiveness; their only notion of why someone might go easy on a transgressor, is to spare an ally, or use them as a puppet, or being too lazy or cowardly to carry out the vengeance. The word for wrong is the same symbol as mercy, you see." The Xenopsychologist shook her head. "Punishment of non-punishers is very much a way of life, with them. A Manichaean, dualistic view of reality. They may have literally believed that we ate babies, at first, just because we didn't open fire on them."

Akon frowned. "Do you really think so? Wouldn't that make them... well, a bit unimaginative?"

The Ship's Master of Fandom was there; he spoke up. "I've been trying to read Babyeater literature," he said. "It's not easy, what with all the translation difficulties," and he sent a frown at the Lord Programmer, who returned it. "In one sense, we're lucky enough that the Babyeaters have a concept of fiction, let alone science fiction -"

"Lucky?" said the Lord Pilot. "You've got to have an imagination to make it to the stars. The sort of species that wouldn't invent science fiction, probably wouldn't even invent the wheel -"

"But," interrupted the Master, "just as most of their science fiction deals with crystalline entities - the closest they come to postulating human anatomy, in any of the stories I've read, was a sort of giant sentient floppy sponge - so too, nearly all of the aliens their explorers meet, eat their own children. I doubt the authors spent much time questioning the assumption; they didn't want anything so alien that their readers couldn't empathize. The purpose of storytelling is to stimulate the moral instincts, which is why all stories are fundamentally about personal sacrifice and loss - that's their theory of literature. Though you can find stories where the wise, benevolent elder aliens explain how the need to control tribal population is the great selective transition, and how no species can possibly evolve sentience and cooperation without eating babies, and even if they did, they would war among themselves and destroy themselves."

"Hm," said the Xenopsychologist. "The Babyeaters might not be too far wrong - stop staring at me like that, I don't mean it that way. I'm just saying, the Babyeater civilization didn't have all that many wars. In fact, they didn't have any wars at all after they finished adopting the scientific method. It was the great watershed moment in their history - the notion of a reasonable mistake, that you didn't have to kill all the adherents of a mistaken hypothesis. Not because you were forgiving them, but because they'd made the mistake by reasoning on insufficient data, rather than any inherent flaw. Up until then, all wars were wars of total extermination - but afterward, the theory was that if a large group of people could all do something wrong, it was probably a reasonable mistake. Their conceptualization of probability theory - of a formally correct way of manipulating uncertainty - was followed by the dawn of their world peace."

"But then -" said the Lady Sensory.

"Of course," added the Xenopsychologist, "anyone who departs from the group norm due to an actual inherent flaw still has to be destroyed. And not everyone agreed at first that the scientific method was moral - it does seem to have been highly counterintuitive to them - so their last war was the one where the science-users killed off all the nonscientists. After that, it was world peace."

"Oh," said the Lady Sensory softly.

"Yes," the Xenopsychologist said, "after that, all the Babyeaters banded together as a single super-group that only needed to execute individual heretics. They now have a strong cultural taboo against wars between tribes."

"Unfortunately," said the Master of Fandom, "that taboo doesn't let us off the hook. You can also find science fiction stories - though they're much rarer - where the Babyeaters and the aliens don't immediately join together into a greater society. Stories of horrible monsters who don't eat their children. Monsters who multiply like bacteria, war among themselves like rats, hate all art and beauty, and destroy everything in their pathway. Monsters who have to be exterminated down to the last strand of their DNA - er, last nucleating crystal."

Akon spoke, then. "I accept full responsibility," said the Conference Chair, "for the decision to send the Babyeaters the texts and holos we did. But the fact remains that they have more than enough information about us to infer that we don't eat our children. They may be able to guess how we would see them. And they haven't sent anything to us, since we began transmitting to them."

"So the question then is - now what?"

To be continued...

Referenced by: Three Worlds Collide (0/8) • Belief in Self-Deception

Original with comments: The Baby-Eating Aliens (1/8)

War and/or Peace (2/8)

Eliezer Yudkowsky, 31 January 2009 08:42AM

(Part 2 of 8 in "Three Worlds Collide")

..."So the question then is - now what?"

The Lord Pilot jumped up, then, his face flushed. "Put up shields.  Now. We don't gain anything by leaving them down. This is madness!"

"No," said the Ship's Confessor in professional tones, "not madness."

The Pilot slammed his fists on the table. "We're all going to die!"

"They're not as technologically advanced as us," Akon said.  "Suppose the Babyeaters do decide that we need to be exterminated.  Suppose they open fire. Suppose they kill us. Suppose they follow the starline we opened and find the Huygens system. Then what?"

The Master nodded. "Even with surprise on their side... no. They can't actually wipe out the human species. Not unless they're a lot smarter than they seem to be, and it looks to me like, on average, they're actually a bit dumber than us." The Master glanced at the Xenopsychologist, who waved her hand in a maybe-gesture.

"But if we leave the ship's shields down," Akon said, "we preserve whatever chance we have of a peaceful resolution to this."

"Peace," said the Lady Sensory, in a peculiar flat tone.

Akon looked at her.

"You want peace with the Babyeaters?"

"Of course -" said Akon, then stopped short.

The Lady Sensory looked around the table. "And the Babyeater children? What about them?"

The Master of Fandom spoke, his voice uncertain. "You can't impose human standards on -"

With a blur of motion and a sharp crack, the Lady Sensory slapped him.

The Ship's Confessor grabbed her arm. "No."

The Lady Sensory stared at the Ship's Confessor.

"No," the Confessor repeated. "No violence. Only argument. Violence doesn't distinguish truth from falsehood, my Lady."

The Lady Sensory slowly lowered her hand, but not her eyes.

"But..." said the Master. "But, my Lady, if they want to be eaten -"

"They don't," said the Xenopsychologist. "Of course they don't.  They run from their parents when the terrible winnowing comes. The Babyeater children aren't emotionally mature - I mean they don't have their adult emotional state yet. Evolution would take care of anyone who wanted to get eaten. And they're still learning, still making mistakes, so they don't yet have the instinct to exterminate violators of the group code. It's a simpler time for them. They play, they explore, they try out new ideas. They're..." and the Xenopsychologist stopped. "Damn," she said, and turned her head away from the table, covering her face with her hands. "Excuse me." Her voice was unsteady. "They're a lot like human children, really."

"And if they were human children," said the Lady Sensory into the silence, "do you think that, just because the Babyeater species wanted to eat human children, that would make it right for them to do it?"

"No," said the Lord Pilot.

"Then what difference does it make?" said the Lady Sensory.

"No difference at all," said the Lord Pilot.

Akon looked back and forth between the two of them, and saw what was coming, and somehow couldn't speak.

"We have to save them," said the Lady Sensory. "We have to stop this. No matter what it takes. We can't let this go on."

Couldn't say that one word -

The Lord Pilot nodded. "Destroy their ship. Preserve our advantage of surprise. Go back, tell the world, create an overwhelming human army... and pour into the Babyeater starline network. And rescue the children."

"No," Akon said.

No?

"I know," said the Lord Pilot. "A lot of Babyeaters will die at first, but they're killing ten times more children than their whole adult population, every year -"

"And then what?" said the Master of Fandom. "What happens when the children grow up?"

The Lord Pilot fell silent.

The Master of Fandom completed the question. "Are you going to wipe out their whole race, because their existence is too horrible to be allowed to go on? I read their stories, and I didn't understand them, but -" The Master of Fandom swallowed. "They're not... evil. Don't you understand? They're not. Are you going to punish me, because I don't want to punish them?"

"We could..." said the Lord Pilot. "Um. We could modify their genes so that they only gave birth to a single child at a time."

"No," said the Xenopsychologist. "They would grow up loathing themselves for being unable to eat babies. Horrors in their own eyes.  It would be kinder just to kill them."

"Stop," said Akon. His voice wasn't strong, wasn't loud, but everyone in the room looked at him. "Stop. We are not going to fire on their ship."

"Why not?" said the Lord Pilot. "They -"

"They haven't raised shields," said Akon.

"Because they know it won't make a difference!" shouted the Pilot.

"They didn't fire on us!" shouted Akon. Then he stopped, lowered his voice. "They didn't fire on us. Even after they knew that we didn't eat babies. I am not going to fire on them. I refuse to do it."

"You think they're innocent?" demanded the Lady Sensory. "What if it was human children that were being eaten?"

Akon stared out a viewscreen, showing in subdued fires a computer-generated graphic of the nova debris. He just felt exhausted, now. "I never understood the Prisoner's Dilemma until this day. Do you cooperate when you really do want the highest payoff? When it doesn't even seem fair for both of you to cooperate? When it seems right to defect even if the other player doesn't? That's the payoff matrix of the true Prisoner's Dilemma. But all the rest of the logic - everything about what happens if you both think that way, and both defect - is the same. Do we want to live in a universe of cooperation or defection?"

"But -" said the Lord Pilot.

"They know," Akon said, "that they can't wipe us out. And they can guess what we could do to them. Their choice isn't to fire on us and try to invade afterward! Their choice is to fire on us and run from this star system, hoping that no other ships follow.  It's their whole species at stake, against just this one ship. And they still haven't fired."

"They won't fire on us," said the Xenopsychologist, "until they decide that we've defected from the norm. It would go against their sense of... honor, I could call it, but it's much stronger than the human version -"

"No," Akon said. "Not that much stronger." He looked around, in the silence. "The Babyeater society has been at peace for centuries. So too with human society. Do you want to fire the opening shot that brings war back into the universe? Send us back to the darkness-before-dawn that we only know from reading history books, because the holos are too horrible to watch? Are you really going to press the button, knowing that?"

The Lord Pilot took a deep breath. "I will. You will not remain commander of the Impossible, my lord, if the greater conference votes no confidence against you. And they will, my lord, for the sake of the children."

"What," said the Master, "are you going to do with the children?"

"We, um, have to do something," said the Ship's Engineer, speaking up for the first time. "I've been, um, looking into what Babyeater science knows about their brain mechanisms. It's really quite fascinating, they mix electrical and mechanical interactions, not the same way our own brain pumps ions, but -"

"Get to the point," said Akon. "Immediately."

"The children don't die right away," said the Engineer. "The brain is this nugget of hard crystal, that's really resistant to, um, the digestive mechanisms, much more so than the rest of the body. So the child's brain is in, um, probably quite a lot of pain, since the whole body has been amputated, and in a state of sensory deprivation, and then the processing slowly gets degraded, and I think the whole process gets completed about a month after -"

The Lady Sensory threw up. A few seconds later, so did the Xenopsychologist and the Master.

"If human society permits this to go on," said the Lord Pilot, his voice very soft, "I will resign from human society, and I will have friends, and we will visit the Babyeater starline network with an army. You'll have to kill me to stop me."

"And me," said the Lady Sensory through tears.

Akon rose from his chair, and leaned forward; a dominating move that he had learned in classrooms, very long ago when he was first studying to be an Administrator. But most in humanity's promotion-conscious society would not risk direct defiance of an Administrator. In a hundred years he'd never had his authority really tested, until now...  "I will not permit you to fire on the alien ship. Humanity will not be first to defect in the Prisoner's Dilemma."

The Lord Pilot stood up, and Akon realized, with a sudden jolt, that the Pilot was four inches taller; the thought had never occurred to him before. The Pilot didn't lean forward, not knowing the trick, or not caring. The Pilot's eyes were narrow, surrounding facial muscles tensed and tight.

"Get out of my way," said the Lord Pilot.

Akon opened his mouth, but no words came out.

"It is time," said the Lord Pilot, "to see this calamity to its end." Spoken in Archaic English: the words uttered by Thomas Clarkson in 1785, at the beginning of the end of slavery. "I have set my will against this disaster; I will break it, or it will break me." Ira Howard in 2014. "I will not share my universe with this shadow," and that was the Lord Pilot, in an anger hotter than the nova's ashes.  "Help me if you will, or step aside if you lack decisiveness; but do not make yourself my obstacle, or I will burn you down, and any that stand with you -"

"HOLD."

Every head in the room jerked toward the source of the voice. Akon had been an Administrator for a hundred years, and a Lord Administrator for twenty. He had studied all the classic texts, and watched holos of famous crisis situations; nearly all the accumulated knowledge of the Administrative Field was at his beck and call; and he'd never dreamed that a word could be spoken with such absolute force.

The Ship's Confessor lowered his voice. "My Lord Pilot. I will not permit you to declare your crusade, when you have not said what you are crusading for. It is not enough to say that you do not like the way things are. You must say how you will change them, and to what. You must think all the way to your end. Will you wipe out the Babyeater race entirely? Keep their remnants under human rule forever, in despair under our law? You have not even faced your hard choices, only congratulated yourself on demanding that something be done. I judge that a violation of sanity, my lord."

The Lord Pilot stood rigid. "What -" his voice broke. "What do you suggest we do?"

"Sit down," said the Ship's Confessor, "keep thinking. My Lord Pilot, my Lady Sensory, you are premature. It is too early for humanity to divide over this issue, when we have known about it for less than twenty-four hours.  Some rules do not change, whether it is money at stake, or the fate of an intelligent species. We should only, at this stage, be discussing the issue in all its aspects, as thoroughly as possible; we should not even be placing solutions on the table, as yet, to polarize us into camps. You know that, my lords, my ladies, and it does not change."

"And after that?" said the Master of Fandom suddenly. "Then it's okay to split humanity? You wouldn't object?"

The featureless blur concealed within the Confessor's Hood turned to face the Master, and spoke; and those present thought they heard a grim smile, in that voice. "Oh," said the Confessor, "that would be interfering in politics. I am charged with guarding sanity, not morality. If you want to stay together, do not split. If you want peace, do not start wars. If you want to avoid genocide, do not wipe out an alien species. But if these are not your highest values, then you may well end up sacrificing them. What you are willing to trade off, may end up traded away - be you warned!  But if that is acceptable to you, then so be it. The Order of Silent Confessors exists in the hope that, so long as humanity is sane, it can make choices in accordance with its true desires. Thus there is our Order dedicated only to that, and sworn not to interfere in politics. So you will spend more time discussing this scenario, my lords, my ladies, and only then generate solutions. And then... you will decide."

"Excuse me," said the Lady Sensory. The Lord Pilot made to speak, and Sensory raised her voice. "Excuse me, my lords. The alien ship has just sent us a new transmission. Two megabytes of text."

"Translate and publish," ordered Akon.

They all glanced down and aside, waiting for the file to come up.

It began:

THE UTTERMOST ABYSS OF JUSTIFICATION
A HYMN OF LOGIC
PURE LIKE STONES AND SACRIFICE
FOR STRUGGLES OF THE YOUNG SLIDING DOWN YOUR THROAT-

Akon looked away, wincing. He hadn't tried to read much of the alien corpus, and hadn't gotten the knack of reading the "translations" by that damned program.

"Would someone," Akon said, "please tell me - tell the conference - what this says?

There was a long, stretched moment of silence.

Then the Xenopsychologist made a muffled noise that could have been a bark of incredulity, or just a sad laugh. "Stars beyond," said the Xenopsychologist, "they're trying to persuade us to eat our own children."

"Using," said the Lord Programmer, "what they assert to be arguments from universal principles, rather than appeals to mere instincts that might differ from star to star."

"Such as what, exactly?" said the Ship's Confessor.

Akon gave the Confessor an odd look, then quickly glanced away, lest the Confessor catch him at it. No, the Confessor couldn't be carefully maintaining an open mind about that. It was just curiosity over what particular failures of reasoning the aliens might exhibit.

"Let me search," said the Lord Programmer. He was silent for a time. "Ah, here's an example. They point out that by producing many offspring, and winnowing among them, they apply greater selection pressures to their children than we do. So if we started producing hundreds of babies per couple and then eating almost all of them - I do emphasize that this is their suggestion, not mine - evolution would proceed faster for us, and we would survive longer in the universe.  Evolution and survival are universals, so the argument should convince anyone." He gave a sad chuckle. "Anyone here feel convinced?"

"Out of curiosity," said the Lord Pilot, "have they ever tried to produce even more babies - say, thousands instead of hundreds - so they could speed up their evolution even more?"

"It ought to be easily within their current capabilities of bioengineering," said the Xenopsychologist, "and yet they haven't done it. Still, I don't think we should make the suggestion.""

"Agreed," said Akon.

"But humanity uses gamete selection," said the Lady Sensory. "We aren't evolving any slower. If anything, choosing among millions of sperm and hundreds of eggs gives us much stronger selection pressures."

The Xenopsychologist furrowed her brow. "I'm not sure we sent them that information in so many words... or they may have just not gotten that far into what we sent them..."

"Um, it wouldn't be trivial for them to understand," said the Ship's Engineer. "They don't have separate DNA and proteins, just crystal patterns tiling themselves. The two parents intertwine and stay that way for, um, days, nucleating portions of supercooled liquid from their own bodies to construct the babies. The whole, um, baby, is constructed together by both parents. They don't have separate gametes they could select on."

"But," said the Lady Sensory, "couldn't we maybe convince them, to work out some equivalent of gamete selection and try that instead -"

"My lady," said the Xenopsychologist. Her voice, now, was somewhat exasperated. "They aren't really doing this for the sake of evolution. They were eating babies millions of years before they knew what evolution was."

"Huh, this is interesting," said the Lord Programmer. "There's another section here where they construct their arguments using appeals to historical human authorities."

Akon raised his eyebrows. "And who, exactly, do they quote in support?"

"Hold on," said the Lord Programmer. "This has been run through the translator twice, English to Babyeater to English, so I need to write a program to retrieve the original text..." He was silent a few moments. "I see. The argument starts by pointing out how eating your children is proof of sacrifice and loyalty to the tribe, then they quote human authorities on the virtue of sacrifice and loyalty. And ancient environmentalist arguments about population control, plus... oh, dear. I don't think they've realized that Adolf Hitler is a bad guy."

"They wouldn't," said the Xenopsychologist. "Humans put Hitler in charge of a country, so we must have considered him a preeminent legalist of his age. And it wouldn't occur to the Babyeaters that Adolf Hitler might be regarded by humans as a bad guy just because he turned segments of his society into lampshades - they have a custom against that nowadays, but they don't really see it as evil.  If Hitler thought that gays had defected against the norm, and tried to exterminate them, that looks to a Babyeater like an honest mistake -"  The Xenopsychologist looked around the table. "All right, I'll stop there. But the Babyeaters don't look back on their history and see obvious villains in positions of power - certainly not after the dawn of science. Any politician who got to the point of being labeled "bad" would be killed and eaten. The Babyeaters don't seem to have had humanity's coordination problems. Or they're just more rational voters. Take your pick."

Akon was resting his head in his hands. "You know," Akon said, "I thought about composing a message like this to the Babyeaters. It was a stupid thought, but I kept turning it over in my mind. Trying to think about how I might persuade them that eating babies was... not a good thing."

The Xenopsychologist grimaced. "The aliens seem to be even more given to rationalization than we are - which is maybe why their society isn't so rigid as to actually fall apart - but I don't think you could twist them far enough around to believe that eating babies was not a babyeating thing."

"And by the same token," Akon said, "I don't think they're particularly likely to persuade us that eating babies is good." He sighed. "Should we just mark the message as spam?"

"One of us should read it, at least," said the Ship's Confessor. "They composed their argument honestly and in all good will. Humanity also has epistemic standards of honor to uphold."

"Yes," said the Master. "I don't quite understand the Babyeater standards of literature, my lord, but I can tell that this text conforms to their style of... not exactly poetry, but... they tried to make it aesthetic as well as persuasive." The Master's eyes flickered, back and forth. "I think they even made some parts constant in the total number of light pulses per argumentative unit, like human prosody, hoping that our translator would turn it into a human poem.  And... as near as I can judge such things, this took a lot of effort. I wouldn't be surprised to find that everyone on that ship was staying up all night working on it."

"Babyeaters don't sleep," said the Engineer sotto vocce.

"Anyway," said the Master. "If we don't fire on the alien ship - I mean, if this work is ever carried back to the Babyeater civilization - I suspect the aliens will consider this one of their great historical works of literature, like Hamlet or Fate/stay night -"

The Lady Sensory cleared her throat. She was pale, and trembling.

With a sudden black premonition of doom like a training session in Unrestrained Pessimism, Akon guessed what she would say.

The Lady Sensory said, in an unsteady voice, "My lords, a third ship has jumped into this system. Not Babyeater, not human."

To be continued...

Referenced by: Three Worlds Collide (0/8) • The Baby-Eating Aliens (1/8)

Original with comments: War and/or Peace (2/8)

The Super Happy People (3/8)

Eliezer Yudkowsky, 01 February 2009 08:18AM

(Part 3 of 8 in "Three Worlds Collide")

...The Lady Sensory said, in an unsteady voice, "My lords, a third ship has jumped into this system. Not Babyeater, not human."

The holo showed a triangle marked with three glowing dots, the human ship and the Babyeater ship and the newcomers. Then the holo zoomed in, to show -

- the most grotesque spaceship that Akon had ever seen, like a blob festooned with tentacles festooned with acne festooned with small hairs. Slowly, the tentacles of the ship waved, as if in a gentle breeze; and the acne on the tentacles pulsated, as if preparing to burst. It was a fractal of ugliness, disgusting at every level of self-similarity.

"Do the aliens have deflectors up?" said Akon.

"My lord," said Lady Sensory, "they don't have any shields raised. The nova ashes' radiation doesn't seem to bother them. Whatever material their ship is made from, it's just taking the beating."

A silence fell around the table.

"All right," said the Lord Programmer, "that's impressive."

The Lady Sensory jerked, like someone had just slapped her. "We - we just got a signal from them in human-standard format, content encoding marked as Modern English text, followed by a holo -"

"What?" said Akon. "We haven't transmitted anything to them, how could they possibly -"

"Um," said the Ship's Engineer. "What if these aliens really do have, um, 'big angelic powers'?"

"No," said the Ship's Confessor. His hood tilted slightly, as if in wry humor. "It is only history repeating itself."

"History repeating itself?" said the Master of Fandom. "You mean that the ship is from an alternate Everett branch of Earth, or that they somehow independently developed ship-to-ship communication protocols exactly similar to our -"

"No, you dolt," said the Lord Programmer, "he means that the Babyeaters sent the new aliens a massive data dump, just like they sent us. Only this time, the Babyeater data dump included all the data that we sent the Babyeaters. Then the new aliens ran an automatic translation program, like the one we used."

"You gave it away," said the Confessor. There was a slight laugh in his voice. "You should have let them figure it out on their own. One so rarely encounters the apparently supernatural, these days."

Akon shook his head, "Confessor, we don't have time for - never mind. Sensory, show the text message."

The Lady Sensory twitched a finger and -

HOORAY!

WE ARE SO GLAD TO MEET YOU!

THIS IS THE SHIP "PLAY GAMES FOR LOTS OF FUN"

(OPERATED BY CHARGED PARTICLE FINANCIAL FIRMS)

WE LOVE YOU AND WE WANT YOU TO BE SUPER HAPPY.

WOULD YOU LIKE TO HAVE SEX?

Slowly, elaborately, Akon's head dropped to the table with a dull thud. "Why couldn't we have been alone in the universe?"

"No, wait," said the Xenopsychologist, "this makes sense."

The Master of Fandom nodded. "Seems quite straightforward."

"Do enlighten," came a muffled tone from where Akon's head rested on the table.

The Xenopsychologist shrugged. "Evolutionarily speaking, reproduction is probably the single best guess for an activity that an evolved intelligence would find pleasurable. When you look at it from that perspective, my lords, my lady, their message makes perfect sense - it's a universal friendly greeting, like the Pioneer engraving."

Akon didn't raise his head. "I wonder what these aliens do," he said through his shielding arms, "molest kittens?"

"My lord..." said the Ship's Confessor. Gentle the tone, but the meaning was very clear.

Akon sighed and straightened up. "You said their message included a holo, right? Let's see it."

The main screen turned on.

There was a moment of silence, and then a strange liquid sound as, in unison, everyone around the table gasped in shock, even the Ship's Confessor.

For a time after that, no one spoke. They were just... watching.

"Wow," said the Lady Sensory finally. "That's actually... kind of... hot."

Akon tore his eyes away from the writhing human female form, the writhing human male form, and the writhing alien tentacles. "But..." Akon said. "But why is she pregnant?"

"A better question," said the Lord Programmer, "would be, why are the two of them reciting multiplication tables?" He glanced around. "What, none of you can read lips?"

"Um..." said the Xenopsychologist. "Okay, I've got to admit, I can't even begin to imagine why -"

Then there was a uniform "Ewww..." from around the room.

"Oh, dear," said the Xenopsychologist. "Oh, dear, I don't think they understood that part at all."

Akon made a cutting gesture, and the holo switched off.

"Someone should view the rest of it," said the Ship's Confessor. "It might contain important information."

Akon flipped a hand. "I don't think we'll run short of volunteers to watch disgusting alien pornography. Just post it to the ship's 4chan, and check after a few hours to see if anything was modded up to +5 Insightful."

"These aliens," said the Master of Fandom slowly, "composed that pornography within... seconds, it must have been. We couldn't have done that automatically, could we?"

The Lord Programmer frowned. "No. I don't, um, think so. From a corpus of alien pornography, automatically generate a holo they would find interesting? Um. It's not a problem that I think anyone's tried to solve yet, and they sure didn't get it perfect the first time, but... no."

"How large an angelic power does that imply?"

The Lord Programmer traded glances with the Master. "Big," the Lord Programmer said finally. "Maybe even epic."

"Or they think on a much faster timescale," said the Confessor softly. "There is no law of the universe that their neurons must run at 100Hz."

"My lords," said the Lady Sensory, "we're getting another message; holo with sound, this time. It's marked as a real-time communication, my lords."

Akon swallowed, and his fingers automatically straightened the hood of his formal sweater. Would the aliens be able to tell if his clothes were sloppy? He was suddenly very aware that he hadn't checked his lipstick in three hours. But it wouldn't do to keep the visitors waiting... "All right. Open a channel to them, transmitting only myself."

The holo that appeared did nothing to assuage his insecurities. The man that appeared was perfectly dressed, utterly perfectly dressed, in business casual more intimidating than any formality: crushing superiority without the appearance of effort. The face was the same way, overwhelmingly handsome without the excuse of makeup; the fashionable slit vest exposed pectoral muscles that seemed optimally sculpted without the bulk that comes of exercise -

"Superstimulus!" exclaimed the Ship's Confessor, a sharp warning.

Akon blinked, shrugging off the fog. Of course the aliens couldn't possibly really look like that. A holo, only an overoptimized holo. That was a lesson everyone (every human?) learned before puberty, not to let reality seem diminished by fiction. As the proverb went, It's bad enough comparing yourself to Isaac Newton without comparing yourself to Kimball Kinnison.

"Greetings in the name of humanity," said Akon. "I am Lord Anamaferus Akon, Conference Chair of the Giant Science Vessel Impossible Possible World. We -" come in peace didn't seem appropriate with a Babyeater war under discussion, and many other polite pleasantries, like pleased to meet you, suddenly seemed too much like promises and lies, "- didn't quite understand your last message."

"Our apologies," said the perfect figure on screen. "You may call me Big Fucking Edward; as for our species..." The figure tilted a head in thought. "This translation program is not fully stable; even if I said our proper species-name, who knows how it would come out. I would not wish my kind to forever bear an unaesthetic nickname on account of a translation error."

Akon nodded. "I understand, Big Fucking Edward."

"Your true language is a format inconceivable to us," said the perfect holo. "But we do apologize for any untranslatable 1 you may have experienced on account of our welcome transmission; it was automatically generated, before any of us had a chance to apprehend your sexuality. We do apologize, I say; but who would ever have thought that a species would evolve to find reproduction a painful experience? For us, childbirth is the greatest pleasure we know; to be prolonged, not hurried."

"Oh," said the Lady Sensory in a tone of sudden enlightenment, "that's why the tentacles were pushing the baby back into -"

Out of sight of the visual frame, Akon gestured with his hand for Sensory to shut up. Akon leaned forward. "The visual you're currently sending us is, of course, not real. What do you actually look like? - if the request does not offend."

The perfect false man furrowed a brow, puzzled. "I don't understand. You would not be able to apprehend any communicative cues."

"I would still like to see," Akon said. "I am not sure how to explain it, except that - truth matters to us."

The too-beautiful man vanished, and in his place -

Mad brilliant colors, insane hues that for a moment defeated his vision. Then his mind saw shapes, but not meaning. In utter silence, huge blobs writhed around supporting bars. Extrusions protruded fluidly and interpenetrated -

Writhing, twisting, shuddering, pulsating -

And then the false man reappeared.

Akon fought to keep his face from showing distress, but a prickling of sweat appeared on his forehead. There'd been something jarring about the blobs, even the stable background behind them. Like looking at an optical illusion designed by sadists.

And - those were the aliens, or so they claimed -

"I have a question," said the false man. "I apologize if it causes any distress, but I must know if what our scientists say is correct. Has your kind really evolved separate information-processing mechanisms for deoxyribose nucleic acid versus electrochemical transmission of synaptic spikes?"

Akon blinked. Out of the corner of his eye, he saw figures trading cautious glances around the table. Akon wasn't sure where this question was leading, but, given that the aliens had already understood enough to ask, it probably wasn't safe to lie...

"I don't really understand the question's purpose," Akon said. "Our genes are made of deoxyribose nucleic acid. Our brains are made of neurons that transmit impulses through electrical and chemical -"

The fake man's head collapsed to his hands, and he began to bawl like a baby.

Akon's hand signed Help! out of the frame. But the Xenopsychologist shrugged cluelessly.

This was not going well.

The fake man suddenly unfolded his head from his hands. His cheeks were depicted as streaked with tears, but the face itself had stopped crying. "To wait so long," the voice said in a tone of absolute tragedy. "To wait so long, and come so far, only to discover that nowhere among the stars is any trace of love."

"Love?" Akon repeated. "Caring for someone else? Wanting to protect them, to be with them? If that translated correctly, then 'love' is a very important thing to us."

"But!" cried the figure in agony, at a volume that made Akon jump. "But when you have sex, you do not untranslatable 2! A fake, a fake, these are only imitation words -"

"What is 'untranslatable 2'?" Akon said; and then, as the figure once again collapsed in inconsolable weeping, wished he hadn't.

"They asked if our neurons and DNA were separate," said the Ship's Engineer. "So maybe they have only one system. Um... in retrospect, that actually seems like the obvious way for evolution to do it. If you're going to have one kind of information storage for genes, why have an entirely different system for brains? So -"

"They share each other's thoughts when they have sex," the Master of Fandom completed. "Now there's an old dream. And they would develop emotions around that, whole patterns of feeling we don't have ourselves... Huh. I guess we do lack their analogue of love."

"Probably," said the Xenopsychologist quietly, "sex was their only way of speaking to each other from the beginning. From before the dawn of their intelligence. It really does make a lot of sense, evolutionarily. If you're injecting packets of information anyway -"

"Wait a minute," said the Lady Sensory, "then how are they talking to us?"

"Of course," said the Lord Programmer in a tone of sudden enlightenment. "Humanity has always used new communications technologies for pornography. 'The Internet is for porn' - but with them, it must have been the other way around."

Akon blinked. His mind suddenly pictured the blobs, and the tentacles connecting them to each other -

Somewhere on that ship is a blob making love to an avatar that's supposed to represent me. Maybe a whole Command Orgy.

I've just been cyber-raped. No, I'm being cyber-raped right now.

And the aliens had crossed who knew how much space, searching for who knew how long, yearning to speak / make love to other minds - only to find -

The fake man suddenly jerked upright and screamed at a volume that whited-out the speakers in the Command Conference. Everyone jumped; the Master of Fandom let out a small shriek.

What did I do what did I do what did I do -

And then the holo vanished.

Akon gasped for breath and slumped over in his chair. Adrenaline was still running riot through his system, but he felt utterly exhausted. He wanted to release his shape and melt into a puddle, a blob like the wrong shapes he'd seen on screen - no, not like that.

"My lord," the Ship's Confessor said softly. He was now standing alongside, a gentle hand on Akon's shoulder. "My lord, are you all right?"

"Not really," Akon said. His voice, he was proud to note, was only slighly wobbly. "It's too hard, speaking to aliens. They don't think like you do, and you don't know what you're doing wrong."

"I wonder," the Master of Fandom said with artificial lightness, "if they'll call it 'xenofatigue' and forbid anyone to talk to an alien for longer than five minutes."

Akon just nodded.

"We're getting another signal," the Lady Sensory said hesitantly. "Holo with sound, another real-time communication."

"Akon, you don't have to -" said the Master of Fandom.

Akon jerked himself upright, straightened his clothes. "I do have to," he said. "They're aliens, there's no knowing what a delay might... Just put it through."

The first thing the holo showed, in elegant Modern English script, was the message:

The Lady 3rd Kiritsugu
temporary co-chair of the Gameplayer
Language Translator version 3
Cultural Translator version 2

The screen hovered just long enough to be read, then dissipated -

Revealing a pale white lady.

The translator's depiction of the Lady 3rd Kiritsugu was all white and black and grey; not the colorlessness of a greyscale image, but a colored image of a world with little color in it. Skin the color of the palest human skin that could still be called attractive; not snow white, but pale. White hair; blouse and bracelets and long dress all in coordinated shades of grey. That woman could have been called pretty, but there was none of the overstimulating beauty of the fake man who had been shown before.

Her face was styled in the emotion that humans named "serene".

"I and my sisters have now taken command of this vessel," said the pale Lady.

Akon blinked. A mutiny aboard their ship?

And it was back to the alien incomprehensibility, the knife-edged decisions and unpredictable reactions and the deadly fear of screwing up.

"I am sorry if my words offend," Akon said carefully, "but there is something I wish to know."

The Lady 3rd made a slicing gesture with one hand. "You cannot offend me." Her face showed mild insult at the suggestion.

"What has happened aboard your ship, just now?"

The Lady 3rd replied, "The crew are disabled by emotional distress. They have exceeded the bounds of their obligations, and are returning to the ship's Pleasuring Center for reward. In such a situation I and my two sisters, the kiritsugu of this vessel, assume command."

Did I do that? "I did not intend for my words to cause you psychological harm."

"You are not responsible," the Lady 3rd said. "It was the other ones."

"The Babyeaters?" Akon said without thinking.

"Babyeaters," the Lady 3rd repeated. "If that is the name you have given to the third alien species present at this star system, then yes. The crew, apprehending the nature of the Babyeaters' existence, was incapacitated by their share of the children's suffering."

"I see," Akon said. He felt an odd twitch of shame for humanity, that his own kind could learn of the Babyeaters, and continue functioning with only tears.

The Lady 3rd's gaze grew sharp. "What are your intentions regarding the Babyeaters?"

"We haven't decided," Akon said. "We were just discussing it when you arrived, actually."

"What is your current most preferred alternative?" the Lady 3rd instantly fired back.

Akon helplessly shrugged, palms out. "We were just starting the discussion. All the alternatives suggested seemed unacceptable."

"Which seemed least unacceptable? What is your current best candidate?"

Akon shook his head. "We haven't designated any."

The Lady 3rd's face grew stern, with a hint of puzzlement. "You are withholding the information. Why? Do you think it will cast you in an unfavorable light? Then I must take that expectation into account. Further, you must expect me to take that expectation into account, and so you imply that you expect me to underestimate its severity, even after taking this line of reasoning into account."

"Excuse me," the Ship's Confessor said. His tone was mild, but with a hint of urgency. "I believe I should enter this conversation right now."

Akon's hand signed agreement to the Lady Sensory.

At once the Lady 3rd's eyes shifted to where the Confessor stood beside Akon.

"Human beings," said the Ship's Confessor, "cannot designate a 'current best candidate' without psychological consequences. Human rationalists learn to discuss an issue as thoroughly as possible before suggesting any solutions. For humans, solutions are sticky in a way that would require detailed cognitive science to explain. We would not be able to search freely through the solution space, but would be helplessly attracted toward the 'current best' point, once we named it. Also, any endorsement whatever of a solution that has negative moral features, will cause a human to feel shame - and 'best candidate' would feel like an endorsement. To avoid feeling that shame, humans must avoid saying which of two bad alternatives is better than the other."

Ouch, thought Akon, I never realized how embarrassing that sounds until I heard it explained to an alien.

Apparently the alien was having similar thoughts. "So you cannot even tell me which of several alternatives currently seems best, without your minds breaking down? That sounds quite implausible," the Lady 3rd said doubtfully, "for a species capable of building a spaceship."

There was a hint of laughter in the Confessor's voice. "We try to overcome our biases."

The Lady 3rd's gaze grew more intense. "Are you the true decisionmaker of this vessel?"

"I am not," the Confessor said flatly. "I am a Confessor - a human master rationalist; we are sworn to refrain from leadership."

"This meeting will determine the future of all three species," said the Lady 3rd. "If you have superior competence, you should assume control."

Akon's brows furrowed slightly. Somehow he'd never thought about it in those terms.

The Confessor shook his head. "There are reasons beyond my profession why I must not lead. I am too old."

Too old?

Akon put the thought on hold, and looked back at the Lady 3rd. She had said that all the crew were incapacitated, except her and her two sisters who took charge. And she had asked the Confessor if he held true command.

"Are you," Akon asked, "the equivalent of a Confessor for your own kind?"

"Almost certainly not," replied the Lady 3rd, and -

"Almost certainly not," the Confessor said, almost in the same breath.

There was an eerie kind of unison about it.

"I am kiritsugu," said the Lady 3rd. "In the early days of my species there were those who refrained from happiness in order to achieve perfect skill in helping others, using untranslatable 3 to suppress their emotions and acting only on their abstract knowledge of goals. These were forcibly returned to normality by massive untranslatable 4. But I descend from their thought-lineage and in emergency invoke the shadow of their untranslatable 5."

"I am a Confessor," said the Ship's Confessor, "the descendant of those in humanity's past who most highly valued truth, who sought systematic methods for finding truth. But Bayes's Theorem will not be different from one place to another; the laws in their purely mathematical form will be the same, just as any sufficiently advanced species will discover the same periodic table of elements."

"And being universals," said the Lady 3rd, "they bear no distinguishing evidence of their origin. So you should understand, Lord Akon, that a kiritsugu's purpose is not like that of a Confessor, even if we exploit the same laws."

"But we are similar enough to each other," the Confessor concluded, "to see each other as distorted mirror images. Heretics, you might say. She is the ultimate sin forbidden to a Confessor - the exercise of command."

"As you are flawed on my own terms," the Lady 3rd concluded, "one who refuses to help."

Everyone else at the Conference table was staring at the alien holo, and at the Confessor, in something approaching outright horror.

The Lady 3rd shifted her gaze back to Akon. Though it was only a movement of the eyes, there was something of a definite force about the motion, as if the translator was indicating that it stood for something much stronger. Her voice was given a demanding, compelling quality: "What alternatives did your kind generate for dealing with the Babyeaters? Enumerate them to me."

Wipe out their species, keep them in prison forever on suicide watch, ignore them and let the children suffer.

Akon hesitated. An odd premonition of warning prickled at him. Why does she need this information?

"If you do not give me the information," the Lady 3rd said, "I will take into account the fact that you do not wish me to know it."

The proverb went through his mind, The most important part of any secret is the fact that the secret exists.

"All right," Akon said. "We found unacceptable the alternative of leaving the Babyeaters be. We found unacceptable the alternative of exterminating them. We wish to respect their choices and their nature as a species, but their children, who do not share that choice, are unwilling victims; this is unacceptable to us. We desire to keep the children alive but we do not know what to do with them once they become adult and start wanting to eat their own babies. Those were all the alternatives we had gotten as far as generating, at the very moment your ship arrived."

"That is all?" demanded the Lady 3rd. "That is the sum of all your thought? Is this one of the circumstances under which your species sends signals that differ against internal belief, such as 'joking' or 'politeness'?"

"No," said Akon. "I mean, yes. Yes, that's as far as we got. No, we're not joking."

"You should understand," the Confessor said, "that this crew, also, experienced a certain distress, interfering with our normal function, on comprehending the Babyeaters. We are still experiencing it."

And you acted to restore order, thought Akon, though not the same way as a kiritsugu...

"I see," the Lady 3rd said.

She fell silent. There were long seconds during which she sat motionless.

Then, "Why have you not yet disabled the Babyeater ship? Your craft possesses the capability of doing so, and you must realize that your purpose now opposes theirs."

"Because," Akon said, "they did not disable our ship."

The Lady 3rd nodded. "You are symmetrists, then."

Again the silence.

Then the holo blurred, and in that blur appeared the words:

Cultural Translator version 3.

The blur resolved itself back into that pale woman; almost the same as before, except that the serenity of her came through with more force.

The Lady 3rd drew herself erect, and took on a look of ritual, as though she were about to recite a composed poem.

"I now speak," the Lady 3rd, "on behalf of my species, to yours."

A chill ran down Akon's spine. This is too much, this is all too large for me -

"Humankind!" the Lady 3rd said, as though addressing someone by name. "Humankind, you prefer the absence of pain to its presence. When my own kind attained to technology, we eliminated the causes of suffering among ourselves. Bodily pain, embarrassment, and romantic conflicts are no longer permitted to exist. Humankind, you prefer the presence of pleasure to its absence. We have devoted ourselves to the intensity of pleasure, of sex and childbirth and untranslatable 2. Humankind, you prefer truth to lies. By our nature we do not communicate statements disbelieved, as you do with humor, modesty, and fiction; we have even learned to refrain from withholding information, though we possess that capability. Humankind, you prefer peace to violence. Our society is without crime and without war. Through symmetric sharing and untranslatable 4, we share our joys and are pleasured together. Our name for ourselves is not expressible in your language. But to you, humankind, we now name ourselves after the highest values we share: we are the Maximum Fun-Fun Ultra Super Happy People."

There were muffled choking sounds from the human Conference table.

"Um," Akon said intelligently. "Um... good for you?"

"Humankind! Humankind, you did not likewise repair yourselves when you attained to technology. We are still unsure if it is somehow a mistake, if you did not think it through, or if your will is truly so different from ours. For whatever reason, you currently permit the existence of suffering which our species has eliminated. Bodily pain, embarrassment, and romantic troubles are still known among you. Your existence, therefore, is shared by us as pain. Will you, humankind, by your symmetry, remedy this?"

An electric current of shock and alarm ran through the Conference. The Lord Pilot glanced significantly at the Ship's Engineer, and the Engineer just as significantly shook his head. There was nothing they could do against the alien vessel; and their own shields would scarcely help, if they were attacked.

Akon drew in a ragged breath. He was suddenly distracted, almost to the point of his brain melting, by a sense of futures twisting around these moments: the fate of star systems, the destiny of all humanity being warped and twisted and shaped.

So to you, then, it is humanity that molests kittens.

He should have foreseen this possibility, after the experience of the Babyeaters. If the Babyeaters' existence was morally unacceptable to humanity, then the next alien species might be intolerable as well - or they might find humanity's existence a horror of unspeakable cruelty. That was the other side of the coin, even if a human might find it harder to think of it.

Funny. It doesn't seem that bad from in here...

"But -" Akon said, and only then became aware that he was speaking.

"'But'?" said the Lady 3rd. "Is that your whole reply, humankind?" There was a look on her face of something like frustration, even sheer astonishment.

He hadn't planned out this reply in any detail, but -

"You say that you feel our existence as pain," Akon said, "sharing sympathy with our own suffering. So you, also, believe that under some circumstances pain is preferable to pleasure. If you did not hurt when others hurt - would you not feel that you were... less the sort of person you wanted to be? It is the same with us -"

But the Lady 3rd was shaking her head. "You confuse a high conditional likelihood from your hypothesis to the evidence with a high posterior probability of the hypothesis given the evidence," she said, as if that were all one short phrase in her own language. "Humankind, we possess a generalized faculty to feel what others feel. That is the simple, compact relation. We did not think to complicate that faculty to exclude pain. We did not then assign dense probability that other sentient species would traverse the stars, and be encountered by us, and yet fail to have repaired themselves. Should we encounter some future species in circumstances that do not permit its repair, we will modify our empathic faculty to exclude sympathy with pain, and substitute an urge to meliorate pain."

"But -" Akon said.

Dammit, I'm talking again.

"But we chose this; this is what we want."

"That matters less to our values than to yours," replied the Lady 3rd. "But even you, humankind, should see that it is moot. We are still trying to untangle the twisting references of emotion by which humans might prefer pleasure to pain, and yet endorse complex theories that uphold pain over pleasure. But we have already determined that your children, humankind, do not share the grounding of these philosophies. When they incur pain they do not contemplate its meaning, they only call for it to stop. In their simplicity -"

They're a lot like our own children, really.

"- they somewhat resemble the earlier life stages of our own kind."

There was a electric quality now about that pale woman, a terrible intensity. "And you should understand, humankind, that when a child anywhere suffers pain and calls for it to stop, then we will answer that call if it requires sixty-five thousand five hundred and thirty-six ships."

"We believe, humankind, that you can understand our viewpoint. Have you options to offer us?"

To be continued...

Referenced by: Three Worlds Collide (0/8) • War and/or Peace (2/8) • Wise Pretensions v.0 • Belief in Self-Deception • My Way

Original with comments: The Super Happy People (3/8)

Interlude with the Confessor (4/8)

Eliezer Yudkowsky, 02 February 2009 09:11AM

(Part 4 of 8 in "Three Worlds Collide")

The two of them were alone now, in the Conference Chair's Privilege, the huge private room of luxury more suited to a planet than to space. The Privilege was tiled wall-to-wall and floor-to-ceiling with a most excellent holo of the space surrounding them: the distant stars, the system's sun, the fleeing nova ashes, and the glowing ember of the dwarf star that had siphoned off hydrogen from the main sun until its surface had briefly ignited in a nova flash. It was like falling through the void.

Akon sat on the edge of the four-poster bed in the center of the room, resting his head in his hands. Weariness dulled him at the moment when he most needed his wits; it was always like that in crisis, but this was unusually bad. Under the circumstances, he didn't dare snort a hit of caffeine - it might reorder his priorities. Humanity had yet to discover the drug that was pure energy, that would improve your thinking without the slightest touch on your emotions and values.

"I don't know what to think," Akon said.

The Ship's Confessor was standing stately nearby, in full robes and hood of silver.  From beneath the hood came the formal response: "What seems to be confusing you, my friend?"

"Did we go wrong?" Akon said. No matter how hard he tried, he couldn't keep the despair out of his voice. "Did humanity go down the wrong path?"

The Confessor was silent a long time.

Akon waited. This was why he couldn't have talked about the question with anyone else. Only a Confessor would actually think before answering, if asked a question like that.

"I've often wondered that myself," the Confessor finally said, surprising Akon. "There were so many choices, so many branchings in human history - what are the odds we got them all right?"

The hood turned away, angling in the direction of the Superhappy ship - though it was too far away to be visible, everyone on board the Impossible Possible World knew where it was. "There are parts of your question I can't help you with, my lord. Of all people on this ship, I might be most poorly suited to answer... But you do understand, my lord, don't you, that neither the Babyeaters nor the Superhappies are evidence that we went wrong? If you weren't worried before, you shouldn't be any more worried now. The Babyeaters strive to do the baby-eating thing to do, the Superhappies output the Super Happy thing to do. None of that tells us anything about the right thing to do. They are not asking the same question we are - no matter what word of their language the translator links to our 'should'. If you're confused at all about that, my lord, I might be able to clear it up."

"I know the theory," Akon said. Exhaustion in his voice. "They made me study metaethics when I was a little kid, sixteen years old and still in the children's world. Just so that I would never be tempted to think that God or ontologically basic moral facts or whatever had the right to override my own scruples." Akon slumped a little further. "And somehow - none of that really makes a difference when you're looking at the Lady 3rd, and wondering why, when there's a ten-year-old with a broken finger in front of you, screaming and crying, we humans only partially numb the area."

The Confessor's hood turned back to look at Akon. "You do realize that your brain is literally hardwired to generate error signals when it sees other human-shaped objects stating a different opinion from yourself. You do realize that, my lord?"

"I know," Akon said. "That, too, we are taught. Unfortunately, I am also just now realizing that I've only been going along with society all my life, and that I never thought the matter through for myself, until now."

A sigh came from that hood. "Well... would you prefer a life entirely free of pain and sorrow, having sex all day long?"

"Not... really," Akon said.

The shoulders of the robe shrugged. "You have judged. What else is there?"

Akon stared straight at that anonymizing robe, the hood containing a holo of dark mist, a shadow that always obscured the face inside. The voice was also anonymized - altered slightly, not in any obtrusive way, but you wouldn't know your own Confessor to hear him speak. Akon had no idea who the Confessor might be, outside that robe. There were rumors of Confessors who had somehow arranged to be seen in the company of their own secret identity...

Akon drew a breath. "You said that you, of all people, could not say whether humanity had gone down the wrong path. The simple fact of being a Confessor should have no bearing on that; rationalists are also human. And you told the Lady 3rd that you were too old to make decisions for your species. Just how old are you... honorable ancestor?"

There was a silence.

It didn't last long.

As though the decision had already been foreseen, premade and preplanned, the Confessor's hands moved easily upward and drew back the hood - revealing an unblended face, strangely colored skin and shockingly distinctive features. A face out of forgotten history, which could only have come from a time before the genetic mixing of the 21st century, untouched by DNA insertion or diaspora.

Even though Akon had been half-expecting it, he still gasped out loud. Less than one in a million: That was the percentage of the current human population that had been born on Earth before the invention of antiagathics or star travel, five hundred years ago.

"Congratulations on your guess," the Confessor said. The unaltered voice was only slightly different; but it was stronger, more masculine.

"Then you were there," Akon said. He felt almost breathless, and tried not to show it. "You were alive - all the way back in the days of the initial biotech revolution! That would have been when humanity first debated whether to go down the Super Happy path."

The Confessor nodded.

"Which side did you argue?"

The Confessor's face froze for a moment, and then he emitted a brief chuckle, one short laugh. "You have entirely the wrong idea about how things were done, back then. I suppose it's natural."

"I don't understand," Akon said.

"And there are no words that I can speak to make you understand. It is beyond your imagining. But you should not imagine that a violent thief whose closest approach to industry was selling uncertified hard drugs - you should not imagine, my lord, my honorable descendant, that I was ever asked to take sides."

Akon's eyes slid away from the hot gaze of the unmixed man; there was something wrong about the thread of anger still there in the memory after five hundred years.

"But time passed," the Confessor said, "time moved forward, and things changed." The eyes were no longer focused on Akon, looking now at something far away. "There was an old saying, to the effect that while someone with a single bee sting will pay much for a remedy, to someone with five bee stings, removing just one sting seems less attractive. That was humanity in the ancient days. There was so much wrong with the world that the small resources of altruism were splintered among ten thousand urgent charities, and none of it ever seemed to go anywhere. And yet... and yet..."

"There was a threshold crossed somewhere," said the Confessor, "without a single apocalypse to mark it. Fewer wars. Less starvation. Better technology. The economy kept growing.  People had more resource to spare for charity, and the altruists had fewer and fewer causes to choose from. They came even to me, in my time, and rescued me. Earth cleaned itself up, and whenever something threatened to go drastically wrong again, the whole attention of the planet turned in that direction and took care of it. Humanity finally got its act together."

The Confessor worked his jaws as if there were something stuck in his throat. "I doubt you can even imagine, my honorable descendant, just how much of an impossible dream that once was. But I will not call this path mistaken."

"No, I can't imagine," Akon said quietly. "I once tried to read some of the pre-Dawn Net. I thought I wanted to know, I really did, but I - just couldn't handle it. I doubt anyone on this ship can handle it except you. Honorable ancestor, shouldn't we be asking you how to deal with the Babyeaters and the Superhappies? You are the only one here who's ever dealt with that level of emergency."

"No," said the Confessor, like an absolute order handed down from outside the universe. "You are the world that we wanted to create. Though I can't say we.  That is just a distortion of memory, a romantic gloss on history fading into mist. I wasn't one of the dreamers, back then. I was just wrapped up in my private blanket of hurt. But if my pain meant anything, Akon, it is as part of the long price of a better world than that one. If you look back at ancient Earth, and are horrified - then that means it was all for something, don't you see? You are the beautiful and shining children, and this is your world, and you are the ones who must decide what to do with it now."

Akon started to speak, to demur -

The Confessor held up a hand. "I mean it, my lord Akon. It is not polite idealism. We ancients can't steer. We remember too much disaster. We're too cautious to dare the bold path forward. Do you know there was a time when nonconsensual sex was illegal?"

Akon wasn't sure whether to smile or grimace. "The Prohibition, right?  During the first century pre-Net? I expect everyone was glad to have that law taken off the books. I can't imagine how boring your sex lives must have been up until then - flirting with a woman, teasing her, leading her on, knowing the whole time that you were perfectly safe because she couldn't take matters into her own hands if you went a little too far -"

"You need a history refresher, my Lord Administrator. At some suitably abstract level. What I'm trying to tell you - and this is not public knowledge - is that we nearly tried to overthrow your government."

"What?" said Akon. "The Confessors?"

"No, us.  The ones who remembered the ancient world. Back then we still had our hands on a large share of the capital and tremendous influence in the grant committees. When our children legalized rape, we thought that the Future had gone wrong."

Akon's mouth hung open. "You were that prude?"

The Confessor shook his head. "There aren't any words," the Confessor said, "there aren't any words at all, by which I ever could explain to you. No, it wasn't prudery. It was a memory of disaster."

"Um," Akon said. He was trying not to smile. "I'm trying to visualize what sort of disaster could have been caused by too much nonconsensual sex -"

"Give it up, my lord," the Confessor said. He was finally laughing, but there was an undertone of pain to it. "Without, shall we say, personal experience, you can't possibly imagine, and there's no point in trying."

"Well, out of curiosity - how much did you lose?"

The Confessor seemed to freeze, for a moment. "What?"

"How much did you lose in the legislative prediction markets, betting on whatever dreadful outcome you thought would happen?"

"You really wouldn't ever understand," the Confessor said. His smile was entirely real, now. "But now you know, don't you? You know, after speaking to me, that I can't ever be allowed to make decisions for humankind."

Akon hesitated. It was odd... he did know, on some gut level. And he couldn't have explained on any verbal level why.  Just - that hint of wrongness.

"So now you know," the Confessor repeated. "And because we do remember so much disaster - and because it is a profession that benefits from being five hundred years old - many of us became Confessors. Being the voice of pessimism comes easily to us, and few indeed are those among the human kind who must rationally be nudged upward... We advise, but do not lead. Debate, but do not decide. We're going along for your ride, and trying not to be too shocked so that we can be almost as delighted as you. You might find yourself in a similar situation in five hundred years... if humanity survives this week."

"Ah, yes," Akon said dryly. "The aliens. The current problem of discourse."

"Yes. Have you had any thoughts on the subject?"

"Only that I really do wish that humanity had been alone in the universe."  Akon's hand suddenly formed a fist and smashed hard against the bed. "Fuck it! I know how the Superhappies felt when they discovered that we and the Babyeaters hadn't 'repaired ourselves'. You understand what this implies about what the rest of the universe looks like, statistically speaking? Even if it's just a sample of two? I'm sure that somewhere out there are likable neighbors. Just as somewhere out there, if we go far enough through the infinite universe, there's a person who's an exact duplicate of me down to the atomic level. But every other species we ever actually meet is probably going to be -" Akon drew a breath. "It wasn't supposed to be like this, damn it! All three of our species have empathy, we have sympathy, we have a sense of fairness - the Babyeaters even tell stories like we do, they have art.  Shouldn't that be enough? Wasn't that supposed to be enough? But all it does is put us into enough of the same reference frame that we can be horrible by each others' standards."

"Don't take this the wrong way," the Confessor said, "but I'm glad that we ran across the Babyeaters."

Words stuck in Akon's throat. "What?"

A half-smile twisted up one corner of the Confessor's face. "Because if we hadn't run across the Babyeaters, we couldn't possibly rescue the babies, now could we? Not knowing about their existence wouldn't mean they weren't there. The Babyeater children would still exist. They would still die in horrible agony. We just wouldn't be able to help them. If we didn't know it wouldn't be our fault, our responsibility - but that's not something you're supposed to optimize for." The Confessor paused.  "Of course I understand how you feel. But on this vessel I am humanity's token attempt at sanity, and it is my duty to think certain strange yet logical thoughts."

"And the Superhappies?" Akon said. "The race with superior technology that may decide to exterminate us, or keep us in prison, or take our children away? Is there any silver lining to that?"

"The Superhappies aren't so far from us," the Confessor said. "We could have gone down the Super Happy path. We nearly did - you might have trouble imagining just how attractive the absence of pain can sound, under certain circumstances. In a sense, you could say that I tried to go down that path - though I wasn't a very competent neuroengineer. If human nature had been only slightly different, we could easily have been within that attractor. And the Super Happy civilization is not hateful to us, whatever we are to them. That's good news at least, for how the rest of the universe might look." The Confessor paused. "And..."

"And?"

The Confessor's voice became harder. "And the Superhappies will rescue the Babyeater children no matter what, I think, even if humanity should fail in the task. Considering how many Babyeater children are dying, and in what pain - that could outweigh even our own extermination.  Shut up and multiply, as the saying goes."

"Oh, come on!" Akon said, too surprised to be shocked. "If the Superhappies hadn't shown up, we would have - well, we would have done something about the Babyeaters, once we decided what. We wouldn't have just let the, the -"

"Holocaust," the Confessor offered.

"Good word for it. We wouldn't have just let the Holocaust go on."

"You would be astounded, my lord, at what human beings will just let go on. Do you realize the expenditure of capital, labor, maybe even human lives required to invade every part of the Babyeater civilization? To trace out every part of their starline network, push our technological advantage to its limit to build faster ships that can hunt down every Babyeater ship that tries to flee? Do you realize -"

"I'm sorry. You are simply mistaken as a question of fact." Boy, thought Akon, you don't often get to say that to a Confessor. "This is not your birth era, honorable ancestor. We are the humanity that has its shit together. If the Superhappies had never come along, humanity would have done whatever it took to rescue the Babyeater children. You saw the Lord Pilot, the Lady Sensory; they were ready to secede from civilization if that's what it took to get the job done. And that, honorable ancestor, is how most people would react."

"For a moment," said the Confessor. "In the moment of first hearing the news. When talk was cheap. When they hadn't yet visualized the costs. But once they did, there would be an uneasy pause, while everyone waited to see if someone else might act first. And faster than you imagine possible, people would adjust to that state of affairs. It would no longer sound quite so shocking as it did at first. Babyeater children are dying horrible, agonizing deaths in their parents' stomachs? Deplorable, of course, but things have always been that way. It would no longer be news. It would all be part of the plan."

"Are you high on something?" Akon said. It wasn't the most polite way he could have phrased it, but he couldn't help himself.

The Confessor's voice was as cold and hard as an iron sun, after the universe had burned down to embers. "Innocent youth, when you have watched your older brother beaten almost to death before your eyes, and seen how little the police investigate - when you have watched all four of your grandparents wither away like rotten fruit and cease to exist, while you spoke not one word of protest because you thought it was normal - then you may speak to me of what human beings will tolerate."

"I don't believe we would do that," Akon said as mildly as possible.

"Then you fail as a rationalist," the Confessor said. His unhooded head turned toward the false walls, to look out at the accurately represented stars. "But I - I will not fail again."

"Well, you're damn right about one thing," Akon said. He was too exhausted to be tactful. "You can't ever be allowed to make decisions for the human species."

"I know. Believe me, I know. Only youth can Administrate. That is the pact of immortality."

Akon stood up from the bed. "Thank you, Confessor. You have helped me."

With an easy, practiced motion, the Confessor slid the hood of his robe over his head, and the stark features vanished into shadow. "I have?" the Confessor said, and his recloaked voice sounded strangely mild, after that earlier masculine power. "How?"

Akon shrugged. He didn't think he could put it into words. It had something to do with the terrible vast sweep of Time across the centuries, and so much true change that had already happened, deeper by far than anything he had witnessed in his own lifetime; the requirement of courage to face the future, and the sacrifices that had been made for it; and that not everyone had been saved, once upon a time.

"I guess you reminded me," Akon said, "that you can't always get everything you want."

To be continued...

Referenced by: Three Worlds Collide (0/8) • The Super Happy People (3/8) • Can Humanism Match Religion's Output? • Church vs. Taskforce • This Failing Earth

Original with comments: Interlude with the Confessor (4/8)

Three Worlds Decide (5/8)

Eliezer Yudkowsky, 03 February 2009 09:14AM

(Part 5 of 8 in "Three Worlds Collide")

Akon strode into the main Conference Room; and though he walked like a physically exhausted man, at least his face was determined. Behind him, the shadowy Confessor followed.

The Command Conference looked up at him, and exchanged glances.

"You look better," the Ship's Master of Fandom ventured.

Akon put a hand on the back of his seat, and paused. Someone was absent. "The Ship's Engineer?"

The Lord Programmer frowned. "He said he had an experiment to run, my lord. He refused to clarify further, but I suppose it must have something to do with the Babyeaters' data -"

"You're joking," Akon said. "Our Ship's Engineer is off Nobel-hunting? Now? With the fate of the human species at stake?"

The Lord Programmer shrugged. "He seemed to think it was important, my lord."

Akon sighed. He pulled his chair back and half-slid, half-fell into it. "I don't suppose that the ship's markets have settled down?"

The Lord Pilot grinned sardonically. "Read for yourself."

Akon twitched, calling up a screen. "Ah, I see. The ship's Interpreter of the Market's Will reports, and I quote, 'Every single one of the underlying assets in my market is going up and down like a fucking yo-yo while the ship's hedgers try to adjust to a Black Swan that's going to wipe out ninety-eight percent of their planetside risk capital. Even the spot prices on this ship are going crazy; either we've got bubble traders coming out of the woodwork, or someone seriously believes that sex is overvalued relative to orange juice.  One derivatives trader says she's working on a contract that will have a clearly defined value in the event that aliens wipe out the entire human species, but she says it's going to take a few hours and I say she's on crack. Indeed I believe an actual majority of the people still trying to trade in this environment are higher than the heliopause. Bid-ask spreads are so wide you could kick a fucking football stadium through them, nothing is clearing, and I have unisolated conditional dependencies coming out of my ass. I have no fucking clue what the market believes. Someone get me a drink.'  Unquote." Akon looked at the Master of Fandom. "Any suggestions get reddited up from the rest of the crew?"

The Master cleared his throat. "My lord, we took the liberty of filtering out everything that was physically impossible, based on pure wishful thinking, or displayed a clear misunderstanding of naturalistic metaethics. I can show you the raw list, if you'd like."

"And what's left?" Akon said. "Oh, never mind, I get it."

"Well, not quite," said the Master. "To summarize the best ideas -" He gestured a small holo into existence.

Ask the Superhappies if their biotechnology is capable of in vivo cognitive alterations of Babyeater children to ensure that they don't grow up wanting to eat their own children. Sterilize the current adults. If Babyeater adults cannot be sterilized and will not surrender, imprison them. If that's too expensive, kill most of them, but leave enough in prison to preserve their culture for the children. Offer the Superhappies an alliance to invade the Babyeaters, in which we provide the capital and labor and they provide the technology.

"Not too bad," Akon said. His voice grew somewhat dry. "But it doesn't seem to address the question of what the Superhappies are supposed to do with us. The analogous treatment -"

"Yes, my lord," the Master said. "That was extensively pointed out in the comments, my lord. And the other problem is that the Superhappies don't really need our labor or our capital." The Master looked in the direction of the Lord Programmer, the Xenopsychologist, and the Lady Sensory.

The Lord Programmer said, "My lord, I believe the Superhappies think much faster than we do. If their cognitive systems are really based on something more like DNA than like neurons, that shouldn't be surprising. In fact, it's surprising that the speedup is as little as -" The Lord Programmer stopped, and swallowed. "My lord. The Superhappies responded to most of our transmissions extremely quickly.  There was, however, a finite delay. And that delay was roughly proportional to the length of the response, plus an additive constant.  Going by the proportion, my lord, I believe they think between fifteen and thirty times as fast as we do, to the extent such a comparison can be made. If I try to use Moore's Law type reasoning on some of the observable technological parameters in their ship - Alderson flux, power density, that sort of thing - then I get a reasonably convergent estimate that the aliens are two hundred years ahead of us in human-equivalent subjective time.  Which means it would be twelve hundred equivalent years since their Scientific Revolution."

"If," the Xenopsychologist said, "their history went as slowly as ours. It probably didn't." The Xenopsychologist took a breath. "My lord, my suspicion is that the aliens are literally able to run their entire ship using only three kiritsugu as sole crew. My lord, this may represent, not only the superior programming ability that translated their communications to us, but also the highly probable case that Superhappies can trade knowledge and skills among themselves by having sex. Every individual of their species might contain the memory of their Einsteins and Newtons and a thousand other areas of expertise, no more conserved than DNA is conserved among humans. My lord, I suspect their version of Galileo was something like thirty objective years ago, as the stars count time, and that they've been in space for maybe twenty years."

The Lady Sensory said, "Their ship has a plane of symmetry, and it's been getting wider on the axis through that plane, as it sucks up nova dust and energy. It's growing on a smooth exponential at 2% per hour, which means it can split every thirty-five hours in this environment."

"I have no idea," the Xenopsychologist said, "how fast the Superhappies can reproduce themselves - how many children they have per generation, or how fast their children sexually mature. But all things considered, I don't think we can count on their kids taking twenty years to get through high school."

There was silence.

When Akon could speak again, he said, "Are you all quite finished?"

"If they let us live," the Lord Programmer said, "and if we can work out a trade agreement with them under Ricardo's Law of Comparative Advantage, interest rates will -"

"Interest rates can fall into an open sewer and die. Any further transmissions from the Superhappy ship?"

The Lady Sensory shook her head.

"All right," Akon said. "Open a transmission channel to them."

There was a stir around the table. "My lord -" said the Master of Fandom. "My lord, what are you going to say?"

Akon smiled wearily. "I'm going to ask them if they have any options to offer us."

The Lady Sensory looked at the Ship's Confessor. The hood silently nodded: He's still sane.

The Lady Sensory swallowed, and opened a channel. On the holo there first appeared, as a screen:

The Lady 3rd Kiritsugu
temporary co-chair of the Gameplayer
Language Translator version 9
Cultural Translator version 16

The Lady 3rd in this translation was slightly less pale, and looked a bit more concerned and sympathetic. She took in Akon's appearance at a glance, and her eyes widened in alarm. "My lord, you're hurting!"

"Just tired, milady," Akon said. He cleared his throat. "Our ship's decision-making usually relies on markets and our markets are behaving erratically. I'm sorry to inflict that on you as shared pain, and I'll try to get this over with quickly. Anyway -"

Out of the corner of his eye, Akon saw the Ship's Engineer re-enter the room; the Engineer looked as if he had something to say, but froze when he saw the holo.

There was no time for that now.

"Anyway," Akon said, "we've worked out that the key decisions depend heavily on your level of technology. What do you think you can actually do with us or the Babyeaters?"

The Lady 3rd sighed. "I really should get your independent component before giving you ours - you should at least think of it first - but I suppose we're out of luck on that. How about if I just tell you what we're currently planning?"

Akon nodded. "That would be much appreciated, milady." Some of his muscles that had been tense, started to relax. Cultural Translator version 16 was a lot easier on his brain. Distantly, he wondered if some transformed avatar of himself was making skillful love to the Lady 3rd -

"All right," the Lady 3rd said. "We consider that the obvious starting point upon which to build further negotiations, is to combine and compromise the utility functions of the three species until we mutually satisfice, providing compensation for all changes demanded. The Babyeaters must compromise their values to eat their children at a stage where they are not sentient - we might accomplish this most effectively by changing the lifecycle of the children themselves. We can even give the unsentient children an instinct to flee and scream, and generate simple spoken objections, but prevent their brain from developing self-awareness until after the hunt."

Akon straightened. That actually sounded - quite compassionate - sort of -

"Our own two species," the Lady 3rd said, "which desire this change of the Babyeaters, will compensate them by adopting Babyeater values, making our own civilization of greater utility in their sight: we will both change to spawn additional infants, and eat most of them at almost the last stage before they become sentient."

The Conference room was frozen. No one moved. Even their faces didn't change expression.

Akon's mind suddenly flashed back to those writhing, interpenetrating, visually painful blobs he had seen before.

A cultural translator could change the image, but not the reality.

"It is nonetheless probable," continued the Lady 3rd, "that the Babyeaters will not accept this change as it stands; it will be necessary to impose these changes by force. As for you, humankind, we hope you will be more reasonable. But both your species, and the Babyeaters, must relinquish bodily pain, embarrassment, and romantic troubles. In exchange, we will change our own values in the direction of yours. We are willing to change to desire pleasure obtained in more complex ways, so long as the total amount of our pleasure does not significantly decrease. We will learn to create art you find pleasing. We will acquire a sense of humor, though we will not lie. From the perspective of humankind and the Babyeaters, our civilization will obtain much utility in your sight, which it did not previously possess. This is the compensation we offer you. We furthermore request that you accept from us the gift of untranslatable 2, which we believe will enhance, on its own terms, the value that you name 'love'. This will also enable our kinds to have sex using mechanical aids, which we greatly desire. At the end of this procedure, all three species will satisfice each other's values and possess great common ground, upon which we may create a civilization together."

Akon slowly nodded. It was all quite unbelievably civilized. It might even be the categorically best general procedure when worlds collided.

The Lady 3rd brightened. "A nod - is that assent, humankind?"

"It's acknowledgment," Akon said. "We'll have to think about this."

"I understand," the Lady 3rd said. "Please think as swiftly as you can.  Babyeater children are dying in horrible agony as you think."

"I understand," Akon said in return, and gestured to cut the transmission.

The holo blinked out.

There was a long, terrible silence.

"No."

The Lord Pilot said it. Cold, flat, absolute.

There was another silence.

"My lord," the Xenopsychologist said, very softly, as though afraid the messenger would be torn apart and dismembered, "I do not think they were offering us that option."

"Actually," Akon said, "The Superhappies offered us more than we were going to offer the Babyeaters. We weren't exactly thinking about how to compensate them." It was strange, Akon noticed, his voice was very calm, maybe even deadly calm. "The Superhappies really are a very fair-minded people. You get the impression they would have proposed exactly the same solution whether or not they happened to hold the upper hand. We might have just enforced our own will on the Babyeaters and told the Superhappies to take a hike. If we'd held the upper hand. But we don't. And that's that, I guess."

"No!" shouted the Lord Pilot. "That's not -"

Akon looked at him, still with that deadly calm.

The Lord Pilot was breathing deeply, not as if quieting himself, but as if preparing for battle on some ancient savanna plain that no longer existed. "They want to turn us into something inhuman. It - it cannot - we cannot - we must not allow -"

"Either give us a better option or shut up," the Lord Programmer said flatly.  "The Superhappies are smarter than us, have a technological advantage, think faster, and probably reproduce faster. We have no hope of holding them off militarily. If our ships flee, the Superhappies will simply follow in faster ships. There's no way to shut a starline once opened, and no way to conceal the fact that it is open -"

"Um," the Ship's Engineer said.

Every eye turned to him.

"Um," the Ship's Engineer said. "My Lord Administrator, I must report to you in private."

The Ship's Confessor shook his head. "You could have handled that better, Engineer."

Akon nodded to himself. It was true. The Ship's Engineer had already betrayed the fact that a secret existed. Under the circumstances, easy to deduce that it had come from the Babyeater data. That was eighty percent of the secret right there. And if it was relevant to starline physics, that was half of the remainder.

"Engineer," Akon said, "since you have already revealed that a secret exists, I suggest you tell the full Command Conference. We need to stay in sync with each other. Two minds are not a committee. We'll worry later about keeping the secret classified."

The Ship's Engineer hesitated. "Um, my lord, I suggest that I report to you first, before you decide -"

"There's no time," Akon said. He pointed to where the holo had been.

"Yes," the Master of Fandom said, "we can always slit our own throats afterward, if the secret is that awful." The Master of Fandom gave a small laugh -

- then stopped, at the look on the Engineer's face.

"At your will, my lord," the Engineer said.

He drew a deep breath. "I asked the Lord Programmer to compare any identifiable equations and constants in the Babyeater's scientific archive, to the analogous scientific data of humanity. Most of the identified analogues were equal, of course. In some places we have more precise values, as befits our, um, superior technological level.  But one anomaly did turn up: the Babyeater figure for Alderson's Coupling Constant was ten orders of magnitude larger than our own."

The Lord Pilot whistled. "Stars above, how did they manage to make that mistake -"

Then the Lord Pilot stopped abruptly.

"Alderson's Coupling Constant," Akon echoed. "That's the... coupling between Alderson interactions and the..."

"Between Alderson interactions and the nuclear strong force," the Lord Pilot said. He was beginning to smile, rather grimly. "It was a free parameter in the standard model, and so had to be established experimentally. But because the interaction is so incredibly... weak... they had to build an enormous Alderson generator to find the value. The size of a very small moon, just to give us that one number. Definitely not something you could check at home. That's the story in the physics textbooks, my lords, my lady."

The Master of Fandom frowned. "You're saying... the physicists faked the result in order to... fund a huge project...?" He looked puzzled.

"No," the Lord Pilot said. "Not for the love of power. Engineer, the Babyeater value should be testable using our own ship's Alderson drive, if the coupling constant is that strong. This you have done?"

The Ship's Engineer nodded. "The Babyeater value is correct, my lord."

The Ship's Engineer was pale. The Lord Pilot was clenching his jaw into a sardonic grin.

"Please explain," Akon said. "Is the universe going to end in another billion years, or something? Because if so, the issue can wait -"

"My lord," the Ship's Confessor said, "suppose the laws of physics in our universe had been such that the ancient Greeks could invent the equivalent of nuclear weapons from materials just lying around.  Imagine the laws of physics had permitted a way to destroy whole countries with no more difficulty than mixing gunpowder. History would have looked quite different, would it not?"

Akon nodded, puzzled. "Well, yes," Akon said. "It would have been shorter."

"Aren't we lucky that physics didn't happen to turn out that way, my lord? That in our own time, the laws of physics don't permit cheap, irresistable superweapons?"

Akon furrowed his brow -

"But my lord," said the Ship's Confessor, "do we really know what we think we know? What different evidence would we see, if things were otherwise? After all - if you happened to be a physicist, and you happened to notice an easy way to wreak enormous destruction using off-the-shelf hardware - would you run out and tell you?"

"No," Akon said. A sinking feeling was dawning in the pit of his stomach.  "You would try to conceal the discovery, and create a cover story that discouraged anyone else from looking there."

The Lord Pilot emitted a bark that was half laughter, and half something much darker. "It was perfect. I'm a Lord Pilot and I never suspected until now."

"So?" Akon said. "What is it, actually?"

"Um," the Ship's Engineer said. "Well... basically... to skip over the technical details..."

The Ship's Engineer drew a breath.

"Any ship with a medium-sized Alderson drive can make a star go supernova."

Silence.

"Which might seem like bad news in general," the Lord Pilot said, "but from our perspective, right here, right now, it's just what we need. A mere nova wouldn't do it. But blowing up the whole star - " He gave that bitter bark of laughter, again. "No star, no starlines. We can make the main star of this system go supernova - not the white dwarf, the companion.  And then the Superhappies won't be able to get to us. That is, they won't be able to get to the human starline network. We will be dead. If you care about tiny irrelevant details like that." The Lord Pilot looked around the Conference Table. "Do you care? The correct answer is no, by the way."

"I care," the Lady Sensory said softly. "I care a whole lot. But..."  She folded her hands atop the table and bowed her head.

There were nods from around the Table.

The Lord Pilot looked at the Ship's Engineer. "How long will it take for you to modify the ship's Alderson Drive -"

"It's done," said the Ship's Engineer. "But... we should, um, wait until the Superhappies are gone, so they don't detect us doing it."

The Lord Pilot nodded. "Sounds like a plan. Well, that's a relief. And here I thought the whole human race was doomed, instead of just us." He looked inquiringly at Akon. "My lord?"

Akon rested his head in his hands, suddenly feeling more weary than he had ever felt in his life. From across the table, the Confessor watched him - or so it seemed; the hood was turned in his direction, at any rate.

I told you so, the Confessor did not say.

"There is a certain problem with your plan," Akon said.

"Such as?" the Lord Pilot said.

"You've forgotten something," Akon said. "Something terribly important. Something you once swore you would protect."

Puzzled faces looked at him.

"If you say something bloody ridiculous like 'the safety of the ship' -" said the Lord Pilot.

The Lady Sensory gasped. "Oh, no," she murmured. "Oh, no. The Babyeater children."

The Lord Pilot looked like he had been punched in the stomach. The grim smiles that had begun to spread around the table were replaced with horror.

"Yes," Akon said. He looked away from the Conference Table. He didn't want to see the reactions. "The Superhappies wouldn't be able to get to us. And they couldn't get to the Babyeaters either. Neither could we. So the Babyeaters would go on eating their own children indefinitely. And the children would go on dying over days in their parents' stomachs. Indefinitely. Is the human race worth that?"

Akon looked back at the Table, just once. The Xenopsychologist looked sick, tears were running down the Master's face, and the Lord Pilot looked like he were being slowly torn in half. The Lord Programmer looked abstracted, the Lady Sensory was covering her face with her hands. (And the Confessor's face still lay in shadow, beneath the silver hood.)

Akon closed his eyes. "The Superhappies will transform us into something not human," Akon said.  "No, let's be frank. Something less than human. But not all that much less than human. We'll still have art, and stories, and love. I've gone entire hours without being in pain, and on the whole, it wasn't that bad an experience -" The words were sticking in his throat, along with a terrible fear. "Well. Anyway. If remaining whole is that important to us - we have the option. It's just a question of whether we're willing to pay the price. Sacrifice the Babyeater children -"

They're a lot like human children, really.

"- to save humanity."

Someone in the darkness was screaming, a thin choked wail that sounded like nothing Akon had ever heard or wanted to hear. Akon thought it might be the Lord Pilot, or the Master of Fandom, or maybe the Ship's Engineer. He didn't open his eyes to find out.

There was a chime.

"In-c-c-coming c-call from the Super Happy," the Lady Sensory spit out the words like acid, "ship, my lord."

Akon opened his eyes, and felt, somehow, that he was still in darkness.

"Receive," Akon said.

The Lady 3rd Kiritsugu appeared before him. Her eyes widened once, as she took in his appearance, but she said nothing.

That's right, my lady, I don't look super happy.

"Humankind, we must have your answer," she said simply.

The Lord Administrator pinched the bridge of his nose, and rubbed his eyes. Absurd, that one human being should have to answer a question like that. He wanted to foist off the decision on a committee, a majority vote of the ship, a market - something that wouldn't demand that anyone accept full responsibility. But a ship run that way didn't work well under ordinary circumstances, and there was no reason to think that things would change under extraordinary circumstances. He was an Administrator; he had to accept all the advice, integrate it, and decide. Experiment had shown that no organizational structure of non-Administrators could match what he was trained to do, and motivated to do; anything that worked was simply absorbed into the Administrative weighting of advice.

Sole decision. Sole responsibility if he got it wrong. Absolute power and absolute accountability, and never forget the second half, my lord, or you'll be fired the moment you get home. Screw up indefensibly, my lord, and all your hundred and twenty years of accumulated salary in escrow, producing that lovely steady income, will vanish before you draw another breath.

Oh - and this time the whole human species will pay for it, too.

"I can't speak for all humankind," said the Lord Administrator. "I can decide, but others may decide differently. Do you understand?"

The Lady 3rd made a light gesture, as if it were of no consequence. "Are you an exceptional case of a human decision-maker?"

Akon tilted his head. "Not... particularly..."

"Then your decision is strongly indicative of what other human decisionmakers will decide," she said. "I find it hard to imagine that the options exactly balance in your decision mechanism, whatever your inability to admit your own preferences."

Akon slowly nodded. "Then..."

He drew a breath.

Surely, any species that reached the stars would understand the Prisoner's Dilemma.  If you couldn't cooperate, you'd just destroy your own stars. A very easy thing to do, as it had turned out. By that standard, humanity might be something of an impostor next to the Babyeaters and the Superhappies. Humanity had kept it a secret from itself. The other two races - just managed not to do the stupid thing. You wouldn't meet anyone out among the stars, otherwise.

The Superhappies had done their very best to press C. Cooperated as fairly as they could.

Humanity could only do the same.

"For myself, I am inclined to accept your offer."

He didn't look around to see how anyone had reacted to that.

"There may be other things," Akon added, "that humanity would like to ask of your kind, when our representatives meet. Your technology is advanced beyond ours."

The Lady 3rd smiled. "We will, of course, be quite positively inclined toward any such requests. As I believe our first message to you said - 'we love you and we want you to be super happy'.  Your joy will be shared by us, and we will be pleasured together."

Akon couldn't bring himself to smile. "Is that all?"

"This Babyeater ship," said the Lady 3rd, "the one that did not fire on you, even though they saw you first. Are you therefore allied with them?"

"What?" Akon said without thinking. "No -"

"My lord!" shouted the Ship's Confessor -

Too late.

"My lord," the Lady Sensory said, her voice breaking, "the Superhappy ship has fired on the Babyeater vessel and destroyed it."

Akon stared at the Lady 3rd in horror.

"I'm sorry," the Lady 3rd Kiritsugu said. "But our negotiations with them failed, as predicted. Our own ship owed them nothing and promised them nothing. This will make it considerably easier to sweep through their starline network when we return. Their children would be the ones to suffer from any delay. You understand, my lord?"

"Yes," Akon said, his voice trembling. "I understand, my lady kiritsugu." He wanted to protest, to scream out. But the war was only beginning, and this - would admittedly save -

"Will you warn them?" the Lady 3rd asked.

"No," Akon said. It was the truth.

"Transforming the Babyeaters will take precedence over transforming your own species. We estimate the Babyeater operation may take several weeks of your time to conclude. We hope you do not mind waiting. That is all," the Lady 3rd said.

And the holo faded.

"The Superhappy ship is moving out," the Lady Sensory said. She was crying, silently, as she steadily performed her duty of reporting. "They're heading back toward their starline origin."

"All right," Akon said. "Take us home. We need to report on the negotiations -"

There was an inarticulate scream, like that throat was trying to burst the walls of the Conference chamber, as the Lord Pilot burst out of his chair, burst all restraints he had placed on himself, and lunged forward.

But standing behind his target, unnoticed, the Ship's Confessor had produced from his sleeve the tiny stunner - the weapon which he alone on the ship was authorized to use, if he made a determination of outright mental breakdown. With a sudden motion, the Confessor's arm swept out...

1. ... and anesthetized the Lord Pilot.

2. ... [This option will become the True Ending only if someone suggests it in the comments before the previous ending is posted tomorrow. Otherwise, the first ending is the True one.]

Referenced by: Three Worlds Collide (0/8) • Interlude with the Confessor (4/8) • (Moral) Truth in Fiction?

Original with comments: Three Worlds Decide (5/8)

Normal Ending: Last Tears (6/8)

Eliezer Yudkowsky, 04 February 2009 08:45AM

(Part 6 of 8 in "Three Worlds Collide")

Today was the day.

The streets of ancient Earth were crowded to overbursting with people looking up at the sky, faces crowded up against windows.

Waiting for their sorrows to end.

Akon was looking down at their faces, from the balcony of a room in a well-guarded hotel. There were many who wished to initiate violence against him, which was understandable. Fear showed on most of the faces in the crowd, rage in some; a very few were smiling, and Akon suspected they might have simply given up on holding themselves together. Akon wondered what his own face looked like, right now.

The streets were less crowded than they might have been, only a few weeks earlier.

No one had told the Superhappies about that part. They'd sent an ambassadorial ship "in case you have any urgent requests we can help with", arriving hard on the heels of the Impossible. That ship had not been given any of the encryption keys to the human Net, nor allowed to land. It had made the Superhappies extremely suspicious, and the ambassadorial ship had disgorged a horde of tiny daughters to observe the rest of the human starline network -

But if the Superhappies knew, they would have tried to stop it. Somehow.

That was a price that no one was willing to include into the bargain, no matter what. There had to be that - alternative.

A quarter of the Impossible Possible World's crew had committed suicide, when the pact and its price became known. Others, Akon thought, had waited only to be with their families. The percentage on Earth... would probably be larger. The government, what was left of it, had refused to publish statistics. All you saw was the bodies being carried out of the apartments - in plain, unmarked boxes, in case the Superhappy ship was using optical surveillance.

Akon swallowed. The fear was already drying his own throat, the fear of changing, of becoming something else that wasn't quite him. He understood the urge to end that fear, at any price. And yet at the same time, he didn't, couldn't understand the suicides. Was being dead a smaller change? To die was not to leave the world, not to escape somewhere else; it was the simultaneous change of every piece of yourself into nothing.

Many parents had made that choice for their children. The government had tried to stop it. The Superhappies weren't going to like it, when they found out. And it wasn't right, when the children themselves wouldn't be so afraid of a world without pain. It wasn't as if the parents and children were going somewhere together. The government had done its best, issued orders, threatened confiscations - but there was only so much you could do to coerce someone who was going to die anyway.

So more often than not, they carried away the mother's body with her daughter's, the father with the son.

The survivors, Akon knew, would regret that far more vehemently, once they were closer to the Superhappy point of view.

Just as they would regret not eating the tiny bodies of the infants.

A hiss went up from the crowd, the intake of a thousand breaths. Akon looked up, and he saw in the sky the cloud of ships, dispersing from the direction of the Sun and the Huygens starline. Even at this distance they twinkled faintly. Akon guessed - and as one ship grew closer, he knew that he was right - that the Superhappy ships were no longer things of pulsating ugliness, but gently shifting iridescent crystal, designs that both a human and a Babyeater would find beautiful. The Superhappies had been swift to follow through on their own part of the bargain. Their new aesthetic senses would already be an intersection of three worlds' tastes.

The ship drew closer, overhead. It was quieter in the air than even the most efficient human ships, twinkling brightly and silently; the way that someone might imagine a star in the night sky would look close up, if they had no idea of the truth.

The ship stopped, hovering above the roads, between the buildings.

Other bright ships, still searching for their destinations, slid by overhead like shooting stars.

Long, graceful iridescent tendrils extended from the ship, down toward the crowd. One of them came toward his own balcony, and Akon saw that it was marked with the curves of a door.

The crowd didn't break, didn't run, didn't panic. The screams failed to spread, as the strong hugged the weak and comforted them. That was something to be proud of, in the last moments of the old humanity.

The tendril reaching for Akon halted just before him. The door marked at its end dilated open.

And wasn't it strange, now, the crowd was looking up at him.

Akon took a deep breath. He was afraid, but -

There wasn't much point in standing here, going on being afraid, experiencing futile disutility.

He stepped through the door, into a neat and well-lighted transparent capsule.

The door slid shut again.

Without a lurch, without a sound, the capsule moved up toward the alien ship.

One last time, Akon thought of all his fear, of the sick feeling in his stomach and the burning that was becoming a pain in his throat. He pinched himself on the arm, hard, very hard, and felt the warning signal telling him to stop.

Goodbye, Akon thought; and the tears began falling down his cheek, as though that one silent word had, for the very last time, broken his heart.

And he lived happily ever after.

Referenced by: Three Worlds Collide (0/8) • Three Worlds Decide (5/8)

Original with comments: Normal Ending: Last Tears (6/8)

True Ending: Sacrificial Fire (7/8)

Eliezer Yudkowsky, 05 February 2009 10:57AM

(Part 7 of 8 in "Three Worlds Collide")

Standing behind his target, unnoticed, the Ship's Confessor had produced from his sleeve the tiny stunner - the weapon which he alone on the ship was authorized to use, if he made a determination of outright mental breakdown. With a sudden motion, his arm swept outward -

- and anesthetized the Lord Akon.

Akon crumpled almost instantly, as though most of his strings had already been cut, and only a few last strands had been holding his limbs in place.

Fear, shock, dismay, sheer outright surprise: that was the Command Conference staring aghast at the Confessor.

From the hood came words absolutely forbidden to originate from that shadow: the voice of command. "Lord Pilot, take us through the starline back to the Huygens system. Get us moving now, you are on the critical path. Lady Sensory, I need you to enforce an absolute lockdown on all of this ship's communication systems except for a single channel under your direct control. Master of Fandom, get me proxies on the assets of every being on this ship. We are going to need capital."

For a moment, the Command Conference was frozen, voiceless and motionless, as everyone waited for someone else do to something.

And then -

"Moving the Impossible now, my lord," said the Lord Pilot. His face was sane once again. "What's your plan?"

"He is not your lord!" cried the Master of Fandom. Then his voice dropped. "Excuse me. Confessor - it did not appear to me that our Lord Administrator was insane. And you, of all people, cannot just seize power -"

"True," said the one, "Akon was sane. But he was also an honest man who would keep his word once he gave it, and that I could not allow. As for me - I have betrayed my calling three times over, and am no longer a Confessor." With that same response, the once-Confessor swept back the hood -

At any other time, the words and the move and the revealed face would have provoked shock to the point of fainting. On this day, with the whole human species at stake, it seemed merely interesting. Chaos had already run loose, madness was already unleashed into the world, and a little more seemed of little consequence.

"Ancestor," said the Master, "you are twice prohibited from exercising any power here."

The former Confessor smiled dryly. "Rules like that only exist within our own minds, you know. Besides," he added, "I am not steering the future of humanity in any real sense, just stepping in front of a bullet. That is not even advice, let alone an order. And it is... appropriate... that I, and not any of you, be the one who orders this thing done -"

"Fuck that up the ass with a hedge trimmer," said the Lord Pilot. "Are we going to save the human species or not?"

There was a pause while the others figured out the correct answer.

Then the Master sighed, and inclined his head in assent to the once-Confessor. "I shall follow your orders... kiritsugu."

Even the Kiritsugu flinched at that, but there was work to be done, and not much time in which to do it.

In the Huygens system, the Impossible Possible World was observed to return from its much-heralded expedition, appearing on the starline that had shown the unprecedented anomaly. Instantly, without a clock tick's delay, the Impossible broadcast a market order.

That was already a dozen ways illegal. If the Impossible had made a scientific discovery, it should have broadcast the experimental results openly before attempting to trade on them. Otherwise the result was not profit but chaos, as traders throughout the market refused to deal with you; just conditioning on the fact that you wanted to sell or buy from them, was reason enough for them not to. The whole market seized up as hedgers tried to guess what the hidden experimental results could have been, and which of their counterparties had private information.

The Impossible ignored the rules. It broadcast the specification of a new prediction contract, signed with EMERGENCY OVERRIDE and IMMINENT HARM and CONFESSOR FLAG - signatures that carried extreme penalties, up to total confiscation, for misuse; but any one of which ensured that the contract would appear on the prediction markets at almost the speed of the raw signal.

The Impossible placed an initial order on the contract backed by nearly the entire asset base of its crew.

The prediction's plaintext read:

In three hours and forty-one minutes, the starline between Huygens and Earth will become impassable.

Within thirty minutes after, every human being remaining in this solar system will die.

All passage through this solar system will be permanently denied to humans thereafter.

(The following plaintext is not intended to describe the contract's terms, but justifies why a probability estimate on the underlying proposition is of great social utility:

ALIENS. ANYONE WITH A STARSHIP, FILL IT WITH CHILDREN AND GO! GET OUT OF HUYGENS, NOW!)

In the Huygens system, there was almost enough time to draw a single breath.

And then the markets went mad, as every single trader tried to calculate the odds, and every married trader abandoned their positions and tried to get their children to a starport.

"Six," murmured the Master of Fandom, "seven, eight, nine, ten, eleven -"

A holo appeared within the Command Conference, a signal from the President of the Huygens Central Clearinghouse, requesting (or perhaps "demanding" would have been a better word) an interview with the Lord Administrator of the Impossible Possible World.

"Put it through," said the Lord Pilot, now sitting in Akon's chair as the figurehead anointed by the Kiritsugu.

"Aliens?" the President demanded, and then her eye caught the Pilot's uniform. "You're not an Administrator -"

"Our Lord Administrator is under sedation," said the Kiritsugu beside; he was wearing his Confessor's hood again, to save on explanations. "He placed himself under more stress than any of us -"

The President made an abrupt cutting gesture. "Explain this - contract. And if this is a market manipulation scheme, I'll see you all tickled until the last sun grows cold!"

"We followed the starline that showed the anomalous behavior," the Lord Pilot said, "and found that a nova had just occurred in the originating system. In other words, my Lady President, it was a direct effect of the nova and thus occurred on all starlines leading out of that system. We've never found aliens before now - but that's reflective of the probability of any single system we explore having been colonized. There might even be a starline leading out of this system that leads to an alien domain - but we have no way of knowing which one, and opening a new starline is expensive. The nova acted as a common rendezvous signal, my Lady President. It reflects the probability, not that we and the aliens encounter each other by direct exploration, but the probability that we have at least one neighboring world in common."

The President was pale. "And the aliens are hostile."

The Lord Pilot involuntarily looked to the Kiritsugu.

"Our values are incompatible," said the Kiritsugu.

"Yes, that's one way of putting it," said the Lord Pilot. "And unfortunately, my Lady President, their technology is considerably in advance of ours."

"Lord... Pilot," the President said, "are you certain that the aliens intend to wipe out the human species?"

The Lord Pilot gave a very thin, very flat smile. "Incompatible values, my Lady President. They're quite skilled with biotechnology. Let's leave it at that."

Sweat was running down the President's forehead. "And why did they let you go, then?"

"We arranged for them to be told a plausible lie," the Lord Pilot said simply. "One of the reasons they're more advanced than us is that they're not very good at deception."

"None of this," the President said, and now her voice was trembling, "none of this explains why the starline between Huygens and Earth will become impassable. Surely, if what you say is true, the aliens will pour through our world, and into Earth, and into the human starline network. Why do you think that this one starline will luckily shut down?"

The Lord Pilot drew a breath. It was good form to tell the exact truth when you had something to hide. "My Lady President, we encountered two alien species at the nova. The first species exchanged scientific information with us. It is the second species that we are running from. But, from the first species, we learned a fact which this ship can use to shut down the Earth starline. For obvious reasons, my Lady President, we do not intend to share this fact publicly. That portion of our final report will be encrypted to the Chair of the Interstellar Association for the Advancement of Science, and to no other key."

The President started laughing. It was wild, hysterical laughter that caused the Kiritsugu's hood to turn toward her. From the corner of the screen, a gloved hand entered the view; the hand of the President's own Confessor. "My lady..." came a soft female voice.

"Oh, very good," the President said. "Oh, marvelous. So it's your ship that's going to be responsible for this catastrophe. You admit that, eh? I'm amazed. You probably managed to avoid telling a single direct lie. You plan to blow up our star and kill fifteen billion people, and you're trying to stick to the literal truth."

The Lord Pilot slowly nodded. "When we compared the first aliens' scientific database to our own -"

"No, don't tell me. I was told it could be done by a single ship, but I'm not supposed to know how. Astounding that an alien species could be so peaceful they don't even consider that a secret. I think I would like to meet these aliens. They sound much nicer than the other ones - why are you laughing?"

"My Lady President," the Lord Pilot said, getting a grip on himself, "forgive me, we've been through a lot. Excuse me for asking, but are you evacuating the planet or what?"

The President's gaze suddenly seemed sharp and piercing like the fire of stars. "It was set in motion instantly, of course. No comparable harm done, if you're wrong. But three hours and forty-one minutes is not enough time to evacuate ten percent of this planet's children." The President's eyes darted at something out of sight. "With eight hours, we could call in ships from the Earth nexus and evacuate the whole planet."

"My lady," a soft voice came from behind the President, "it is the whole human species at stake. Not just the entire starline network beyond Earth, but the entire future of humanity. Any incrementally higher probability of the aliens arriving within that time -"

The President stood in a single fluid motion that overturned her chair, moving so fast that the viewpoint bobbed as it tried to focus on her and the shadow-hooded figure standing beside. "Are you telling me," she said, and her voice rose to a scream, "to shut up and multiply?"

"Yes."

The President turned back to the camera angle, and said simply, "No. You don't know the aliens are following that close behind you - do you? We don't even know if you can shut down the starline! No matter what your theory predicts, it's never been tested - right? What if you create a flare bright enough to roast our planet, but not explode the whole sun? Billions would die, for nothing! So if you do not promise me a minimum of - let's call it nine hours to finish evacuating this planet - then I will order your ship destroyed before it can act."

No one from the Impossible spoke.

The President's fist slammed her desk. "Do you understand me? Answer! Or in the name of Huygens, I will destroy your ship -"

Her Confessor caught her President's body, very gently supporting it as it collapsed.

Even the Lord Pilot was pale and silent. But that, at least, had been within law and tradition; no one could have called that thinking sane.

On the display, the Confessor bowed her hood. "I will inform the markets that the Lady President was driven unstable by your news," she said quietly, "and recommend to the government that they carry out the evacuation without asking further questions of your ship. Is there anything else you wish me to tell them?" Her hood turned slightly, toward the Kiritsugu. "Or tell me?"

There was a strange, quick pause, as the shadows from within the two hoods stared at each other.

Then: "No," replied the Kiritsugu. "I think it has all been said."

The Confessor's hood nodded. "Goodbye."

"There it goes," the Ship's Engineer said. "We have a complete, stable positive feedback loop."

On screen was the majesty that was the star Huygens, of the inhabited planet Huygens IV. Overlaid in false color was the recirculating loop of Alderson forces which the Impossible had steadily fed.

Fusion was now increasing in the star, as the Alderson forces encouraged nuclear barriers to break down; and the more fusions occurred, the more Alderson force was generated. Round and round it went. All the work of the Impossible, the full frantic output of their stardrive, had only served to subtly steer the vast forces being generated; nudge a fraction into a circle rather than a line. But now -

Did the star brighten? It was only their imagination, they knew. Photons take centuries to exit a sun, under normal circumstances. The star's core was trying to expand, but it was expanding too slowly - all too slowly - to outrun the positive feedback that had begun.

"Multiplication factor one point oh five," the Engineer said. "It's climbing faster now, and the loop seems to be intact. I think we can conclude that this operation is going to be... successful. One point two."

"Starline instability detected," the Lady Sensory said.

Ships were still disappearing in frantic waves on the starline toward Earth. Still connected to the Huygens civilization, up to the last moment, by tiny threads of Alderson force.

"Um, if anyone has anything they want to add to our final report," the Ship's Engineer said, "they've got around ten seconds."

"Tell the human species from me -" the Lord Pilot said.

"Five seconds."

The Lord Pilot shouted, fist held high and triumphant: "To live, and occasionally be unhappy!"

This concludes
the full and final report
of the Impossible Possible World.

(To be completed.)

Referenced by: Three Worlds Collide (0/8) • Three Worlds Decide (5/8)

Original with comments: True Ending: Sacrificial Fire (7/8)

Epilogue: Atonement (8/8)

Eliezer Yudkowsky, 06 February 2009 11:52AM

(Part 8 of 8 in "Three Worlds Collide")

Fire came to Huygens.

The star erupted.

Stranded ships, filled with children doomed by a second's last delay, still milled around the former Earth transit point. Too many doomed ships, far too many doomed ships. They should have left a minute early, just to be sure; but the temptation to load in that one last child must have been irresistable. To do the warm and fuzzy thing just this one time, instead of being cold and calculating. You couldn't blame them, could you...?

Yes, actually, you could.

The Lady Sensory switched off the display. It was too painful.

On the Huygens market, the price of a certain contract spiked to 100%. They were all rich in completely worthless assets for the next nine minutes, until the supernova blast front arrived.

"So," the Lord Pilot finally said. "What kind of asset retains its value in a market with nine minutes to live?"

"Booze for immediate delivery," the Master of Fandom said promptly. "That's what you call a -"

"Liquidity preference," the others chorused.

The Master laughed. "All right, that was too obvious. Well... chocolate, sex -"

"Not necessarily," said the Lord Pilot. "If you can use up the whole supply of chocolate at once, does demand outstrip supply? Same with sex - the value could actually drop if everyone's suddenly willing. Not to mention: Nine minutes?"

"All right then, expert oral sex from experienced providers. And hard drugs with dangerous side effects; the demand would rise hugely relative to supply -"

"This is inane," the Ship's Engineer commented.

The Master of Fandom shrugged. "What do you say in the unrecorded last minutes of your life that is not inane?"

"It doesn't matter," said the Lady Sensory. Her face was strangely tranquil. "Nothing that we do now matters. We won't have to live with the consequences. No one will. All this time will be obliterated when the blast front hits. The role I've always played, the picture that I have of me... it doesn't matter. There's... a peace... in not having to be Dalia Ancromein any more."

The others looked at her. Talk about killing the mood.

"Well," the Master of Fandom said, "since you raise the subject, I suppose it would be peaceful if not for the screaming terror."

"You don't have to feel the screaming terror," the Lady Sensory said. "That's just a picture you have in your head of how it should be. The role of someone facing imminent death. But I don't have to play any more roles. I don't have to feel screaming terror. I don't have to frantically pack in a few last moments of fun. There are no more obligations."

"Ah," the Master of Fandom said, "so I guess this is when we find out who we really are." He paused for a moment, then shrugged. "I don't seem to be anyone in particular. Oh well."

The Lady Sensory stood up, and walked across the room to where the Lord Pilot stood looking at the viewscreen.

"My Lord Pilot," the Lady Sensory said.

"Yes?" the Lord Pilot said. His face was expectant.

The Lady Sensory smiled. It was bizarre, but not frightening. "Do you know, my Lord Pilot, that I had often thought how wonderful it would be to kick you very hard in the testicles?"

"Um," the Lord Pilot said. His arms and legs suddenly tensed, preparing to block.

"But now that I could do it," the Lady Sensory said, "I find that I don't really want to. It seems... that I'm not as awful a person as I thought." She gave a brief sigh. "I wish that I had realized it earlier."

The Lord Pilot's hand swiftly darted out and groped the Lady Sensory's breast. It was so unexpected that no one had time to react, least of all her. "Well, what do you know," the Pilot said, "I'm just as much of a pervert as I thought. My self-estimate was more accurate than yours, nyah nyah -"

The Lady Sensory kneed him in the groin, hard enough to drop him moaning to the floor, but not hard enough to require medical attention.

"Okay," the Master of Fandom said, "can we please not go down this road? I'd like to die with at least some dignity."

There was a long, awkward silence, broken only by a quiet "Ow ow ow ow..."

"Would you like to hear something amusing?" asked the Kiritsugu, who had once been a Confessor.

"If you're going to ask that question," said the Master of Fandom, "when the answer is obviously yes, thus wasting a few more seconds -"

"Back in the ancient days that none of you can imagine, when I was seventeen years old - which was underage even then - I stalked an underage girl through the streets, slashed her with a knife until she couldn't stand up, and then had sex with her before she died. It was probably even worse than you're imagining. And deep down, in my very core, I enjoyed every minute."

Silence.

"I don't think of it often, mind you. It's been a long time, and I've taken a lot of intelligence-enhancing drugs since then. But still - I was just thinking that maybe what I'm doing now finally makes up for that."

"Um," said the Ship's Engineer. "What we just did, in fact, was kill fifteen billion people."

"Yes," said the Kiritsugu, "that's the amusing part."

Silence.

"It seems to me," mused the Master of Fandom, "that I should feel a lot worse about that than I actually do."

"We're in shock," the Lady Sensory observed distantly. "It'll hit us in about half an hour, I expect."

"I think it's starting to hit me," the Ship's Engineer said. His face was twisted. "I - I was so worried I wouldn't be able to destroy my home planet, that I didn't get around to feeling unhappy about succeeding until now. It... hurts."

"I'm mostly just numb," the Lord Pilot said from the floor. "Well, except down there, unfortunately." He slowly sat up, wincing. "But there was this absolute unalterable thing inside me, screaming so loud that it overrode everything. I never knew there was a place like that within me. There wasn't room for anything else until humanity was safe. And now my brain is worn out. So I'm just numb."

"Once upon a time," said the Kiritsugu, "there were people who dropped a U-235 fission bomb, on a place called Hiroshima. They killed perhaps seventy thousand people, and ended a war. And if the good and decent officer who pressed that button had needed to walk up to a man, a woman, a child, and slit their throats one at a time, he would have broken long before he killed seventy thousand people."

Someone made a choking noise, as if trying to cough out something that had suddenly lodged deep in their throat.

"But pressing a button is different," the Kiritsugu said. "You don't see the results, then. Stabbing someone with a knife has an impact on you. The first time, anyway. Shooting someone with a gun is easier. Being a few meters further away makes a surprising difference. Only needing to pull a trigger changes it a lot. As for pressing a button on a spaceship - that's the easiest of all. Then the part about 'fifteen billion' just gets flushed away. And more importantly - you think it was the right thing to do. The noble, the moral, the honorable thing to do. For the safety of your tribe. You're proud of it -"

"Are you saying," the Lord Pilot said, "that it was not the right thing to do?"

"No," the Kiritsugu said. "I'm saying that, right or wrong, the belief is all it takes."

"I see," said the Master of Fandom. "So you can kill billions of people without feeling much, so long as you do it by pressing a button, and you're sure it's the right thing to do. That's human nature." The Master of Fandom nodded. "What a valuable and important lesson. I shall remember it all the rest of my life."

"Why are you saying all these things?" the Lord Pilot asked the Kiritsugu.

The Kiritsugu shrugged. "When I have no reason left to do anything, I am someone who tells the truth."

"It's wrong," said the Ship's Engineer in a small, hoarse voice, "I know it's wrong, but - I keep wishing the supernova would hurry up and get here."

"There's no reason for you to hurt," said the Lady Sensory in a strange calm voice. "Just ask the Kiritsugu to stun you. You'll never wake up."

"...no."

"Why not?" asked the Lady Sensory, in a tone of purely abstract curiosity.

The Ship's Engineer clenched his hands into fists. "Because if hurting is that much of a crime, then the Superhappies are right." He looked at the Lady Sensory. "You're wrong, my lady. These moments are as real as every other moment of our lives. The supernova can't make them not exist." His voice lowered. "That's what my cortex says. My diencephalon wishes we'd been closer to the sun."

"It could be worse," observed the Lord Pilot. "You could not hurt."

"For myself," the Kiritsugu said quietly, "I had already visualized and accepted this, and then it was just a question of watching it play out." He sighed. "The most dangerous truth a Confessor knows is that the rules of society are just consensual hallucinations. Choosing to wake up from the dream means choosing to end your life. I knew that when I stunned Akon, even apart from the supernova."

"Okay, look," said the Master of Fandom, "call me a gloomy moomy, but does anyone have something uplifting to say?"

The Lord Pilot jerked a thumb at the expanding supernova blast front, a hundred seconds away. "What, about that?"

"Yeah," the Master of Fandom said. "I'd like to end my life on an up note."

"We saved the human species," offered the Lord Pilot. "Man, that's the sort of thing you could just repeat to yourself over and over and over again -"

"Besides that."

"Besides WHAT?"

The Master managed to hold a straight face for a few seconds, and then had to laugh.

"You know," the Kiritsugu said, "I don't think there's anyone in modern-day humanity, who would regard my past self as anything but a poor, abused victim. I'm pretty sure my mother drank during pregnancy, which, back then, would give your child something called Fetal Alcohol Syndrome. I was poor, uneducated, and in an environment so entrepreneurially hostile you can't even imagine it -"

"This is not sounding uplifting," the Master said.

"But somehow," the Kiritsugu said, "all those wonderful excuses - I could never quite believe in them myself, afterward. Maybe because I'd also thought of some of the same excuses before. It's the part about not doing anything that got to me. Others fought the war to save the world, far over my head. Lightning flickering in the clouds high above me, while I hid in the basement and suffered out the storm. And by the time I was rescued and healed and educated, in any shape to help others - the battle was essentially over. Knowing that I'd been a victim for someone else to save, one more point in someone else's high score - that just stuck in my craw, all those years..."

"...anyway," the Kiritsugu said, and there was a small, slight smile on that ancient face, "I feel better now."

"So does that mean," asked the Master, "that now your life is finally complete, and you can die without any regrets?"

The Kiritsugu looked startled for a moment. Then he threw back his head and laughed. True, pure, honest laughter. The others began to laugh as well, and their shared hilarity echoed across the room, as the supernova blast front approached at almost exactly the speed of light.

Finally the Kiritsugu stopped laughing, and said:

"Don't be ridicu-"

Referenced by: Three Worlds Collide (0/8) • True Ending: Sacrificial Fire (7/8) • (Moral) Truth in Fiction?

Original with comments: Epilogue: Atonement (8/8)

The Thing That I Protect

Eliezer Yudkowsky, 07 February 2009 07:18PM

Followup to: Something to Protect, Value is Fragile

"Something to Protect" discursed on the idea of wielding rationality in the service of something other than "rationality". Not just that rationalists ought to pick out a Noble Cause as a hobby to keep them busy; but rather, that rationality itself is generated by having something that you care about more than your current ritual of cognition.

So what is it, then, that I protect?

I quite deliberately did not discuss that in "Something to Protect", leaving it only as a hanging implication. In the unlikely event that we ever run into aliens, I don't expect their version of Bayes's Theorem to be mathematically different from ours, even if they generated it in the course of protecting different and incompatible values. Among humans, the idiom of having "something to protect" is not bound to any one cause, and therefore, to mention my own cause in that post would have harmed its integrity. Causes are dangerous things, whatever their true importance; I have written somewhat on this, and will write more about it.

But still - what is it, then, the thing that I protect?

Friendly AI? No - a thousand times no - a thousand times not anymore. It's not thinking of the AI that gives me strength to carry on even in the face of inconvenience.

I would be a strange and dangerous AI wannabe if that were my cause - the image in my mind of a perfected being, an existence greater than humankind. Maybe someday I'll be able to imagine such a child and try to build one, but for now I'm too young to be a father.

Those of you who've been following along recent discussions, particularly "Value is Fragile", might have noticed something else that I might, perhaps, hold precious. Smart agents want to protect the physical representation of their utility function for almost the same reason that male organisms are built to be protective of their testicles. From the standpoint of the alien god, natural selection, losing the germline - the gene-carrier that propagates the pattern into the next generation - means losing almost everything that natural selection cares about. Unless you already have children to protect, can protect relatives, etcetera - few are the absolute and unqualified statements that can be made in evolutionary biology - but still, if you happen to be a male human, you will find yourself rather protective of your testicles; that one, centralized vulnerability is why a kick in the testicles hurts more than being hit on the head.

To lose the pattern of human value - which, for now, is physically embodied only in the human brains that care about those values - would be to lose the Future itself; if there's no agent with those values, there's nothing to shape a valuable Future.

And this pattern, this one most vulnerable and precious pattern, is indeed at risk to be distorted or destroyed.  Growing up is a hard problem either way, whether you try to edit existing brains, or build de novo Artificial Intelligence that mirrors human values. If something more powerful than humans, and not sharing human values, comes into existence - whether by de novo AI gone wrong, or augmented humans gone wrong - then we can expect to lose, hard. And value is fragile; losing just one dimension of human value can destroy nearly all of the utility we expect from the future.

So is that, then, the thing that I protect?

If it were - then what inspired me when times got tough would be, say, thinking of people being nice to each other. Or thinking of people laughing, and contemplating how humor probably exists among only an infinitesimal fraction of evolved intelligent species and their descendants. I would marvel at the power of sympathy to make us feel what others feel -

But that's not quite it either.

I once attended a small gathering whose theme was "This I Believe". You could interpret that phrase in a number of ways; I chose "What do you believe that most other people don't believe which makes a corresponding difference in your behavior?" And it seemed to me that most of how I behaved differently from other people boiled down to two unusual beliefs. The first belief could be summarized as "intelligence is a manifestation of order rather than chaos"; this accounts both for my attempts to master rationality, and my attempt to wield the power of AI.

And the second unusual belief could be summarized as: "Humanity's future can be a WHOLE LOT better than its past."

Not desperately darwinian robots surging out to eat as much of the cosmos as possible, mostly ignoring their own internal values to try and grab as many stars as possible, with most of the remaining matter going into making paperclips.

Not some bittersweet ending where you and I fade away on Earth while the inscrutable robots ride off into the unknowable sunset, having grown beyond such merely human values as love or sympathy.

Screw bittersweet. To hell with that melancholy-tinged crap. Why leave anyone behind? Why surrender a single thing that's precious?

(And the compromise-futures are all fake anyway; at this difficulty level, you steer precisely or you crash.)

The pattern of fun is also lawful. And, though I do not know all the law - I do think that written in humanity's value-patterns is the implicit potential of a happy future. A seriously goddamn FUN future. A genuinely GOOD outcome. Not something you'd accept with a sigh of resignation for nothing better being possible. Something that would make you go "WOOHOO!"

In the sequence on Fun Theory, I have given you, I hope, some small reason to believe that such a possibility might be consistently describable, if only it could be made real. How to read that potential out of humans and project it into reality... might or might not be as simple as "superpose our extrapolated reflected equilibria". But that's one way of looking at what I'm trying to do - to reach the potential of the GOOD outcome, not the melancholy bittersweet compromise. Why settle for less?

To really have something to protect, it has to be able to bring tears to your eyes. That, generally, requires something concrete to visualize - not just abstract laws. Reading the Laws of Fun doesn't bring tears to my eyes. I can visualize a possibility or two that makes sense to me, but I don't know if it would make sense to others the same way.

What does bring tears to my eyes? Imagining a future where humanity has its act together. Imagining children who grow up never knowing our world, who don't even understand it. Imagining the rescue of those now in sorrow, the end of nightmares great and small. Seeing in reality the real sorrows that happen now, so many of which are unnecessary even now. Seeing in reality the signs of progress toward a humanity that's at least trying to get its act together and become something more - even if the signs are mostly just symbolic: a space shuttle launch, a march that protests a war.

(And of course these are not the only things that move me. Not everything that moves me has to be a Cause. When I'm listening to e.g. Bach's Jesu: Joy of Man's Desiring, I don't think about how every extant copy might be vaporized if things go wrong. That may be true, but it's not the point. It would be as bad as refusing to listen to that melody because it was once inspired by belief in the supernatural.)

To really have something to protect, you have to be able to protect it, not just value it. My battleground for that better Future is, indeed, the fragile pattern of value. Not to keep it in stasis, but to keep it improving under its own criteria rather than randomly losing information. And then to project that through more powerful optimization, to materialize the valuable future. Without surrendering a single thing that's precious, because losing a single dimension of value could lose it all.

There's no easy way to do this, whether by de novo AI or by editing brains. But with a de novo AI, cleanly and correctly designed, I think it should at least be possible to get it truly right and win completely. It seems, for all its danger, the safest and easiest and shortest way (yes, the alternatives really are that bad). And so that is my project.

That, then, is the service in which I wield rationality. To protect the Future, on the battleground of the physical representation of value. And my weapon, if I can master it, is the ultimate hidden technique of Bayescraft - to explicitly and fully know the structure of rationality, to such an extent that you can shape the pure form outside yourself - what some call "Artificial General Intelligence" and I call "Friendly AI". Which is, itself, a major unsolved research problem, and so it calls into play the more informal methods of merely human rationality. That is the purpose of my art and the wellspring of my art.

That's pretty much all I wanted to say here about this Singularity business...

...except for one last thing; so after tomorrow, I plan to go back to posting about plain old rationality on Monday.

Referenced by: ...And Say No More Of It

Original with comments: The Thing That I Protect

...And Say No More Of It

Eliezer Yudkowsky, 09 February 2009 12:15AM

Followup to: The Thing That I Protect

Anything done with an ulterior motive has to be done with a pure heart. You cannot serve your ulterior motive, without faithfully prosecuting your overt purpose as a thing in its own right, that has its own integrity. If, for example, you're writing about rationality with the intention of recruiting people to your utilitarian Cause, then you cannot talk too much about your Cause, or you will fail to successfully write about rationality.

This doesn't mean that you never say anything about your Cause, but there's a balance to be struck. "A fanatic is someone who can't change his mind and won't change the subject."

In previous months, I've pushed this balance too far toward talking about Singularity-related things. And this was for (first-order) selfish reasons on my part; I was finally GETTING STUFF SAID that had been building up painfully in my brain for FRICKIN' YEARS. And so I just kept writing, because it was finally coming out. For those of you who have not the slightest interest, I'm sorry to have polluted your blog with that.

When Less Wrong starts up, it will, by my own request, impose a two-month moratorium on discussion of "Friendly AI" and other Singularity/intelligence explosion-related topics.

There's a number of reasons for this. One of them is simply to restore the balance. Another is to make sure that a forum intended to have a more general audience, doesn't narrow itself down and disappear.

But more importantly - there are certain subjects which tend to drive people crazy, even if there's truth behind them. Quantum mechanics would be the paradigmatic example; you don't have to go funny in the head but a lot of people do. Likewise Godel's Theorem, consciousness, Artificial Intelligence -

The concept of "Friendly AI" can be poisonous in certain ways. True or false, it carries risks to mental health. And not just the obvious liabilities of praising a Happy Thing. Something stranger and subtler that drains enthusiasm.

If there were no such problem as Friendly AI, I would probably be devoting more or less my entire life to cultivating human rationality; I would have already been doing it for years.

And though I could be mistaken - I'm guessing that I would have been much further along by now.

Partially, of course, because it's easier to tell people things that they're already prepared to hear. "Rationality" doesn't command universal respect, but it commands wide respect and recognition. There is already the New Atheist movement, and the Bayesian revolution; there are already currents flowing in that direction.

One has to be wary, in life, of substituting easy problems for hard problems. This is a form of running away. "Life is what happens to you while you are making other plans", and it takes a very strong and non-distractable focus to avoid that...

But I'd been working on directly launching a Singularity movement for years, and it just wasn't getting traction. At some point you also have to say, "This isn't working the way I'm doing it," and try something different.

There are many ulterior motives behind my participation in Overcoming Bias / Less Wrong. One of the simpler ones is the idea of "First, produce rationalists - people who can shut up and multiply - and then, try to recruit some of them." Not all. You do have to care about the rationalist community for its own sake. You have to be willing not to recruit all the rationalists you create. The first rule of acting with ulterior motives is that it must be done with a pure heart, faithfully serving the overt purpose.

But more importantly - the whole thing only works if the strange intractability of the direct approach - the mysterious slowness of trying to build an organization directly around the Singularity - does not contaminate the new rationalist movement.

There's an old saw about the lawyer who works in a soup kitchen for an hour in order to purchase moral satisfaction, rather than work the same hour at the law firm and donate the money to hire 5 people to work at the soup kitchen. Personal involvement isn't just pleasurable, it keeps people involved; the lawyer is more likely to donate real money to the soup kitchen later. Research problems don't have a lot of opportunity for outsiders to get personally involved, including FAI research. (This is why scientific research isn't usually supported by individuals, I suspect; instead scientists fight over the division of money that has been block-allocated by governments and foundations. I should write about this later.)

If it were the Cause of human rationality - if that had always been the purpose I'd been pursuing - then there would have been all sorts of things people could have done to personally help out, to keep their spirits high and encourage them to stay involved. Writing letters to the editor, trying to get heuristics and biases taught in organizations and in classrooms; holding events, handing out flyers; starting a magazine, increasing the number of subscribers; students handing out copies of the "Twelve Virtues of Rationality" at campus events...

It might not be too late to start going down that road - but only if the "Friendly AI" meme doesn't take over and suck out the life and motivation.

In a purely utilitarian sense - the sort of thinking that would lead a lawyer to actually work that extra hour at the law firm and donate the money - someone who thinks that handing out flyers is important to the Cause of human rationality, should be strictly less enthusiastic than someone who thinks that handing out flyers for human rationality has directly rationality-related benefits and might help a Friendly AI project. It's a strictly added benefit; it should result in strictly more enthusiasm...

But in practice - it's as though the idea of "Friendly AI" exerts an attraction that sucks the emotional energy out of its own subgoals.

You only press the "Run" button after you finish coding and teaching a Friendly AI; which happens after the theory has been worked out; which happens after theorists have been recruited; which happens after (a) mathematically smart people have comprehended cognitive naturalism on a deep gut level and (b) a regular flow of funding exists to support these professional specialists; which first requires that the whole project get sufficient traction; for which handing out flyers may be involved...

But something about the fascination of finally building the AI, seems to make all the mere preliminaries pale in emotional appeal. Or maybe it's that the actual researching takes on an aura of the sacred magisterium, and then it's impossible to scrape up enthusiasm for any work outside the sacred magisterium.

If you're handing out flyers for the Cause of human rationality... it's not about a faraway final goal that makes the mere work seem all too mundane by comparison, and there isn't a sacred magisterium that you're not part of.

And this is only a brief gloss on the mental health risks of "Friendly AI"; there are others I haven't even touched on, though the others are relatively more obvious. Import morality.crazy, import metaethics.crazy, import AI.crazy, import Noble Cause.crazy, import Happy Agent.crazy, import Futurism.crazy, etcetera.

But it boils down to this: From my perspective, my participation in Overcoming Bias / Less Wrong has many different ulterior motives, and many different helpful potentials, many potentially useful paths leading out of it. But the meme of Friendly AI potentially poisons many of those paths, if it interacts in the wrong way; and so the ability to shut up about the Cause is more than usually important, here. Not shut up entirely - but the rationality part of it needs to have its own integrity. Part of protecting that integrity is to not inject comments about "Friendly AI" into any post that isn't directly about "Friendly AI".

I would like to see "Friendly AI" be a rationalist Cause sometimes discussed on Less Wrong, alongside other rationalist Causes whose members likewise hang out there for companionship and skill acquisition. This is as much as is necessary to recruit a fraction of the rationalists created. Anything more would poison the community, I think. Trying to find hooks to steer every arguably-related conversation toward your own Cause is not virtuous, it is dangerously and destructively greedy. All Causes represented on LW will have to bear this in mind, on pain of their clever conversational hooks being downvoted to oblivion.

And when Less Wrong starts up, its integrity will be protected in a simpler way: shut up about the Singularity entirely for two months.

...and that's it.

Back to rationality.

WHEW.

(This would be a great time to announce that Less Wrong is ready to go, but they're still working on it. Possibly later this week, possibly not.)

Referenced by: Less Meta • The End (of Sequences)

Original with comments: ...And Say No More Of It

(Moral) Truth in Fiction?

Eliezer Yudkowsky, 09 February 2009 05:26PM

A comment by Anonymous on Three Worlds Collide:

After reading this story I feel myself agreeing with Eliezer more on his views and that seems to be a sign of manipulation and not of rationality.

Philosophy expressed in form of fiction seems to have a very strong effect on people - even if the fiction isn't very good (ref. Ayn Rand).

Robin has similar qualms:

Since people are inconsistent but reluctant to admit that fact, their moral beliefs can be influenced by which moral dilemmas they consider in what order, especially when written by a good writer. I expect Eliezer chose his dilemmas in order to move readers toward his preferred moral beliefs, but why should I expect those are better moral beliefs than those of all the other authors of fictional moral dilemmas?

If I'm going to read a literature that might influence my moral beliefs, I'd rather read professional philosophers and other academics making more explicit arguments.

Irepliedthat Ihadtaken considerable pains to set out theexplicit argumentsbefore daring to publish the story. And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light. (The opposing viewpoint is the counterpart of the villain; you want it to lookas reasonable as possiblefor purposes of dramatic conflict, the same principle whereby Frodo confronts the Dark Lord Sauron rather than a cockroach.)

Robindidn'tfind this convincing:

I don't think readers should much let down their guard against communication modes where sneaky persuasion is more feasible simply because the author has made some more explicit arguments elsewhere... Academic philosophy offers exemplary formats and styles for low-sneak ways to argue about values.

I think that this understates the power and utility of fiction. I once read a book that was called something like "How to Read" (no, not "How to Read a Book") which said that nonfiction was about communicatingknowledge,while fiction was about communicatingexperience.

If I want to communicate something about theexperienceof being a rationalist, I can best do it by writing a short story with a rationalist character. Not only would identical abstract statements about proper responses haveless impact, they wouldn't even communicate thesame thought.

FromThe Failures of Eld Science:

"...Work expands to fill the time allotted, as the saying goes. But people can think important thoughts in far less than thirty years, if theyexpectspeed of themselves." Jeffreyssai suddenly slammed down a hand on the arm of Brennan's chair. "How long do you have to dodge a thrown knife?"

"Very little time, sensei!"

"Less than a second! Two opponents are attacking you! How long do you have to guess who's more dangerous?"

"Less than a second, sensei!"

"The two opponents have split up and are attacking two of your girlfriends! How long do you have to decide which one you truly love?"

"Less than a second, sensei!"

"A new argument shows your precious theory is flawed! How long does it take you to change your mind?"

"Less than a second, sensei!"

"WRONG! DON'T GIVE ME THE WRONG ANSWER JUST BECAUSE IT FITS A CONVENIENT PATTERN AND I SEEM TO EXPECT IT OF YOU! How long does it really take, Brennan?"

Sweat was forming on Brennan's back, but he stopped and actually thought about it -

"ANSWER, BRENNAN!"

"No sensei! I'm not finished thinking sensei! An answer would be premature! Sensei!"

"Very good! Continue! But don't take thirty years!"

This is an experience about how to avoidcompleting the patternwhen the pattern happens to be blatantly wrong, and how to think quickly without thinkingtooquickly.

Forget the question of whether you can write the equivalent abstract argument that communicates the same thought inless space. Can you do it atall? Is thereanyseries of abstract arguments that creates the same learning experience in the reader? Entering a series of believed propositions into your belief pool is not the same as feeling yourself in someone else's shoes, and reacting to the experience, and forming an experiential skill-memory of how to do it next time.

And it seems to me that to communicate experience is avalid form of moral argumentas well.

Uncle Tom's Cabinwas not just a historically powerful argument against slavery, it was avalidargument against slavery. If human beings were constructed withoutmirror neurons, if we didn't hurt when we see a nonenemy hurting, then we would exist in the reference frame of a different morality, and we would decide what to do by asking a different question, "What should* we do?" Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves. (SeeInseparably RightandNo License To Be Human.)

Putting someone into the shoes of a slave and letting their mirror neurons feel the suffering of a husband separated from a wife, a mother separated from a child, a man whipped for refusing to whip a fellow slave - it's not just persuasive, it'svalid. It fires the mirror neurons that physically implement that part of our moral frame.

I'm sure many have turned against slavery without readingUncle Tom's Cabin- maybe even due to purely abstract arguments, without ever seeing the carving "Am I Not a Man and a Brother?" But for some people, orfor a not-much-different intelligent species, readingUncle Tom's Cabinmight be theonlyargument that can turn you against slavery. Any amount of abstract argument thatdidn'tfire the experiential mirror neurons, would not activate the part of your implicit should-function that disliked slavery. You would just seem to be making a good profit on something you owned.

Can fiction be abused? Of course. Suppose that blacks hadno subjective experiences. ThenUncle Tom's Cabinwould have been a lie in a deeper sense than being fictional, and anyone moved by it would have been deceived.

Or to give a more subtle case not involving a direct "lie" of this sort: On the SL4 mailing list, Stuart Armstrong posted an argument against TORTURE in the infamousTorture vs. Dust Specksdebate, consisting ofa short story describing the fate of the person to be tortured. Myreplywas that the appropriate counterargument would be3^^^3stories about someone getting a dust speck in their eye. I actually did try to send a long message consisting only of

DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK

for a thousand lines or so, but the mailing software stopped it. (Ideally, I should have created a webpage using Javascript and bignums, that, if run on a sufficiently large computer, would print out exactly 3^^^3 copies of a brief story about someone getting a dust speck in their eye. It probably would have been the world's longest finite webpage. Alas, I lack time for many of my good ideas.)

Then there's the sort of standard polemic used in e.g.Atlas Shrugged(as well as many less famous pieces of science fiction) in which Your Beliefs are put into the minds of strong empowered noble heroes, and the Opposing Beliefs are put into the mouths of evil and contemptible villains, and then the consequences of Their Way are depicted as uniformly disastrous while Your Way offers butterflies and apple pie. That's not even subtle, but it works on people predisposed to hear the message.

But to entirely turn your back on fiction is, I think, taking it too far. Abstract argument can be abused too. In fact, I would say that abstract argument is if anythingeasierto abuse because it hasmoredegrees of freedom. Which is easier, to say "Slavery is good for the slave", or to write a believable story about slavery benefiting the slave? Youcando both, but the second is at least moredifficult; your brain is more likely to notice the non-sequiturs when they're played out as a written experience.

Stories may not get us completely intoNear mode, but they get uscloserinto Near mode than abstract argument. If it's words on paper, you can end up believing that yououghtto do just about anything. If you're in the shoes of a character encountering the experience, your reactions may be harder to twist.

Contrast a verbal argument against the verbal belief that "non-Catholics go to Hell"; versus reading a story about a good and decent person, who happens to be a Protestant, and dies trying to save a child's life, who is condemned to hell and has molten lead poured down her throat; versus the South Park episode where a crowd of newly dead souls is at the entrance to hell, and the Devil says, "Sorry, it was the Mormons" and everyone goes "Awwwww..."

Yes, abstraction donerightcan keep you going where concrete visualization breaks down - the torture vs. dust specks thing being an archetypal example; you can't actually visualize that many dust specks, but if you try to choose SPECKS you'll end up withcircular preferences. But so far as I can organize my metaethics, the ground level of morality lies in our preferences over particular, concrete situations - and when these can be comprehended as concrete images at all, it's best to visualize them as concretely as possible. Unless we know specifically where the concrete image is going wrong, and have to apply an abstract correction. The moral abstraction is built ontopof the ground level.

I am also, of course, worried about the idea that stories aren't "respectable" because they don't look sufficiently solemn and dull; or the idea that something isn't "respectable" if can beunderstoodby a mere popular audience. Yes, there are technical fields that are genuinely impossible to explain to your grandmother in an hour; but ceteris paribus, people who can write at a more popular levelwithoutdistorting technical reality are performing a huge service to that field. I've heard that Carl Sagan was held in some disrepute by his peers for the crime of speaking to the general public. If true, this is merely stupid.

Explaining things ishard. Explainers need every tool they can get their hands on - as a matter of public interest.

And inmoralphilosophy - well, Isupposeit could be the case that moral philosophers have discovered moral truths that are deductive consequences of most humans' moral frames, but which are so difficult and technical that they simply can't be explained to a popular audience within a one-hour lecture. But it would be a tad more suspicious than the corresponding case in, say, physics.

I realize that I speak as someone who does a lot of popularizing, but even so - fiction ought to be a respectable form of moral argument. And a respectable way of communicatingexperiences, in particular the experience of applying certain types of thinking skills.

I've always been of two minds about publishing longer fiction pieces about the future and its consequences. Not so much because of the potential for abuse, but because even whennotabused, fiction canstillbypass critical faculties and end up poured directly into the brains of at least some readers. Telling people aboutthe logical fallacy of generalization from fictional evidencedoesn't make it go away; people may just go on generalizing from the story as though they had actually seen it happen. And you simply can't have a story that's a rational projection; it's not just a matter of plot, it's a matter of the story needing to bespecific, rather than depicting a state of epistemic uncertainty.

But to make shorter philosophical points? Sure.

And... oh, what the hell. Just on the off-chance, are there any OB readers who could get agoodmovie made? Either outside Hollywood, or able to bypass the usual dumbing-down process that creates a money-losing flop? The probabilities are infinitesimal, I know, but I thought I'd check.

Referenced by: Teaching the Unteachable • Rationalist Fiction

Original with comments: (Moral) Truth in Fiction?

Informers and Persuaders

Eliezer Yudkowsky, 10 February 2009 08:22PM

Suppose we lived in this completely alternate universe where nothing in academia was about status, and no one had any concept of style. A universe where people wrote journal articles, and editors approved them, without the tiniest shred of concern for what "impression" it gave - without trying to look serious or solemn or sophisticated, and without being afraid of looking silly or even stupid. We shall even suppose that readers, correspondingly, have no such impressions.

In this simpler world, academics write papers from only two possible motives:

First, they may have some theory of which they desire to persuade others; this theory may or may not be true, and may or may not be believed for virtuous reasons or with very strong confidence, but the writer of the paper desires to gain adherents for it.

Second, there will be those who write with an utterly pure and virtuous love of the truthfinding process; they desire solely to give people more unfiltered evidence and to see evidence correctly added up, without a shred of attachment to their or anyone else's theory.

People in the first group may want to signal membership in the second group, but people in the second group only want their readers to be well-informed. In any case, to first order we must suppose that none of this is about signaling - that all such motives are just blanked out.

What do journal articles in this world look like, and how do the Persuaders' articles differ from the Informers'?

First, I would argue that both groups write much less formal journal articles than our own. I've read probably around a hundred books on writing (they're addictive); and they all treated formality as entropy to be fought - a state of disorder into which writing slides. It is easier to use big words than small words, easier to be abstract than concrete, easier to use passive -ation words than their active counterparts. Perhaps formality first became associated with Authority, back in the dawn of time, because Authorities put in less effort and forced their audience to read anyway. Formality became associated with Wisdom by being hard to understand. Why suppose that scientific formality was ever about preventing propaganda?

Both groups still use technical language, because they both care about being precise. They even use big words or phrases for their technical concepts: To carve out ever-finer slices through reality, you need new words, more words, hence bigger words (so you don't run out of namespace).

However, since neither group has a care for their image, they use the simplest words they can apart from that, and sentences as easy as possible apart from the big words. From our standpoint, their style would seem inconsistent, discongruous. A sentence might start with small words that just anyone could read, and then terminate in an exceptionally precise structure of technically sophisticated concepts accessible to only advanced audiences.

In this world it's not just eminent physicists who - secure in their reputation as Real Scientists - invent labels like "gluon", "quark", "black hole" and "Big Bang".

Other aspects of scientific taboo may still carry over. A Persuader might use vicious insults and character assassination. An Informer never would. But an Informer might point out - evenhandedly, wherever it happened to be true - that a supposedly relevant paper came from a small unheard-of organization and hadn't yet been replicated, or that the author of an exciting new paper had previously retracted other results...

If Persuaders want to look like Informers, they will, of course, restrain their ad-hominem attacks to sounding like the sort of things an Informer might point out; but this is a second-order phenomenon. First-order Persuaders would use all-out invective against their opponents.

What about emotions in general?

Suppose that there were only Informers and that they weren't concerned about preventing invasion by Persuaders. The Informers might well make a value-laden remark or two in the conclusions of their papers - after balancing the probability that the conclusion would later need to be retracted and that the emotion might interfere, versus the importance of the values in question. Even an Informer might say, in the conclusion of a paper on asteroid strikes, "We can probably breathe a sigh of relief about getting hit in the immediate future, but when you consider the sheer size of the catastrophe and the millions of dead and injured, we really ought to have a spacewatch program."

But Persuaders have an immensely stronger first-order drive to invoke powerful affective emotions and lade the reader's judgments with value. To second order, Persuaders will try to disguise this method as much as possible - let the reader draw conclusions, so long as they're the desired conclusions - try to pretend to abstract dispassionate language so that they can look like Informers, while still lingering on the emotion-arousing facts. Formality is a very easy disguise to wear, which is one reason I give it so little credit.

Informers, who have no desire to look like Informers, might go ahead and leave in a value judgment or two that seemed really unlikely to interfere with anyone's totting up the evidence. If Informers trusted their own judgment about that sort of thing, that is.

(Persuaders and Informers writing about policy arguments or moral arguments would be a whole 'nother class of issue. Then both types are offering value-laden arguments and dealing in facts that trigger emotions, and the question is who's collecting them evenhandedly versus lopsidedly.)

How about writing short stories?

Persuaders obviously have a motive to do it. Do Informers ever do it, if they're not worried about looking like Persuaders?

If you try to blank out the conventions of our own world, and imagine what would really be useful...

Then I can well imagine that it would be de rigueur to write small stories - story fragments, rather - describing the elements of your experimental procedure.

"The subjects were admininistered Progenitorivox" actually gives you very little information, just the dull sensation of having been told an authoritative fact.

Compare: "James is one of my typical subjects. Every Wednesday, he would visit me in my lab at 2pm, and, grimacing, swallow down two yellow pills from his bottle, while I watched. At the end of the study, I watched James and the other students file into the classroom, sit down, and fill out the surveys on each desk; as they left, I gave each of them a check for $50."

This, which conveys something of the experience of running the experiment and just begs you to go out and do your own... also gives you valuable information: that the Progenitorivox or placebo was taken at regular intervals with the investigator watching, and when and where and how the survey data was collected.

Maybe this is the most efficient way to communicate that information, and maybe not. To me it actually does seem efficient, and I would guess that the only reason people don't do this more often is that they would look insufficiently Distant and Authoritative. I have no trouble imagining an Informer writing a story fragment or two into their journal article.

Robin says: "I thus tend to avoid emotion, color, flash, stories, vagueness, repetition, rambling, and even eloquence."

I would guess that, to first order and before all signaling:

Persuaders actively seek out emotion, color, flash, and eloquence. They are vague when they have something to hide. They rehearse their favored arguments, but not to where it becomes annoying. They try to avoid rambling because no one wants to read that. They use stories where they expect stories to be persuasive - which, by default, they are - and avoid stories where they don't want their readers visualizing things in too much detail.

Informers avoid emotions that they fear may bias their readers. If they can't actually avoid the emotion - e.g., the paper is about slavery - they'll explicitly point it out, along with its potential biasing effect, and they'll go to whatever lengths they can to avoid provoking or inflaming the emotion further (short of actually obscuring the subject matter). Informers may use color to highlight the most important parts of their article, but won't usually give extra color to a single piece of specific evidence. Informers have no use for flash. They won't avoid being eloquent when their discussion happens to have an elegant logical structure. Informers use the appropriate level of abstraction or maybe a little lower; they are vague when details are unknowable or almost certainly irrelevant. Informers don't rehearse evidence, but they might find it useful to repeat some details of the experimental procedure. Informers use stories when they have an important experience to communicate, or when a set of details is most easily conveyed in story form. In papers that are about judgments of simple fact, Informers never use a story to arouse emotion.

I finally note, with regret, that in a world containing Persuaders, it may make sense for a second-order Informer to be deliberately eloquent if the issue has already been obscured by an eloquent Persuader - just exactly as elegant as the previous Persuader, no more, no less. It's a pity that this wonderful excuse exists, but in the real world, well...

Original with comments: Informers and Persuaders

Cynicism in Ev-Psych (and Econ?)

Eliezer Yudkowsky, 11 February 2009 03:06PM

Though I know more about the former than the latter, I begin to suspect that different styles of cynicism prevail in evolutionary psychology than in microeconomics.

Evolutionary psychologists are absolutely and uniformly cynical about the real reason why humans are universally wired with a chunk of complex purposeful functional circuitry X (e.g. an emotion) - we have X because it increased inclusive genetic fitness in the ancestral environment, full stop.

Evolutionary psychologists are mildly cynical about the environmental circumstances that activate and maintain an emotion. For example, if you fall in love with the body, mind, and soul of some beautiful mate, an evolutionary psychologist would like to check up on you in ten years to see whether the degree to which you think your mate's mind is still beautiful, correlates with independent judges' ratings of how physically attractive that mate still is.

But it wouldn't be conventionally ev-psych cynicism to suppose that you don't really love your mate, and that you were actually just attracted to their body all along, but that instead you told yourself a self-deceiving story about virtuously loving them for their mind, in order to falsely signal commitment.

Robin, on the other hand, often seems to think that this general type of cynicism is the default explanation and that anything else bears a burden of proof - why suppose an explanation that invokes a genuine virtue, when a selfish desire will do?

Of course my experience with having deep discussions with economists mostly consists of talking to Robin, but I suspect that this is at least partially reflective of a difference between the ev-psych and economic notions of parsimony.

Ev-psychers are trying to be parsimonious with how complex of an adaptation they postulate, and how cleverly complicated they are supposing natural selection to have been.

Economists... well, it's not my field, but maybe they're trying be parsimonious by having just a few simple motives that play out in complex ways via consequentialist calculations?

Quoth Leda Cosmides and John Tooby (famous EPers):

"The science of understanding living organization is very different from physics or chemistry, where parsimony makes sense as a theoretical criterion. The study of organisms is more like reverse engineering, where one may be dealing with a large array of very different components whose heterogeneous organization is explained by the way in which they interact to produce a functional outcome. Evolution, the constructor of living organisms, has no privileged tendency to build into designs principles of operation that are simple and general."

One consequence of this is that it's more parsimonious - under the evolutionary prior - to postulate many smaller simpler adaptations than one big clever complicated adaptation.

One simple way to signal quality X is by having quality X. But then other simple modifications might accrete around that.

So cynicism in the style of evolutionary psychology might be, "Why yes, so far as your explicit cognition is concerned, you love them for their beautiful mind. It's just that without the beautiful body, you probably wouldn't find yourself loving their mind so much. And once the beautiful body fades, you may find their ideas appearing less attractive too." Mind you, this is not an actual experimental result. It's just the sort of thing that a cynical evolutionary psychologist would look for - a cross-wiring between a couple of emotional circuits.

Even then, the cynic would bear a burden of proof, because a devil's-advocate parsimonious evolutionary psychologist would say, "How do you know the extra circuit is there? Maybe evolution wasn't that clever. Mates looked for signals of long-term commitment, so their partners evolved an actual long-term commitment mechanism when a mate was of high enough quality - can you show me an experiment that demonstrates it's any more complicated than that?"

By and large, evolutionary psychologists don't expect people to be clever, just evolution. It's a foundational assumption that there's no explicit cognitive desire to increase inclusive genetic fitness, and no reason to think that anyone (except a professional evolutionary psychologist) would explicitly know in advance which behaviors increased fitness in the ancestral environment. The organism, rather than being programmed with machiavellian subconscious long-term knowledge, is programmed with (genuine) emotions that activate under the right circumstances to steer them the right way (in the ancestral environment).

In economics, perhaps, it is more conventional and less alarming to suppose that people are doing explicitly clever and complicated things in the pursuit of explicit goals. But of this it is not really my place to speak; I'm just trying to describe my own side of the contrast I see.

Now it makes sense to suppose that we have certain general faculties - simple emotional circuits - that make us seek high status: that we are magnetically attracted to behaviors whenever we imagine that behavior will make others look on us fondly.

And it makes to suppose that we have a general faculty - a relatively simple emotional circuit - that makes us flinch away from explanations and views of our own behavior that put us in a negative light, and flinch toward explanations that put ourselves in a positive light.

So from an ev-psych standpoint, we can expect a lot of cynicism to be, in general, justified. It wouldn't even be surprising if people were relatively more attracted to bodies, and relatively less attracted to minds, on a purely psychological level, than they said/thought they were.

But to modify the emotional ontology by entirely deleting virtuous emotions... to say that, even on a psychological level, no human being was ever attracted to a mate's mind, nor ever wanted to be honest in a business transaction and not just signal honesty... is not quite what evolutionary psychologists do, most of the time. They are out to come up with an evolutionary explanation of why humans have the standard emotions, rather than telling us that we have nonstandard emotions instead. Maybe in economics this sounds less alarming because people routinely come up with simplified models? But in ev-psych it seems to hearken back to the bad old Freudian days of counterintuitiveness - we're excited that the intuitive view of human emotion turns out to be evolutionarily explainable. Including a lot of things that people would rather not talk about but which they do recognize as realistic. And including a lot of phenomena that go on behind the scenes but which don't much change our view of which emotions we have, just our view of when emotions activate, in what real context. On that score we are happy to be cynical and challenge intuition.

Referenced by: An Especially Elegant Evpsych Experiment

Original with comments: Cynicism in Ev-Psych (and Econ?)

The Evolutionary-Cognitive Boundary

Eliezer Yudkowsky, 12 February 2009 04:44PM

I tend to draw a very sharp line between anything that happens inside a brain and anything that happened in evolutionary history. There are good reasons for this! Anything originally computed in a brain can be expected to be recomputed, on the fly, in response to changing circumstances.

Consider, for example, the hypothesis that managers behave rudely toward subordinates "to signal their higher status". This hypothesis then has two natural subdivisions:

If rudeness is an executing adaptation as such - something historically linked to the fact it signaled high status, but not psychologically linked to status drives - then we might experiment and find that, say, the rudeness of high-status men to lower-status men depended on the number of desirable women watching, but that they weren't aware of this fact. Or maybe that people are just as rude when posting completely anonymously on the Internet (or more rude; they can now indulge their adapted penchant to be rude without worrying about the now-nonexistent reputational consequences).

If rudeness is a conscious or subconscious strategy to signal high status (which is itself a universal adapted desire), then we're more likely to expect the style of rudeness to be culturally variable, like clothes or jewelry; different kinds of rudeness will send different signals in different places. People will be most likely to be rude (in the culturally indicated fashion) in front of those whom they have the greatest psychological desire to impress with their own high status.

When someone says, "People do X to signal Y", I tend to hear, "People do X when they consciously or subconsciously expect it to signal Y", not, "Evolution built people to do X as an adaptation that executes given such-and-such circumstances, because in the ancestral environment, X signaled Y."

I apologize, Robin, if this means I misunderstood you. But I think it really is important to use different words that draw a hard boundary between the evolutionary computation and the cognitive computation - "People are adapted to do X because it signaled Y", versus "People do X because they expect it to signal Y".

"Distal cause" and "proximate cause" doesn't seem good enough, when there's such a sharp boundary within the causal network about what gets computed, how it got computed, and when it will be recomputed. Yes, we have epistemic leakage across this boundary - we can try to fill in our leftover uncertainty about psychology using evolutionary predictions - but it's epistemic leakage between two very different subjects.

I've noticed that I am, in general, less cynical than Robin, and I would offer up the guess for refutation (it is dangerous to reason about other people's psychologies) that Robin doesn't draw a sharp boundary across his cynicism at the evolutionary-cognitive boundary. When Robin asks "Are people doing X mostly for the sake of Y?" he seems to answer the same "Yes", and feel more or less the same way about that answer, whether or not the reasoning goes through an evolutionary step along the way.

I would be very disturbed to learn that parents, in general, showed no grief for the loss of a child who they consciously believed to be sterile. The actual experiment which shows that parental grief correlates strongly to the expected reproductive potential of a child of that age in a hunter-gatherer society - not the different reproductive curve in a modern society - does not disturb me.

There was a point more than a decade ago when I would have seen that as a puppeteering of human emotions by evolutionary selection pressures, and hence something to be cynical about. Yet how could parental grief come into existence at all, without a strong enough selection pressure to carve it into the genome from scratch? All that should matter for saying "The parent truly cares about the child" is that the grief in the parent's mind is cognitively real and unconditional and not even subconsciously for the sake of any ulterior motive; and so it does not update for modern reproductive curves.

Of course the emotional circuitry is ultimately there for evolutionary-historical reasons. But only conscious or subconscious computations can gloom up my day; natural selection is an alien thing whose 'decisions' can't be the target of my cynicism or admiration.

I suppose that is a merely moral consequence - albeit it's one that I care about quite a lot. Cynicism does have hedonic effects. Part of my grand agenda that I have to put forward about rationality, has to do with arguing against many various propositions "Rationality should make us cynical about X" (e.g. "physical lawfulness -> choice is a meaningless illusion") that I happen to disagree with. So you can see why I'm concerned about drawing the proper boundary of cynicism around evolutionary psychology (especially since I think the proper boundary is a sharp full stop).

But the same boundary also has major consequences for what we can expect people to recompute or not recompute - for the way that future behaviors will change as the environment changes. So once again, I advocate for language that separates out evolutionary causes and clearly labels them, especially in discussions of signaling. It has major effects, not just on how cynical I end up about human nature, but on what 'signaling' behaviors to expect, when.

Referenced by: An Especially Elegant Evpsych Experiment

Original with comments: The Evolutionary-Cognitive Boundary

An Especially Elegant Evpsych Experiment

Eliezer Yudkowsky, 13 February 2009 02:58PM

Followup to: Adaptation-Executers not Fitness-Maximizers, The Evolutionary-Cognitive Boundary

"In a 1989 Canadian study, adults were asked to imagine the death of children of various ages and estimate which deaths would create the greatest sense of loss in a parent. The results, plotted on a graph, show grief growing until just before adolescence and then beginning to drop. When this curve was compared with a curve showing changes in reproductive potential over the life cycle (a pattern calculated from Canadian demographic data), the correlation was fairly strong. But much stronger - nearly perfect, in fact - was the correlation between the grief curves of these modern Canadians and the reproductive-potential curve of a hunter-gatherer people, the !Kung of Africa. In other words, the pattern of changing grief was almost exactly what a Darwinian would predict, given demographic realities in the ancestral environment... The first correlation was .64, the second an extremely high .92."

(Robert Wright, summarizing: "Human Grief: Is Its Intensity Related to the Reproductive Value of the Deceased?" Crawford, C. B., Salter, B. E., and Lang, K.L. Ethology and Sociobiology 10:297-307.)

Disclaimer: I haven't read this paper because it (a) isn't online and (b) is not specifically relevant to my actual real job. But going on the given description, it seems like a reasonably awesome experiment. [Gated version here, thanks Benja Fallenstein. Odd, I thought I searched for that. Reading now... seems to check out on the basics. Correlations are as described, N=221.]

The most obvious inelegance of this study, as described, is that it was conducted by asking human adults to imagine parental grief, rather than asking real parents with children of particular ages. (Presumably that would have cost more / allowed fewer subjects.) However, my understanding is that the results here squared well with the data from closer studies of parental grief that were looking for other correlations (i.e., a raw correlation between parental grief and child age).

That said, consider some of this experiment's elegant aspects:

- A correlation of .92(!) This may sound suspiciously high - could evolution really do such exact fine-tuning? - until you realize that this selection pressure was not only great enough to fine-tune parental grief, but, in fact, carve it out of existence from scratch in the first place.

- People who say that evolutionary psychology hasn't made any advance predictions are (ironically) mere victims of "no one knows what science doesn't know" syndrome. You wouldn't even think of this as an experiment to be performed if not for evolutionary psychology.

- The experiment illustrates as beautifully and as cleanly as any I have ever seen, the distinction between a conscious or subconscious ulterior motive and an executing adaptation with no realtime sensitivity to the original selection pressure that created it.

The parental grief is not even subconsciously about reproductive value - otherwise it would update for Canadian reproductive value instead of !Kung reproductive value. Grief is an adaptation that now simply exists, real in the mind and continuing under its own inertia.

Parents do not care about children for the sake of their reproductive contribution. Parents care about children for their own sake; and the non-cognitive, evolutionary-historical reason why such minds exist in the universe in the first place, is that children carry their parents' genes.

Indeed, evolution is the reason why there are any minds in the universe at all. So you can see why I'd want to draw a sharp line through my cynicism about ulterior motives at the evolutionary-cognitive boundary; otherwise, I might as well stand up in a supermarket checkout line and say, "Hey! You're only correctly processing visual information while bagging my groceries in order to maximize your inclusive genetic fitness!"

• 0.92 is, I think, the highest correlation I've ever seen in any ev-psych experiment, and indeed, one of the highest correlations I've seen in any psychology experiment. (Albeit I've seen e.g. a correlation of 0.98 reported for asking one group of subjects "How similar is A to B?" and another group "What is the probability of A given B?" on questions like "How likely are you to draw 60 red balls and 40 white balls from this barrel of 800 red balls and 200 white balls?" - in other words, these are simply processed as the same question.)

Since we are all Bayesians here, we may take our priors into account and ask if at least some of this unexpectedly high correlation is due to luck. The evolutionary fine-tuning we can probably take for granted; this is a huge selection pressure we're talking about. The remaining sources of suspiciously low variance are, (a) whether a large group of adults could correctly envision, on average, relative degrees of parental grief (apparently they can). And, (b), whether the surviving !Kung are typical ancestral hunter-gatherers in this dimension, or whether variance between hunter-gatherer tribal types should have been too high to allow a correlation of .92.

But even after taking into account any skeptical priors, correlation .92 and N=221 is pretty strong evidence, and our posteriors should be less skeptical on all these counts.

• You might think it an inelegance of the experiment that it was performed prospectively on imagined grief, rather than retrospectively on real grief. But it is prospectively imagined grief that will actually operate to steer parental behavior away from losing the child! From an evolutionary standpoint, an actual dead child is a sunk cost; evolution "wants" the parent to learn from the pain, not do it again, adjust back to their hedonic set point, and go on raising other children.

• Similarly, the graph that correlates to parental grief is for the future reproductive potential of a child that has survived to a given age, and not the sunk cost of raising the child which has survived to that age. (Might we get an even higher correlation if we tried to take into account the reproductive opportunity cost of raising a child of age X to independent maturity, while discarding all sunk costs to raise a child to age X?)

Humans usually do notice sunk costs - this is presumably either an adaptation to prevent us from switching strategies too often (compensating for an overeager opportunity-noticer?) or an unfortunate spandrel of pain felt on wasting resources.

Evolution, on the other hand - it's not that evolution "doesn't care about sunk costs", but that evolution doesn't even remotely "think" that way; "evolution" is just a macrofact about the real historical reproductive consequences.

So - of course - the parental grief adaptation is fine-tuned in a way that has nothing to do with past investment in a child, and everything to do with the future reproductive consequences of losing that child. Natural selection isn't crazy about sunk costs the way we are.

But - of course - the parental grief adaptation goes on functioning as if the parent were living in a !Kung tribe rather than Canada. Most humans would notice the difference.

Humans and natural selection are insane in different stable complicated ways.

Original with comments: An Especially Elegant Evpsych Experiment

Rationality Quotes 26

Eliezer Yudkowsky, 14 February 2009 04:14PM

"Poets, philosophers, acidheads, salesmen: everybody wants to know, 'What is Reality?' Some say it's a vast Unknowable so astounding and raw and naked that it grips the human mind and shakes it like a puppy shakes a rag doll. A lot of good that does us."
-- The Book of the SubGenius

"When they discovered that reality was more complicated than they thought, they just swept the complexity under a carpet of epicycles. That is, they created unnecessary complexity. This is an important point. The universe is complex, but it's usefully complex."
-- Larry Wall

"I can't imagine a more complete and precise answer to the question 'for what reason...?' than 'none'. The fact that you don't like the answer is your problem, not the universe's."
-- Lee Daniel Crocker

"In the end they all moved in fantasies and not in the daily tide of their seemingly useless lives. Souls forever lost in the terrifying freedom of their existence."
-- Shinji and Warhammer40k

"Thus the freer the judgement of a man is in regard to a definite issue, with so much greater necessity will the substance of this judgement be determined."
-- Friedrich Engels, Anti-Dühring

"There will always be some that cannot be saved.
It is impossible to save everyone.
If I have to lose five hundred to earn one thousand,
I will abandon one hundred and save the lives of nine hundred.
That is the most efficient method.
That is the ideal----
Kiritsugu once said that.
Of course I got mad.
I really got mad.
Because I knew that without being told.
Because I myself was saved like that.
I don't even need to be told something as obvious as that.
But still----I believed that someone would be a superhero if they saved everyone even though they think like that.
It may be an idealistic thought or an impossible pipe dream, but a superhero is someone who tries to save everyone in spite of that."
-- Emiya Shirou, in Fate/stay night
(visual novel by Kinoko Nasu; Unlimited Blade Works path, Mirror Moon translation)

Original with comments: Rationality Quotes 26

An African Folktale

Eliezer Yudkowsky, 16 February 2009 01:00AM

This is a folktale of the Hausa, a farming culture of around 30 million people, located primarily in Nigeria and Niger but with other communities scattered around Africa. I find the different cultural assumptions revealed to be... attention-catching; you wouldn't find a tale like this in Aesop. From Hausa Tales and Traditions by Frank Edgar and Neil Skinner; HT Robert Greene.

The Farmer, the Snake and the Heron

There was once a man hoeing away on his farm, when along came some people chasing a snake, meaning to kill it. And the snake came up to the farmer.
Says the snake "Farmer, please hide me." "Where shall I hide you?" said the farmer, and the snake said "All I ask is that you save my life." The farmer couldn't think where to put the snake, and at last bent down and opened his anus, and the snake entered.
Presently the snake's pursuers arrived and said to the farmer "Hey, there! Where's the snake we were chasing and intend to kill? As we followed him, he came in your direction." Says the farmer "I haven't seen him." And the people went back again.
Then the farmer said to the snake "Righto - come out now. They've gone." "Oh no" said the snake, "I've got me a home." And there was the farmer, with his stomach all swollen, for all the world like a pregnant woman!

And the farmer set off, and presently, as he passed, he saw a heron. They put their heads together and whispered, and the heron said to the farmer "Go and excrete. Then, when you have finished, move forward a little, but don't get up. Stay squatting, with your head down and your buttocks up, until I come."
So the man went off and did just exactly as the heron had told him, everything. And the snake put out his head and began to catch flies. Then the heron struck and seized the snake's head. Then he pulled and he pulled until he had got him out, and the man tumbled over. And the heron finished off the snake with his beak.
The man rose and went over to the heron. "Heron" says he, "You've got the snake out for me, now please give me some medicine to drink, for the poison where he was lying."
Says the heron "Go and find yourself some white fowls, six of them. Cook them and eat them - they are the medicine." "Oho" said the man, "White fowl? But that's you" and he grabbed the heron and tied it up and went off home. There he took him into a hut and hung him up, the heron lamenting the while.
Then the man's wife said "Oh, husband! The bird did you a kindness. He saved your life, by getting the trouble out of your stomach for you. And now you seize him and say that you are going to slaughter him!"
So the man's wife loosed the heron, but as he was going out, he pecked out one of her eyes. And so passed and went his way. That's all. For so it always has been - if you see the dust of a fight rising, you will know that a kindness is being repaid! That's all. The story's finished.

I wonder if this has something to do with why Africa stays poor.

I was slightly shocked, reading this story. It seems to reveal a cultural gloominess deeper than its Western analogue of fashionable cynicism. The cynical Western tale would at least have an innocent, virtuous, idealistic fool to be exploited, since our cynicism is mostly about feeling superior to those less sophisticated. Thistale has so much defection that you can't even call the characters hypocrites. This isn't a story told to make the listener feel superior to the fools who still believe in the essential goodness of human nature. This is a tribe whose children are being warned to expect cooperation to be met with defection the same way our own kids are told "slow but steady wins the race".

It's occasionally debated on this blog whetherthe psychological unity of humankindis real and how much room it leaves for humans to be deeply different. Someone might look at this tale and say, "Gratitude in Africa isn't like gratitude in the West." But to me it looks like the people who pass on this tale must have pretty much the same chunk of brain circuitry somewhere that implements the emotion ofgratitude- that's what creates the background against which this story is told. It wouldn't be viewed as important wisdom, otherwise.

But cultural gloominess this deep may be a self-fulfilling prophecy, as powerful as if the emotion of gratitude itself had diminished, or failed.

Original with comments: An African Folktale

Cynical About Cynicism

Eliezer Yudkowsky, 17 February 2009 12:49AM

I'm cynical about cynicism. I don't believe that most cynicism is really about knowing better. When I see someone being cynical, my first thought is that they're trying to show off their sophistication and assert superiority over the naive. As opposed to, say, sharing their uncommon insight about not-widely-understood flaws in human nature.

There are two obvious exceptions to this rule. One is if the speaker has something serious and realistic to say about how to improve matters. Claiming that problems can be fixed will instantly lose you all your world-weary street cred and mark you as another starry-eyed idealistic fool. (Conversely, any "solution" that manages not to disrupt the general atmosphere of doom, does not make me less skeptical: "Humans are evil creatures who slaughter and destroy, but eventually we'll die out from poisoning the environment, so it's all to the good, really.")

No, not every problem is solvable. But by and large, if someone achieves uncommon insight into darkness - if they know more than I do about human nature and its flaws - then it's not unreasonable to expect that they might have a suggestion or two to make about remedy, patching, or minor deflection. If, you know, the problem is one that they really would prefer solved, rather than gloom being milked for a feeling of superiority to the naive herd.

The other obvious exception is for science that has something to say about human nature. A testable hypothesis is a testable hypothesis and the thing to do with it is test it. Though here one must be very careful not to go beyond the letter of the experiment for the sake of signaling hard-headed realism:

Consider the hash that some people make of evolutionary psychology in trying to be cynical - assuming that humans have a subconscious motive to promote their inclusive genetic fitness. Consider the hash that some neuroscientists make of the results of their brain scans, supposing that if a brain potential is visible before the moment of reported decision, this proves the nonexistence of free will. It's not you who chooses, it's your brain!

The facts are one thing, but feeling cynical about those facts is another matter entirely. In some cases it can lead people to overrun the facts - to construct new, unproven, or even outright disproven glooms in the name of signaling realism. Behaviorism probably had this problem - signaling hardheaded realism about human nature was probably one of the reasons they asserted we don't have minds.

I'm especially on guard against cynicism because it seems to be a standard corruption of rationality in particular. If many people are optimists, then true rationalists will occasionally have to say things that sound pessimistic by contrast. If people are trying to signal virtue through their beliefs, then a rationalist may have to advocate contrasting beliefs that don't signal virtue.

Which in turn means that rationalists, and especially apprentice rationalists watching other rationalists at work, are especially at-risk for absorbing cynicism as though it were a virtue in its own right - assuming that whosoever speaks of ulterior motives is probably a wise rationalist with uncommon insight; or believing that it is an entitled benefit of realism to feel superior to the naive herd that still has a shred of hope.

And this is a fearsome mistake indeed, because you can't propose ways to meliorate problems and still come off as world-weary.

TV Tropes proposes a Sliding Scale of Idealism Versus Cynicism. It looks to me like Robin tends to focus his suspicions on that which might be signaling idealism, virtue, or righteousness; while I tend to focus my own skepticism on that which might signal cynicism, world-weary sophistication, or sage maturity.

Referenced by: Why Our Kind Can't Cooperate • Incremental Progress and the Valley • The Sin of Underconfidence

Original with comments: Cynical About Cynicism

Good Idealistic Books are Rare

Eliezer Yudkowsky, 17 February 2009 06:41PM

Saith Robin in "Seeking a Cynic's Library":

Cynicism and Idealism are a classic yin and yang, a contradictory pair where we all seem to need both sides...

Books on education, medicine, government, charity, religion, technology, travel, relationships, etc. mostly present relatively idealistic views, though of course no view is entirely one way or the other. So one reason the young tend to be idealistic is that most reading material they can easily find and understand is idealistic.

My impression of this differs somewhat from Robin's (what a surprise).

I think that what we see in most books of the class Robin describes, areofficialviews. These official views may leave out many unpleasant elements of the story. But because officialism also tries to signal authority and maturity, it's hardly likely to permit itself any real hope or enthusiasm. Perhaps an obligatory if formal nod in the direction of some popular good cause, because this is expected of officialdom. But this is hardly an idealistic voice.

What does a full-blown nonfictional idealism look like? Some examples that I remember from my own youth:

- Jerry Pournelle'sA Step Farther Out,an idealistic view of space travel and more general technological advancement, and the possibility of rising standards of living as opposed to Ehrlichian gloomsaying.

- Brown, Keating, Mellinger, Post, Smith, and Tudor'sThe Incredible Bread Machine,my childhood introduction totraditional capitalist values.

- Eric Drexler'sEngines of Creation(and to a lesser extent Ed Regis'sGreat Mambo Chicken), my introduction totranshumanism.

- Richard Feynman'sSurely You're Joking, Mr. Feynman(for traditional rationalist values).

Supposing youwantedyour child to grow up an idealist - what nonfiction books like these could you find to give them? I don't find it easy to think of many - most nonfiction books arenotlike this.

On the other hand, I suspect that idealisticfictionaimedspecificallyat children is a far greater cultural force than anything they pick up from their school textbooks. Textbooks are marketed to adult textbook-selectors; juvenile fiction and children's television are actually aimed at children.

Of course this just implies a chicken-and-egg problem; why do children enjoy idealism more than cynicism? On this score I would suggest that children in the hunter-gatherer EEA have no chance of successful rebellion, and so haven't yet developed certain emotions that will come into play after puberty. But children will be actively engaged in absorbing tribal mores during their maturation, and may benefit from signaling such absorption.

Teenagers who act as if they could still get together with their friends and split off to form their own tribe, enjoy cynicism aimed at current authority figures and idealism aimed at their new tribe.

When such forces have petered out, I suggest we are left with a mostly socially-determined adult equilibrium: idealism about some distant subjects is used to signal virtue, cynicism about other distant subjects is used to signal sophistication.

Original with comments: Good Idealistic Books are Rare

Against Maturity

Eliezer Yudkowsky, 18 February 2009 11:34PM

I remember the moment of my first break with Judaism. It was in kindergarten, when I was being forced to memorize and recite my first prayer. It was in Hebrew. We were given a transliteration, but not a translation. I asked what the prayer meant. I was told that I didn't need to know - so long as I prayed in Hebrew, it would work even if I didn't understand the words. (Any resemblance to follies inveighed against in my writings is not coincidental.)

Of course I didn't accept this, since it was blatantly stupid, and I figured that God had to be at least as smart as I was. So when I got home, I asked my parents, and they didn't bother arguing with me. They just said, "You're too young to argue with; we're older and wiser; adults know best; you'll understand when you're older."

They were right about that last part, anyway.

Of course there were plenty of places my parents really did know better, even in the realms of abstract reasoning. They were doctorate-bearing folks and not stupid. I remember, at age nine or something silly like that, showing my father a diagram full of filled circles and trying to convince him that the indeterminacy of particle collisions was because they had a fourth-dimensional cross-section and they were bumping or failing to bump in the fourth dimension.

My father shot me down flat. (Without making the slightest effort to humor me or encourage me. This seems to have worked out just fine. He did buy me books, though.)

But he didn't just say, "You'll understand when you're older." He said that physics was math and couldn't even be talked about without math. He talked about how everyone he met tried to invent their own theory of physics and how annoying this was. He may even have talked about the futility of "providing a mechanism", though I'm not actually sure if I originally got that off him or Baez.

You see the pattern developing here. "Adulthood" was what my parents appealed to when they couldn't verbalize any object-level justification. They had doctorates and were smart; if there was a good reason, they usually would at least try to explain it to me. And it gets worse...

The most fearsome damage wreaked upon my parents by their concept of "adulthood", was the idea that being "adult" meant that you were finished - that "maturity" marked the place where you declared yourself done, needing to go no further.

This was displayed most clearly in the matter of religion, where I would try to talk about a question I had, and my parents would smile and say: "Only children ask questions like that; when you're adult, you know that it's pointless to argue about it." They actually said that outright! To ask questions was a manifestation of earnest, childish enthusiasm, earning a smile and a pat on the head. An adult knew better than to waste effort on pointless things.

We never really know our parents; we only know the face of our parents that they turn to us, their children. I don't know if my parents ever thought about the child-adult dichotomy when they weren't talking to me.

But this is what I think my parents were thinking: If they had tried to answer a question as children, and then given up as adults - a quite common pattern in their religious decay - they labeled "mature" the place and act of giving up, by way of consolation. They'd asked the question as children and stopped asking as adults - and the story they told themselves about that was that only children asked that question, and now they had succeeded into the sage maturity of knowing not to argue.

To this very day, I constantly remind myself that, no matter what I do in this world, I will doubtlessly be considered an infant by the standards of future intergalactic civilization, and so there is no point in pretending to be a grown-up. I try to maintain a mental picture of myself as someone who is not mature, so that I can go on maturing.

And more...

From my parents I learned the observational lesson that "adulthood" was something sort of like "peer acceptance", that is, its pursuit made you do stupid things that you wouldn't have done if you were just trying to get it right.

At that age I couldn't have given you a very good definition of "right" outside the realm of pure epistemic accuracy -

- but I understood the concept of asking the wrong question. "Does this help people?" "Will this make anyone happy?" "Is this belief true?"  Those were the sorts of questions to ask, not, "Is this the adult thing to do?"

So I did not divide up the universe into the childish way versus the adult way, nor ever tell myself that I had completed anything by getting older, nor congratulate myself on having stopped being a child. Instead I learned that there were various stereotypes and traps that could take people's attention off the important questions, and instead make them try to match certain unimportant concepts that existed in their minds.  One of these attractor-traps was called "teenager", and one of these attractor-traps was called "adult", and both were to be avoided.

I've previously touched on the massive effect on my youthful psychology of reading a book of advice to parents with teenagers, years before I was actually a teenager; I took one look at the description of the stupid things teenagers did, and said to myself with quiet revulsion, "I'm not going to do that"; and then I actually didn't. I never drank and drove, never drank, never tried a single drug, never lost control to hormones, never paid any attention to peer pressure, and never once thought my parents didn't love me. In a safer world, I would have wished for my parents to have hidden that book better...

...but I had a picture in my mind of what it meant to be a "teenager"; and I determined to avoid it; and I did.

Of course there are a lot of children in this world who don't like being "children" and who try to appear as "adult" or as "mature" as possible. That's why they start smoking, right? So that was also part of the picture that I had in my mind of a "stupid teenager": stupid teenagers deliberately try to be mature.

My parents had a picture in their mind of what it meant to be a "kid", which included "kids desperately want to be adult". I presume, though I don't exactly know, that my parents had a picture of "childishness" which was formed by their own childhood and not updated.

In any case my parents were constantly trying to get me to do things by telling me about how it would make me look adult.

That was their appeal - not, "Do this because it is older and wiser," but, "Do this, because it will make you look adult." To this day I wonder what they could have possibly been thinking. Would a stereotypical teenager listen to their parents' advice about that sort of thing?

Not surprisingly, being constantly urged to do things because they would signal adulthood, which I had no particular desire to do, had the effect of making me strongly notice things that signaled adulthood as mere signals.

I think that a lot of difference in the individual style of rationalists comes down to which signaling behaviors strike us as dangerous, harmful, or, perhaps, personally annoying; and which signaling behaviors seem relatively harmless, or possibly even useful paths to accomplishment (perhaps because they don't annoy us quite so much). Robin is willing to tolerate formality in journals, viewing it as possibly even a useful path to filtering out certain kinds of noise; to me formality seems like a strong net danger to rationality that filters nothing, just making it more difficult to read. I'm willing to tolerate behaviors that signal idealism or caring for others, viewing it as an important pathway to real altruism later on; Robin seems to think such behaviors are relatively more harmful and that they ought to be stopped outright.

It's not that Robin is a cynic or I'm an idealist, but that I'm relatively more annoyed by cynical errors and Robin is relatively more annoyed by idealistic errors. And fitting the same dimension, Robin seems relatively more annoyed by the errors in the style of youth, where I seem relatively more annoyed by errors in the style of maturity.

And so I take a certain dark delight in quoting anime fanfiction at people who expect me to behave like a wise sage of rationality. Why should I pretend to be mature when practically every star in the night sky is older than I am?

Referenced by: Pretending to be Wise • Formative Youth • That Magical Click

Original with comments: Against Maturity

Pretending to be Wise

Eliezer Yudkowsky, 19 February 2009 10:30PM

Followup to: Against Maturity

"The hottest place in Hell is reserved for those who in time of crisis remain neutral."
-- Dante Alighieri, famous hell expert John F. Kennedy, misquoter

A special case of adulthood-signaling worth singling out, is the display of neutrality or suspended judgment, in order to signal maturity, wisdom, impartiality, or just a superior vantage point.

An example would be the case discussed yesterday of my parents, who respond to theological questions like "Why does ancient Egypt, which had good records on many other matters, lack any records of Jews having ever been there?" with "Oh, when I was your age, I also used to ask that sort of question, but now I've grown out of it."

Another example would be the principal who, faced with two children who were caught fighting on the playground, sternly says: "It doesn't matter who started the fight, it only matters who ends it." Of course it matters who started the fight. The principal may not have access to good information about this critical fact, but if so, he should say so, not dismiss the importance of who threw the first punch. Let a parent try punching the principal, and we'll see how far "It doesn't matter who started it" gets in front of a judge. But to adults it is just inconvenient that children fight, and it matters not at all to their convenience which child started it, it is only convenient that the fight end as rapidly as possible.

A similar dynamic, I believe, governs the occasions in international diplomacy where Great Powers sternly tell smaller groups to stop that fighting right now. It doesn't matter to the Great Power who started it - who provoked, or who responded disproportionately to provocation - because the Great Power's ongoing inconvenience is only a function of the ongoing conflict. Oh, can't Israel and Hamas just get along?

This I call "pretending to be Wise". Of course there are many ways to try and signal wisdom. But trying to signal wisdom by refusing to make guesses - refusing to sum up evidence - refusing to pass judgment - refusing to take sides - staying above the fray and looking down with a lofty and condescending gaze - which is to say, signaling wisdom by saying and doing nothing - well, that I find particularly pretentious.

Paolo Freire said, "Washing one's hands of the conflict between the powerful and the powerless means to side with the powerful, not to be neutral." A playground is a great place to be a bully and a terrible place to be a victim, if the teachers don't care who started it. And likewise in international politics: A world where the Great Powers refuse to take sides and only demand immediate truces, is a great world for aggressors and a terrible place for the aggressed. But, of course, it is a very convenient world in which to be a Great Power or a school principal.

So part of this behavior can be chalked up to sheer selfishness on the part of the Wise.

But part of it also has to do with signaling a superior vantage point. After all - what would the other adults think of a principal who actually seemed to be taking sides in a fight between mere children? Why, it would lower his status to a mere participant in the fray!

Similarly with the revered elder - who might be a CEO, a prestigious academic, or a founder of a mailing list - whose reputation for fairness depends on their refusal to pass judgment themselves, when others are choosing sides. Sides appeal to them for support, but almost always in vain; for the Wise are revered judges on the condition that they almost never actually judge - then they would just be another disputant in the fray, no better than any other mere arguer.

(Oddly, judges in the actual legal system can repeatedly hand down real verdicts without automatically losing their reputation for impartiality. Maybe because of the understood norm that they have to judge, that it's their job. Or maybe because judges don't have to repeatedly rule on issues that have split a tribe on which they depend for their reverence.)

There are cases where it is rational to suspend judgment, where people leap to judgment only because of their biases. As Michael Rooney said, "The error here is similar to one I see all the time in beginning philosophy students: when confronted with reasons to be skeptics, they instead become relativists. That is, when the rational conclusion is to suspend judgment about an issue, all too many people instead conclude that any judgment is as plausible as any other."

But then how can we avoid the (related but distinct) pseudo-rationalist behavior of signaling your unbiased impartiality by falsely claiming that the current balance of evidence is neutral? "Oh, well, of course you have a lot of passionate Darwinists out there, but I think the evidence we have doesn't really enable us to make a definite endorsement of natural selection over intelligent design."

On this point I'd advise remembering that neutrality is a definite judgment. It is not staying above anything. It is putting forth the definite and particular position that the balance of evidence in a particular case licenses only one summation, which happens to be neutral. This, too, can be wrong; propounding neutrality is just as attackable as propounding any particular side.

Likewise with policy questions. If someone says that both pro-life and pro-choice sides have good points and that they really should try to compromise and respect each other more, they are not taking a position above the two standard sides in the abortion debate. They are putting forth a definite judgment, every bit as particular as saying "pro-life!" or "pro-choice!"

Incidentally... this is not an invitation to start talking about abortion in the comments, or Israel and Hamas either. Overcoming Bias isn't for having that particular discussion; there are other forums which deal with those perfectly legitimate topics. Maybe if Less Wrong got large enough... but not now.

But it's not that a rationalist is too mature to talk about politics. It's not that a rationalist is above this foolish fray in which only mere political partisans and youthful enthusiasts would stoop to participate.

As Robin describes it, the ability to have potentially divisive conversations is a limited resource. If you can think of ways to pull the rope sideways, you are justified in expending your limited resources on relatively less common issues where marginal discussion offers relatively higher marginal payoffs.

But then the responsibilities that you deprioritize are a matter of your limited resources. Not a matter of floating high above, serene and Wise.

Added: My reply to PG's comment on HN seems like a summary worth reposting: There's a difference between:

- Passing neutral judgment;

- Declining to invest marginal resources;

- Pretending that either of the above is a mark of deep wisdom, maturity, and a superior vantage point; with the corresponding implication that the original sides occupy lower vantage points that are not importantly different from up there.

Referenced by: Wise Pretensions v.0 • No, Really, I've Deceived Myself

Original with comments: Pretending to be Wise

Wise Pretensions v.0

Eliezer Yudkowsky, 20 February 2009 05:02PM

Followup to: Pretending to be Wise

For comparison purposes, here's an essay with similar content to yesterday's "Pretending to be Wise", which I wrote in 2006 in a completely different style, edited down slightly (content has been deleted but not added). Note that the 2006 concept of "pretending to be Wise" hasn't been narrowed down as much compared to the 2009 version; also when I wrote it, I was in more urgent need of persuasive force.

I thought it would be an interesting data point to check whether this essay seems more convincing than yesterday's, following Robin's injuction "to avoid emotion, color, flash, stories, vagueness, repetition, rambling, and even eloquence" - this seems like rather the sort of thing he might have had in mind.

And conversely the stylistic change also seems like the sort of thing Orwell might have had in mind, when Politics and the English Language compared: "I returned and saw under the sun, that the race is not to the swift, nor the battle to the strong, neither yet bread to the wise, nor yet riches to men of understanding, nor yet favour to men of skill; but time and chance happeneth to them all." Versus: "Objective considerations of contemporary phenomena compel the conclusion that success or failure in competitive activities exhibits no tendency to be commensurate with innate capacity, but that a considerable element of the unpredictable must invariably be taken into account." That would be the other side of it.

At any rate, here goes Eliezer_2006...

I do not fit the stereotype of the Wise. I am not Gandalf, Ged, or Gandhi. I do not sit amidst my quiet garden, staring deeply into the truths engraved in a flower or a drop of dew; speaking courteously to all who come before me, and answering them gently regardless of how they speak to me.

If I tried to look Wise, and succeeded, I would receive more respect from my fellows. But there would be a price.

To pretend to be Wise means that you must always appear to give people the benefit of the doubt. Thus people will admire you for your courtesy. But this is not always true.

To pretend to be Wise, you must always pretend that both sides have merit, and solemnly refuse to judge between them. For if you took one side or another, why then, you would no longer be one of the aloof Wise, but merely another partisan, on a level with all the other mere bickerers.

As one of the Wise, you are omnipotent on the condition that you never exercise your power. Otherwise people would start thinking that you were no better than they; and they would no longer hold you in awe.

Ofttimes it is greatly convenient, to pretend to be Wise. When any conflict breaks out, you can sternly chide both sides, saying: "You are equally at fault; you must learn to see each other's viewpoints. I am older and more mature, and I say to you: stop this pointless bickering, children, for you begin to annoy me. Ponder well the wisdom of having everyone get along!" You do not need to examine the dispute, nor wonder if perhaps one side does have more merit than the other. You need not judge between two sides, and risk having your judgment turn out to be embarrassingly wrong, or risk having your judgment questioned as though you were only another ordinary mortal. Indeed you must not ask questions, you must not judge; for if you take sides, you will at once lose your reputation for being Wise, which requires that you stand forever above the fray.

But truth is not handed out in equal parts before the start of a dispute.

And I am not one of the Wise. Even if I wished to be, it is not within my nature. I do not hesitate to place my reputation in jeopardy to aid the side I believe is right. Even if it makes me seem but an ordinary mortal, no better than any other in the fray. The respect I have earned, I have earned by other ways than by appearing gravely solemn; and respect has no purpose but what it can accomplish. Respect is not to be hoarded, but spent. Even when those pretending to be Wise chide me, saying: "Stop this bickering!" - yet I will not pretend to neutrality, nor rise above the fray.

For not all conflicts are balanced; indeed an exactly balanced conflict is very rare. Sometimes - indeed often - I have struck out against both sides in a dispute, saying: "You are both wrong; here is the third way." But never have I told both sides of a dispute: "It doesn't matter who started it, just end it." This is the path of convenience to yourself, and it comes at a cost to others; it is selfish. There are aggressors and aggressed, in wars. When some small nation is invaded by another, or is provoked endlessly, or when one nation provokes another and that other responds disproportionately; then it may prove convenient indeed to the Great Powers, to pretend that all violence is equally wrong and equally the fault of all sides, and selfishly seek a truce for this year, this election. The Great Powers have no need to take sides, when they can more easily tell the two edges of the gaping wound: "Oh, just stop fighting, you foolish children!" Ignoring the rottenness inside... but that is pragmatism to a Great Power, which only wishes that the boat should go unrocked, and does not truly care for the health of lesser nations.

And so too with those who pretend to be Wise: who pretend that there is no aggrieved, that there is no long-term problem to be addressed, that no side is ever in the right nor another in the wrong; that there are no causes for conflicts, only fools who are not Wise and who will spontaneously strike out at each other for no reason. It only takes one to start a war. But the Wise cannot acknowledge this in any particular case, for then they would be taking sides, and they would not be above the fray, merely another combatant. They would lose the awe, in which the Wise are held, and which they most earnestly desire.

I do not say that the Wise do this deliberately; but it is the constraint that settles around them, the invisible chain that governs their behavior. No doubt the Wise truly believe that the combatants are but spoiled children; for if the Wise ceased to believe this, they would have to act, and no longer appear Wise.

Have you not met them? the principal who cares not which child started it? the Chair who is above the mere fray of corporate politics? the Great Power who demands only an immediate truce? the priest who says that all alike are sinners and all must repent? the boss who sternly dictates that the conflict end now? have you not met them, the Wise?

To care about the public image of any virtue - whether that virtue be charity, or wisdom, or rationality itself - is to limit your performance of that virtue to what all others already understand of it. Therefore those who would walk the Way of any virtue must first relinquish all desire to appear virtuous before others, for this will limit and constrain their understanding of the virtue. To earn the respect of others is not hard, if you have a little talent, if that is the limit of your desire. But to know what is true, or to do what is right - that is far harder than convincing an audience of your wisdom. I am not Wise, and I will not be Wise, and no one can be Wise if they would follow the Way of rationality.

For the eye of the Wise is blinded, and it may sometimes miss the gaping obvious.

I prefer yesterday's post (which is why I wrote it). But I also suspect that yesterday's post is morepersuasive,signaling more maturity and deliberately avoiding flashes of eloquence that might provoke skepticism here... while containing pretty much exactly the same message-payload.

On the other hand, this version seems easier to read, and you might find it more persuasive if you had just encountered it on the Net - if you weren'tusedto a different style from me.

(Nowadays if I have something to say that soundssuspiciouslyeloquent, I'll create a character and havethemsay it in dialogue or fiction; this lets me have my cake and eat it too. Though - important disclaimer - many of my characters are also there to say eloquent things that Idisagreewith, c.f. theSuperhappies.)

What think you? Criticism can be addressed to me personally, I guess; Eliezer_2006is still someone who I'd talk about as "me".

Original with comments: Wise Pretensions v.0

Rationality Quotes 27

Eliezer Yudkowsky, 22 February 2009 01:55AM

"Believing this statement will make you happier."
-- Ryan Lortie

"Make changes based on your strongest opportunities, not your most convenient ones."
-- MegaTokyo

"The mind is a cruel, lying, unreliable bastard that can't be trusted with even an ounce of responsibility. If you were dating the mind, all your friends would take you aside, and tell you that you can really do better, and being alone isn't all that bad, anyway. If you hired the mind as a babysitter, you would come home to find all but one of your children in critical condition, and the remaining one crowned 'King of the Pit'."
-- Lore Sjoberg

"Getting bored is a non-trivial cerebral transformation that doubtlessly took many millions of years for nature to perfect."
-- Lee Corbin

"The views expressed here do not necessarily represent the unanimous view of all parts of my mind."
-- Malcolm McMahon

"The boundary between these two classes is more porous than I've made it sound. I'm always running into regular dudes--construction workers, auto mechanics, taxi drivers, galoots in general--who were largely aliterate until something made it necessary for them to become readers and start actually thinking about things. Perhaps they had to come to grips with alcoholism, perhaps they got sent to jail, or came down with a disease, or suffered a crisis in religious faith, or simply got bored. Such people can get up to speed on particular subjects quite rapidly. Sometimes their lack of a broad education makes them over-apt to go off on intellectual wild goose chases, but, hey, at least a wild goose chase gives you some exercise."
-- Neal Stephenson, In the Beginning was the Command Line

Original with comments: Rationality Quotes 27

Fairness vs. Goodness

Eliezer Yudkowsky, 22 February 2009 08:22PM

It seems that back when the Prisoner's Dilemma was still being worked out, Merrill Flood and Melvin Drescher tried a 100-fold iterative PD on two smart but unprepared subjects, Armen Alchian of UCLA and John D. Williams of RAND.

The kicker being that the payoff matrix was asymmetrical, with dual cooperation awarding JW twice as many points as AA:

(AA, JW)

JW: D

JW: C

AA: D

(0, 0.5)

(1, -1)

AA: C

(-1, 2)

(0.5, 1)

The resulting 100 iterations, with a log of comments written by both players, make for fascinating reading.

JW spots the possibilities of cooperation right away, while AA is slower to catch on.

But once AA does catch on to the possibilities of cooperation, AA goes on throwing in an occasional D... because AA thinks the natural meeting point for cooperation is a fair outcome, where both players get around the same number of total points.

JW goes on trying to enforce (C, C) - the option that maximizes total utility for both players - by punishing AA's attempts at defection. JW's log shows comments like "He's crazy. I'll teach him the hard way."

Meanwhile, AA's log shows comments such as "He won't share. He'll punish me for trying!"

I confess that my own sympathies lie with JW, and I don't think I would have played AA's game in AA's shoes. This would seem to indicate that I'm more of a utilitarian than a fair-i-tarian. Life doesn't always hand you fair games, and the best we can do for each other is play them positive-sum.

Though I might have been somewhat more sympathetic to AA, if the (C, C) outcome had actually lost him points, and only (D, C) had made it possible for him to gain them back. For example, this is also a Prisoner's Dilemma:

(AA, JW)

JW: D

JW: C

AA: D

(-2, 2)

(2, 0)

AA: C

(-5, 6)

(-1, 4)

Theoretically, of course, utility functions are invariant up to affine transformation, so a utility's absolute sign is not meaningful. But this is not always a good metaphor for real life.

Of course what we want in this case, societally speaking, is for JW to slip AA a bribe under the table. That way we can maximize social utility while letting AA go on making a profit. But if AA starts out with a negative number in (C, C), how much do we want AA to demand in bribes - from our global, societal perspective?

The whole affair makes for an interesting reminder of the different worldviews that people invent for themselves - seeming so natural and uniquely obvious from the inside - to make themselves the heroes of their own stories.

Original with comments: Fairness vs. Goodness

On Not Having an Advance Abyssal Plan

Eliezer Yudkowsky, 23 February 2009 08:20PM

"Even though he could foresee the problem then, we can see it equally well now. Therefore, if he could foresee the solution then, we should be able to see it now. After all, Seldon was not a magician. There are no trick methods of escaping a dilemma that he can see and we can't."
-- Salvor Hardin

Years ago at the Singularity Institute, the Board was entertaining a proposal to expand somewhat. I wasn't sure our funding was able to support the expansion, so I insisted that - if we started running out of money - we decide in advance who got fired and what got shut down, in what order.

Even over the electronic aether, you could hear the uncomfortable silence.

"Why can't we decide that at the time, if the worst happens?" they said, or something along those lines.

"For the same reason that when you're buying a stock you think will go up, you decide how far it has to decline before it means you were wrong," I said, or something along those lines; this being far back enough in time that I would still have used stock-trading in a rationality example. "If we can make that decision during a crisis, we ought to be able to make it now. And if I can't trust that we can make this decision in a crisis, I can't trust this to go forward."

People are really, really reluctant to plan in advance for the abyss. But what good reason is there not to? How can you be worse off from knowing in advance what you'll do in the worse cases?

I have been trying fairly hard to keep my mouth shut about the current economic crisis. But still -

Why didn't various governments create and publish a plan for what they would do in the event of various forms of financial collapse, before it actually happened?

Never mind hindsight on the real-estate bubble - there are lots of things that could potentially trigger financial catastrophes. I'm willing to bet the American government knows what it will do in terms of immediate rescue operations if an atomic bomb goes off in San Francisco. But if the US government had any advance idea of under which circumstances it would nationalize Fannie Mae or guarantee Bear Stearns's counterparties, this plan was not very much in evidence as various government officials gave every appearance of trying to figure everything out on the fly.

A published, believable advance plan for the worst case - one that you could actually believe the government would carry out, instead of junking the plan to try to keep the top spinning a little longer - would have made the markets that much less uncertain.

If you don't publish a plan for catastrophe - or can't publish a believable plan - then the market just tries to guess what a realistic, believable plan would look like. If that realistic, believable plan involves frantically attempting to bail out the large financial entities in order to keep the whole system from melting down further, you have moral hazard. If they actually do it, that's lemon socialism (privatized upside, public downside).

If that's what happens in the abyssal case - then not publishing that fact, doesn't prevent anyone from foreseeing it. If you publish that plan, maybe it will start a debate about whether to break up Bear Stearns into smaller entities, or change the plan to give counterparties a predictable 10% haircut, or claw back executive bonuses no matter what their contracts read (because you really aren't supposed to screw up so badly that the government has to get involved)...

But if you can't publish a realistic, believable advance abyssal plan that doesn't call for rescuing the huge entities - then who are you even kidding?

Governmental agencies failing to stare into the abyss in advance gives us a double problem: moral hazard as counterparties and investors try to guess what the government will realistically do; and fear and uncertainty in the market when the worst does happen.

It's questionable whether the government should be in the position of trying to forecast the abyss - to put a probability on financial meltdown in any given year due to any given cause. But advance abyssal planning isn't about the probability, as it would be in investing. It's about the possibility. If you can realistically imagine global financial meltdowns of various types being possible, there's no excuse for not war-gaming them. If your brain doesn't literally cease to exist upon facing systemic meltdowns at the time, you ought to be able to imagine plausible systemic meltdowns in advance.

Sure, you might have to make some modifications on-the-fly because you didn't get the exact causes and circumstances right. But it shouldn't be obvious and predictable that the modifications will consist of "Oh dear it's more awful than we planned for and the systemic hazard is worse and now we really do have to bail out everyone even though we said we wouldn't." Then the plan is not believable.

So long as the plan is not wrong in the stupidly obvious directions, it's hard to see how we'd be worse off if the governors of the Federal Reserve had taken a week once per year to play through scenarios more nightmarish than this one in their minds, deciding in advance what to do about it, realistically.

I suppose the main argument against publishing the plan would be that the uninformed public (i.e. Congress) would revolt against the emergency plans, demanding that unbelievable plans be substituted (let the banks burn! don't bail out GM!) and then changing their tune as soon as the worst actually happened.

But at least having the Federal Reserve privately visualizing all sorts of hideous possibilities in advance, war-gaming them with the Board of Governors, and planning for them realistically - so that when the worst starts happening, you don't have everyone running around being vague and visibly unprepared and refusing to talk about what happens if things get even worse - instead you just take out folder #37-B and figure out what needs tweaking - for the lack of that preparedness, there seems to me to be very little excuse. The Federal Reserve should not be in the business of forecasting probabilities - they've already demonstrated that they can't, and they're not investors. They should just be always staring into the abyss.

Of course the Federal Reserve doesn't read this blog, so far as I know. But it's the sort of thing that doesn't require a majority vote for individuals to use in their personal lives.

Original with comments: On Not Having an Advance Abyssal Plan

Formative Youth

Eliezer Yudkowsky, 24 February 2009 11:02PM

Followup to: Against Maturity

"Rule of thumb: Be skeptical of things you learned before you could read. E.g., religion."
-- Ben Casnocha

Looking down on others is fun, and if there's one group we adults can all enjoy looking down on, it's children. At least I assume this is one of the driving forces behind the incredible disregard for... but don't get me started.

Inconveniently, though, most of us were children at one point or another during our lives. Furthermore, many of us, as adults, still believe or choose certain things that we happened to believe or choose as children. This fact is incongruent with the general fun of condescension - it means that your life is being run by a child, even if that particular child happens to be your own past self.

I suspect that most of us therefore underestimate the degree to which our youths were formative - because to admit that your youth was formative is to admit that the course of your life was not all steered by Incredibly Deep Wisdom and uncaused free will.

To give a concrete example, suppose you asked me, "Eliezer, where does your altruism originally come from? What was the very first step in the chain that made you amenable to helping others?"

Then my best guess would be "Watching He-Man and similar TV shows as a very young and impressionable child, then failing to compartmentalize the way my contemporaries did." (Same reason my Jewish education didn't take; I either genuinely believed something, or didn't believe it at all. (Not that I'm saying that I believed He-Man was fact; just that the altruistic behavior I picked up wasn't compartmentalized off into some safely harmless area of my brain, then or later.))

It's my understanding that most people would be reluctant to admit this sort of historical fact, because it makes them sound childish - in the sense that they're still being governed by the causal history of a child.

But I find myself skeptical that others are governed by their childhood causal histories so much less than myself - especially when there's a simple alternative explanation: they're too embarrassed to admit it.

A lovely excuse, of course, is that we at first ended up in a certain place for childish reasons, and then we went back and redid the calculations as adults, and what do you know, it magically ended up with the same bottom line.

Well - of course that can happen. If you ask me why I'm out to save the world, then there's a sense in which I can defend that as a sober utilitarian calculation, "Shut up and multiply", that has nothing to do with spending my childhood reading science fiction about protagonists who saved the world. But if you ask me why I listen to that sober utilitarian calculation, why it actually has the capacity to move me - then yes, the fact that the first "grownup" book I read was Dragonflight may have played a role. It's what F'lar and Lessa would do.

Why not really start over from scratch - throw away our childhoods and redo everything?

For epistemic beliefs that might be sorta-possible, which is why I didn't name an epistemic belief that I think I inherited from the chaos of childhood. That wouldn't be tolerable, and when I look back, I really have rejected a lot of what I once believed epistemically.

But matters of taste? Of personality? Of deeply held ideals and values?

Well, yes, I reformulated my whole metaethics at a certain point and that had a definite influence on my values... but despite that, I think you could draw an obvious line back from where I am now, to factors like reading Dragonlance at age nine and vowing never to end up like Raistlin Majere. (Bitter genius archetype.)

If you can't look back and draw a line between your current adult self and factors like that, I have to wonder if your self-history is really accurate.

In particular, I have to wonder if you're thinking right now of a deceptively obvious-seeming line that someone else might be tempted to draw, but which of course isn't the real reason why you still...

PS: Of course I don't directly justify any of my decisions, these days, by saying "That's what the Thundercats did, therefore it is right." The question is more like whether I ended up finding developed altruistic philosophies more appealing as an adult because, sometime back in my youth, I was bombarded with altruistic messages.

If there are many different stores selling developed philosophies, then which store you walk into to buy your sophisticated adult judgments might depend on a factor like that.

PPS: Several commenters asked why I focused on fiction. I could point to several real-life events in my childhood that I still remember and that seem promisingly characteristic of "me" - for example, the only time I remember my kindergarten classmates ever praising me or liking me was the time I used wooden blocks to build a complicated track that they could "ski" along. Making something clever = peer approval, says this memory.

But because this was a one-off event, I doubt it would have quite as much influence as messages repeated over and over, through many different TV shows with similar themes, or many different books written by science-fiction authors who influenced one another. I couldn't recite the plot of even a single episode of He-Man, but I have some memory of what the opening theme song was, because it was recurring. That's the power of a fictional corpus, relative to any single moment of real life no matter how significant it seems - fictions can repeat the same message over and over.

My childhood universe was very much a universe of books. The nonfiction I read (like the Childcraft books) might have been formative in a sense - but factual beliefs you really can recheck and redo. Hence my citation of fiction as a lingering influence on values and personality.

Original with comments: Formative Youth

Tell Your Rationalist Origin Story

Eliezer Yudkowsky, 25 February 2009 05:16PM

To break up the awkward silence at the start of a recent Overcoming Bias meetup, I asked everyone present to tell their rationalist origin story - a key event or fact that played a role in their becoming rationalists. This worked surprisingly well (and I would recommend it for future meetups).

I think I've already told enough of my own origin story on Overcoming Bias: how I was digging in my parents' yard as a kid and found a tarnished silver amulet inscribed with Bayes's Theorem, and how I wore it to bed that night and dreamed of a woman in white, holding an ancient leather-bound book called Judgment Under Uncertainty: Heuristics and Biases (eds. D. Kahneman, P. Slovic, and A. Tversky, 1982)... but there's no need to go into that again.

So, seriously... how did you originally go down that road?

Added: For some odd reason, many of the commenters here seem to have had a single experience in common - namely, at some point, encountering Overcoming Bias... But I'm especially interested in what it takes to get the transition started - crossing the first divide. This would be very valuable knowledge if it can be generalized. If that did happen at OB, please try to specify what was the crucial "Aha!" insight (down to the specific post if possible).

Referenced by: Tell Your Rationalist Origin Story... at Less Wrong

Original with comments: Tell Your Rationalist Origin Story

Markets are Anti-Inductive

Eliezer Yudkowsky, 26 February 2009 12:55AM

I suspect there's a Pons Asinorum of probability between the bettor who thinks that you make money on horse races by betting on the horse you think will win, and the bettor who realizes that you can only make money on horse races if you find horses whose odds seem poorly calibrated relative to superior probabilistic guesses.

There is, I think, a second Pons Asinorum associated with more advanced finance, and it is the concept that markets are an anti-inductive environment.

Let's say you see me flipping a coin. It is not necessarily a fair coin. It's a biased coin, and you don't know the bias. I flip the coin nine times, and the coin comes up "heads" each time. I flip the coin a tenth time. What is the probability that it comes up heads?

If you answered "ten-elevenths, by Laplace's Rule of Succession", you are a fine scientist in ordinary environments, but you will lose money in finance.

In finance the correct reply is, "Well... if everyone else also saw the coin coming up heads... then by now the odds are probably back to fifty-fifty."

Recently on Hacker News I saw a commenter insisting that stock prices had nowhere to go but down, because the economy was in such awful shape. If stock prices have nowhere to go but down, and everyone knows it, then trades won't clear - remember, for every seller there must be a buyer - until prices have gone down far enough that there is once again a possibility of prices going up.

So you can see the bizarreness of someone saying, "Real estate prices have gone up by 10% a year for the last N years, and we've never seen a drop." This treats the market like it was the mass of an electron or something. Markets are anti-inductive. If, historically, real estate prices have always gone up, they will keep rising until they can go down.

To get an excess return - a return that pays premium interest over the going rate for that level of riskiness - you need to know something that other market participants don't, or they will rush in and bid up whatever you're buying (or bid down whatever you're selling) until the returns match prevailing rates.

If the economy is awful and everyone knows it, no one's going to buy at a price that doesn't take into account that knowledge.

If there's an obvious possibility of prices dropping further, then the market must also believe there's a probability of prices rising to make up for it, or the trades won't clear.

This elementary point has all sorts of caveats I'm not bothering to include here, like the fact that "up" and "down" is relative to the risk-free interest rate and so on. Nobody believes the market is really "efficient", and recent events suggest it is less efficient than previously believed, and I have a certain friend who says it's even less efficient than that... but still, the market does not leave hundred-dollar-bills on the table if everyone believes in them.

There was a time when the Dow systematically tended to drop on Friday and rise on Monday, and once this was noticed and published, the effect went away.

Past history, e.g. "real estate prices have always gone up", is not private info.

And the same also goes for more complicated regularities. Let's say two stock prices are historically anticorrelated - the variance in their returns moves in opposite directions. As soon as everyone believes this, hedge-fund managers will leverage up and buy both stocks. Everyone will do this, meaning that both stocks will rise. As the stocks rise, their returns get more expensive. The hedge-fund managers book profits, though, because their stocks are rising. Eventually the stock prices rise to the point they can go down. Once they do, hedge-fund managers who got in late will have to liquidate some of their assets to cover margin calls. This means that both stock prices will go down - at the same time, even though they were originally anticorrelated. Other hedge funds may lose money on the same two stocks and also sell or liquidate, driving the price down further, etcetera. The correlative structure behaves anti-inductively, because other people can observe it too.

If mortage defaults are historically uncorrelated, so that you can get an excess return on risk by buying lots of mortages and pooling them together, then people will rush in and buy lots of mortgages until (a) rates on mortgages are bid down (b) individual mortgage failure rates rise (c) mortgage failure rates become more correlated, possibly looking uncorrelated in the short-term but having more future scenarios where they all fail at once.

Whatever is believed in, stops being real. The market is literally anti-inductive rather than anti-regular - it's the regularity that enough participants induce, which therefore goes away.

This, as I understand it, is the standard theory of "efficient markets", which should perhaps have been called "inexploitable markets" or "markets that are not easy to exploit because others are already trying to exploit them". Should I have made a mistake thereof, let me be corrected.

Now it's not surprising, on the one hand, to see this screwed up in random internet discussions where a gold bug argues from well-known observations about the past history of gold. (This is the equivalent of trying to make money at horse-racing by betting on the horse that you think will win - failing to cross the Pons Asinorum.)

But it is surprising is to hear histories of the financial crisis in which prestigious actors argued in crowded auditoriums that, previously, real-estate prices had always gone up, or that previously mortage defaults had been uncorrelated. This is naive inductive reasoning of the sort that only works on falling apples and rising suns and human behavior and everything else in the universe except markets. Shouldn't everyone have frowned and said, "But isn't the marketplace an anti-inductive environment?"

Not that this is standard terminology - but perhaps "efficient market" doesn't convey quite the same warning as "anti-inductive". We would appear to need stronger warnings.

PS: To clarify, the coin example is a humorous exaggeration of what the world would be like if most physical systems behaved the same way as market price movements, illustrating the point, "An exploitable pricing regularity that is easily inducted degrades into inexploitable noise." Here the coin coming up "heads" is analogous to getting an above-market return on a publicly traded asset.

Original with comments: Markets are Anti-Inductive

Tell Your Rationalist Origin Story... at Less Wrong

Eliezer Yudkowsky, 27 February 2009 04:06AM

(A beta version of Less Wrong is now live, no old posts imported as yet. Some of the plans for what to do with Less Wrong relative to OB have been revised by further discussion among Robin, Nick, and myself, but for now we're just seeing what happens once LW is up - whether it's stable, what happens to the tone of comments once threading and voting is enabled, etcetera.

Posting by non-admins is disabled for now - today we're just testing out registration, commenting, threading, etcetera.)

To break up the awkward silence at the start of a recent Overcoming Bias meetup, I asked everyone present to tell their rationalist origin story - a key event or fact that played a role in their becoming rationalists. This worked surprisingly well.

I think I've already told enough of my own origin story on Overcoming Bias: how I was digging in my parents' yard as a kid and found a tarnished silver amulet inscribed with Bayes's Theorem, and how I wore it to bed that night and dreamed of a woman in white, holding a leather-bound book called Judgment Under Uncertainty: Heuristics and Biases (eds. D. Kahneman, P. Slovic, and A. Tversky, 1982)... but there's no need to go into that again.

So, seriously... how did you originally go down that road?

Continue reading "Tell Your Rationalist Origin Story" at Less Wrong Âť

Original with comments: Tell Your Rationalist Origin Story... at Less Wrong

The Most Important Thing You Learned

Eliezer Yudkowsky, 27 February 2009 08:15PM

My current plan does still call for me to write a rationality book - at some point, and despite all delays - which means I have to decide what goes in the book, and what doesn't. Obviously the vast majority of my OB content can't go into the book, because there's so much of it.

So let me ask - what was the one thing you learned from my posts on Overcoming Bias, that stands out as most important in your mind? If you like, you can also list your numbers 2 and 3, but it will be understood that any upvotes on the comment are just agreeing with the #1, not the others. If it was striking enough that you remember the exact post where you "got it", include that information. If you think the most important thing is for me to rewrite a post from Robin Hanson or another contributor, go ahead and say so. To avoid recency effects, you might want to take a quick glance at this list of all my OB posts before naming anything from just the last month - on the other hand, if you can't remember it even after a year, then it's probably not the most important thing.

Please also distinguish this question from "What was the most frequently useful thing you learned, and how did you use it?" and "What one thing has to go into the book that would (actually) make you buy a copy of that book for someone else you know?" I'll ask those on Saturday and Sunday.

PS: Do please think of your answer before you read the others' comments, of course.

Referenced by: The Most Frequently Useful Thing • Practical Advice Backed By Deep Theories

Original with comments: The Most Important Thing You Learned

The Most Frequently Useful Thing

Eliezer Yudkowsky, 28 February 2009 06:43PM

Followup to: The Most Important Thing You Learned

What's the most frequently useful thing you've learned on OB - not the most memorable or most valuable, but the thing you use most often? What influences your behavior, factors in more than one decision? Please give a concrete example if you can. This isn't limited to archetypally "mundane" activities: if your daily life involves difficult research or arguing with philosophers, go ahead and describe that too.

Referenced by: The Most Frequently Useful Thing • That You'd Tell All Your Friends

Original with comments: The Most Frequently Useful Thing

The Most Frequently Useful Thing

Eliezer Yudkowsky, 28 February 2009 07:54PM

What's the most frequently useful thing you've learned on OB - not the most memorable or most valuable, but the thing you use most often?  What influences your behavior, factors in more than one decision?  Please give a concrete example if you can. This isn't limited to archetypally "mundane" activities: if your daily life involves difficult research or arguing with philosophers, go ahead and describe that too.

Continue reading "The Most Frequently Useful Thing" at Less Wrong Âť

Original with comments: The Most Frequently Useful Thing

That You'd Tell All Your Friends

Eliezer Yudkowsky, 01 March 2009 12:04PM

Followup to: The Most Frequently Useful Thing

What's the number one thing that goes into a book on rationality, which would make you buy a copy of that book for a friend? We can, of course, talk about all the ways that the rationality of the Distant World At Large needs to be improved. But in this case - I think the more useful data might be the Near question, "With respect to the people I actually know, what do I want to see in that book, so that I can give the book to them to explain it?"

(And again, please think of your own answer-component before reading others' comments.)

Original with comments: That You'd Tell All Your Friends

Unteachable Excellence

Eliezer Yudkowsky, 02 March 2009 03:33PM

There's a whole genre of literature whose authors want to sell you the secret success sauce behind Gates's Microsoft or Buffett's Berkshire Hathaway - the common theme being that you, yes, you can be the next Larry Page.

But probably not even Warren Buffett can teach you to be the next Warren Buffett. That kind of extraordinary success is extraordinary because no one has yet figured out how to teach it reliably.

And so mostly these books are a waste of hope, feeding off the excitement from dangling the possibility of the glorious yet unattainable; which is why I call them "excellence pornography", with subgenres like investment pornography and business pornography, telling every barista how to run the next Starbucks and every MBA student how to be the best CEO in the Fortune 500. Calling this "excellence pornography" might be too unkind to pornography, which is at least overtly fiction.

Now, there are incredibly powerful techniques that civilization has figured out how to teach: techniques like "test your ideas by experiment" or "reinvest your wealth to generate more wealth". You, yes, you can be a scientist! Or maybe not everyone - but enough people can become scientists by using learnable techniques and communicable knowledge, to support our technological civilization.

"You, yes, you can reinvest the proceeds of your earlier investments!" You may not beat the market like Warren Buffett. But if you think about a whole civilization practicing that rule, we do better nowadays than historical societies with no banks or stock markets. (No, really, we still do better on net.) Because the trick of Reinvestment can be taught, can be described in words, can work for ordinary people without extraordinary luck... we don't think of it as an extraordinary triumph. Just anyone can do it, so it must not be important.

Warren Buffett did manage to turn on a lot of people to value investing. He's given out a lot of advice, and it looks like good advice to me on the occasions I've read it. The impression I get, at least, is that if he knew how to communicate what was left, he would just tell you.

But Berkshire Hathaway, and Buffett personally, still spend huge amounts of time looking for managerial excellence. Why? Because they don't know any systematically reliable process to start with bright kids and turn them into Fortune 500 CEOs.

There are things you can learn from the superstars. But you can't expect to eat their whole soul, and the last mile of their extraordinariness will be the hardest for them to teach. You will, at best, learn a few useful tricks that a lot of other people can learn as well, and that won't put you anywhere near the delicious high status of the superstar. Unless, of course, you yourself have the right mix of raw genetic talents and you put years and years into training yourself and have a lot of luck along the way, etcetera, but the point is, you won't get there by reading pornography.

(If someone actually does come up with a new teachable supertrick, so that civilization itself is about to take another lurching step forward, then you should expect to have a lot of fellow superstars by the time you're done learning!)

There's a number of lessons that I draw from this point; but one of the main ones is that much of the most important information we can learn from history is about how to not lose, rather than how to win.

It's easier to avoid duplicating spectacular failures than to duplicate spectacular successes. And it's often easier to generalize failure between domains. The instructions for "how to be a superstar" tend to be highly specific and domain-specialized (Buffett ≠ Einstein) but the lessons for "how not to be an idiot" have a lot more in common between professions.

Ken Lay can teach you how not to be the next Enron a lot more easily than Warren Buffett can teach you to be the next Berkshire Hathaway. Casey Serin can teach you how to lose hope, Lord Kelvin can teach you not to worship your own ignorance...

But that kind of lesson won't make you a glorious superstar. It may prevent your life from becoming miserable, but this is not as glamorous. And even worse - this kind of lesson may end up showing you that you're doing something wrong, that you yes you are about to join the ranks of fools.

It's a lot easier to sell excellence pornography.

Referenced by: Teaching the Unteachable • Mandatory Secret Identities • Great Books of Failure

Original with comments: Unteachable Excellence

Teaching the Unteachable

Eliezer Yudkowsky, 03 March 2009 11:14PM

Previously in series: Unteachable Excellence
Followup to: Artificial Addition

The literary industry that I called "excellence pornography" isn't very good at what it does. But it is failing at a very important job. When you consider the net benefit to civilization of Warren Buffett's superstar skills, versus the less glamorous but more communicable trick of "reinvest wealth to create more wealth" - there's hardly any comparison. You can see how much it would matter, if you could figure out how to communicate just one more skill that used to be a secret sauce. Not the pornographic promise of consuming the entire soul of a superstar. Just figuring out how to reliably teach one more thing, even if it wasn't everything...

What makes a success hard to duplicate?

Naked statistical chance is always incommunicable. No matter what you say about your historical luck, you can't teach someone else to have it. The arts of seizing opportunity, and exposing yourself to positive randomness, are commonly underestimated; I've seen people stopped in their tracks by "bad luck" that a Silicon Valley entrepreneur would drive over like a steamroller flattening speed bumps... Even so, there is still an element of genuine chance left over.

Einstein's superstardom depended on his genetics that gave him the potential to learn his skills. If a skill relies on having that much brainpower, you can't teach it to most people... Though if the potential is one-in-a-million, then six thousand Einsteins around the world would be an improvement. (And if we're going to be really creative, who says genes are incommunicable? It just takes more advanced technology than a blackboard, that's all.)

So when we factor out the genuinely unteachable - what's left? Where you can you push the border? What is it that might be possible to teach - albeit perhaps very difficult - and isn't being taught?

I was once told that half of Nobel laureates were the students of other Nobel laureates. This source seems to assert 155 out of 503. (Interestingly, the same source says that the number of Nobel laureates with Nobel "grandparents" (teachers of teachers) is just 60.) Even after discounting for cherry-picking of students and political pull, this suggests to me that you can learn things by apprenticeship - close supervision, free-form discussion, ongoing error correction over a long period of time - that no Nobel laureate has yet succeeding in putting into any of their many books.

What is it that the students of Nobel laureates learn, but can't put into words?

This subject holds a fascination for me, because how it delves into the meta, the source behind, the gap between the output and the generator. We can explain Einstein's General Relativity to students, but we can't make them Einstein. (If you look at it from the right angle, the whole trick of human intelligence is just an incommunicable insight that humans have and can't explain to a computer.)

The amount of wordless intelligence in our work tends to be underestimated because the words themselves are so much easier to introspect on. But when I'm paying attention, I can see how much of my searchpower takes place in fast flashes of perception that tell me what's important, which thought to think next.

When I met my apprentice Marcello he was already better at mathematical proof than myself, certainly much faster. He'd competed at the national level - but in competitions like that you get told which problems are important. (And also in competitions, you instantly hand in the problem when you're done, and rush on to the next one; without looking over your proof to see if you can simplify it, see it at a glance, learn something more.) But the really critical thing I was trying to teach him - testing to see if it could even be taught at all - was this sense of which AI problems led somewhere. "You can pedal as well as I can," I said to him early on when he asked how he was doing, "but I'm still doing ninety percent of the steering." And it was a constant, tremendous struggle to put anything into words about why I thought that we hadn't yet found the really important insight that was lurking somewhere in a problem, and so we were going to discard Marcello's current proof and reformulate the problem and try again from another angle, to see if this time we would really understand something.

We go through our life events, and our brain uses an opaque algorithm to grind the experiences to grist, and outputs yet another opaque neural net of circuitry: the procedural skill, the source of wordless intuitions that you know so fast you can't see yourself knowing them. "The zeroth step", I called it, the step in reasoning that comes before the first step and goes by so quickly that you don't realize it's there.

I pride myself on being good at putting things into words, at being able to introspect on the momentary flashes and see their pattern and trend, even if I can't print out the circuitry that is their source. But when I tried to communicate my cutting edge, the borderline where I advanced my knowledge - then my words were defeated, and I was left working with Marcello on problem after problem, hoping his brain would pick up that unspoken rhythm of the steering: Turn left, turn right; this is probably worth pursuing, this is not; this seems like a valuable insight, this is just a black box around our ignorance.

I'd expected it to go like that; I'd never had the delusion that the most important parts of thought would be easy to put in words. If it were that simple we really would have had Artificial Intelligence in the 1970s.

Civilization gets by on teaching the output of the generator without teaching the generator. Einstein output his various discoveries, and then the generated knowledge is verbal enough to be passed on to students in university. When another Einstein is needed, civilization just holds its breath and hopes.

But if these wordless skills are the product of experience - then why not communicate the experiences? Or if fiction isn't good enough, and it probably isn't even close, then why not duplicate the experiences - put people through the same events?

(1) Superstars may not know what their critical experiences were.

(2) The critical experiences may be difficult to duplicate - for example, everyone already knows the answer to Special Relativity, and now we can't train people by giving them the problem of Special Relativity. Just knowing that it has something to do with space and time shifting around, is already too much of a spoiler. The really important part of the problem is the one where you stare at a blank sheet of paper until drops of blood form on your forehead, trying to figure out what to think next. The skills of genius are rare, I've suggested, because there is not enough opportunity to practice them.

(3) There may be luck or genetic talent involved in your brain hitting on the right thing to learn - finding a solution of high quality in the space of wordless procedural skills. Even if we put you through the same experiences, there's components of true chance and genetic talent left over in having your brain learn the same wordless skill.

But I think there's still reason to go on trying to describe the indescribable and teach the unteachable.

Consider the transition in gambling skill associated with the invention of probability theory a few centuries back. There's still a leftover art to poker, wordless skills that poker superstars can only partially describe in words. But go back far enough, and no one would have any idea how to calculate the odds of rolling three dice and coming up with all ones. And maybe an experienced enough gambler would have a wordless intuition that some things were likelier than others, but they couldn't have put it into words - couldn't have told anyone else what they'd learned about the chances; except, maybe, through a long process of watching over an apprentice's shoulder and supervising their bets.

The more we learn about a domain, and the more we systematically observe the stars at work, and the more we learn about the human mind in general, the more we can hope for new skills to make the transition from unteachable to apprenticeable to publishable.

And you can hope to trailblaze certain paths, even if you can't set down all the path in words. Even if you yourself got somewhere through luck (including genetic luck), you can hope to diminish the role of luck on future occasions:

(A) Warning against blind alleys that delayed you, is one obvious sort of help.

(B) If you lay down a set of thoughts that are the product of wordless skills, someone reading through the set of thoughts may find their brain picking up the rhythm, making the leap to the unspoken thing behind; and this might require less luck than the events that led to your own original acquisition of those wordless skills.

(C) There are good attractors in the solution-space - clustered sub-solutions which make it easier to reach other solutions in the same attractor. Then - even if some of the thoughts can't be put into words, and even if it took a lot of luck to wander into the attractor the first time through - describing everything that can be put into words, may be enough to anchor the attractor.

(D) Some important experiences are duplicable: for example, you can advise people what areas to study, what books to read.

(E) And finally, the simple advance of science may just describe a domain better, so that you realize what it is you know, and are suddenly able to communicate it outright.

And of course the punchline is that this is the transition I hope to see in certain aspects of human rationality - skills which have been up until now unteachable, or only passed down from master to apprentice. We've learned a lot about the domain in the past few decades, and I think it's time to take another shot at systematizing it.

I aspire to diminish the role of luck and talent in producing rationalists of a higher grade.

Referenced by: A Sense That More Is Possible

Original with comments: Teaching the Unteachable

No, Really, I've Deceived Myself

Eliezer Yudkowsky, 04 March 2009 11:29PM

Followup to: Belief in Belief

I recently spoke with a person who... it's difficult to describe. Nominally, she was an Orthodox Jew. She was also highly intelligent, conversant with some of the archaeological evidence against her religion, and the shallow standard arguments against religion that religious people know about. For example, she knew that Mordecai, Esther, Haman, and Vashti were not in the Persian historical records, but that there was a corresponding old Persian legend about the Babylonian gods Marduk and Ishtar, and the rival Elamite gods Humman and Vashti. She knows this, and she still celebrates Purim. One of those highly intelligent religious people who stew in their own contradictions for years, elaborating and tweaking, until their minds look like the inside of an M. C. Escher painting.

Most people like this will pretend that they are much too wise to talk to atheists, but she was willing to talk with me for a few hours.

As a result, I now understand at least one more thing about self-deception that I didn't explicitly understand before - namely, that you don't have to really deceive yourself so long as you believe you've deceived yourself. Call it "belief in self-deception".

When this woman was in high school, she thought she was an atheist. But she decided, at that time, that she should act as if she believed in God. And then - she told me earnestly - over time, she came to really believe in God.

So far as I can tell, she is completely wrong about that. Always throughout our conversation, she said, over and over, "I believe in God", never once, "There is a God." When I asked her why she was religious, she never once talked about the consequences of God existing, only about the consequences of believing in God. Never, "God will help me", always, "my belief in God helps me". When I put to her, "Someone who just wanted the truth and looked at our universe would not even invent God as a hypothesis," she agreed outright.

She hasn't actually deceived herself into believing that God exists or that the Jewish religion is true. Not even close, so far as I can tell.

On the other hand, I think she really does believe she has deceived herself.

So although she does not receive any benefit of believing in God - because she doesn't - she honestly believes she has deceived herself into believing in God, and so she honestly expects to receive the benefits that she associates with deceiving oneself into believing in God; and that, I suppose, ought to produce much the same placebo effect as actually believing in God.

And this may explain why she was motivated to earnestly defend the statement that she believed in God from my skeptical questioning, while never saying "Oh, and by the way, God actually does exist" or even seeming the slightest bit interested in the proposition.

Sequence: Against Doublethink

Doublethink (Choosing to be Biased)

Belief in Self-Deception

Referenced by: Belief in Self-Deception • Moore's Paradox • Don't Believe You'll Self-Deceive • Can Humanism Match Religion's Output? • Incremental Progress and the Valley

Original with comments: No, Really, I've Deceived Myself

Belief in Self-Deception

Eliezer Yudkowsky, 05 March 2009 03:20PM

Continuation of: No, Really, I've Deceived Myself
Followup to: Dark Side Epistemology

I spoke yesterday of my conversation with a nominally Orthodox Jewish woman who vigorously defended the assertion that she believed in God, while seeming not to actually believe in God at all.

While I was questioning her about the benefits that she thought came from believing in God, I introduced the Litany of Tarski - which is actually an infinite family of litanies, a specific example being:

If the sky is blue
I desire to believe "the sky is blue"
If the sky is not blue
I desire to believe "the sky is not blue".

"This is not my philosophy," she said to me.

"I didn't think it was," I replied to her. "I'm just asking - assuming that God does not exist, and this is known, then should you still believe in God?"

She hesitated. She seemed to really be trying to think about it, which surprised me.

"So it's a counterfactual question..." she said slowly.

I thought at the time that she was having difficulty allowing herself to visualize the world where God does not exist, because of her attachment to a God-containing world.

Now, however, I suspect she was having difficulty visualizing a contrast between the way the world would look if God existed or did not exist, because all her thoughts were about her belief in God, but her causal network modelling the world did not contain God as a node. So she could easily answer "How would the world look different if I didn't believe in God?", but not "How would the world look different if there was no God?"

She didn't answer that question, at the time. But she did produce a counterexample to the Litany of Tarski:

She said, "I believe that people are nicer than they really are."

I tried to explain that if you say, "People are bad," that means you believe people are bad, and if you say, "I believe people are nice", that means you believe you believe people are nice. So saying "People are bad and I believe people are nice" means you believe people are bad but you believe you believe people are nice.

I quoted to her:

"If there were a verb meaning 'to believe falsely', it would not have any
significant first person, present indicative."
-- Ludwig Wittgenstein

She said, smiling, "Yes, I believe people are nicer than, in fact, they are. I just thought I should put it that way for you."

"I reckon Granny ought to have a good look at you, Walter," said Nanny. "I reckon
your mind's all tangled up like a ball of string what's been dropped."
-- Terry Pratchett, Maskerade

And I can type out the words, "Well, I guess she didn't believe that her reasoning ought to be consistent under reflection," but I'm still having trouble coming to grips with it.

I can see the pattern in the words coming out of her lips, but I can't understand the mind behind on an empathic level. I can imagine myself into the shoes of baby-eating aliens and the Lady 3rd Kiritsugu, but I cannot imagine what it is like to be her. Or maybe I just don't want to?

This is why intelligent people only have a certain amount of time (measured in subjective time spent thinking about religion) to become atheists. After a certain point, if you're smart, have spent time thinking about and defending your religion, and still haven't escaped the grip of Dark Side Epistemology, the inside of your mind ends up as an Escher painting.

(One of the other few moments that gave her pause - I mention this, in case you have occasion to use it - is when she was talking about how it's good to believe that someone cares whether you do right or wrong - not, of course, talking about how there actually is a God who cares whether you do right or wrong, this proposition is not part of her religion -

And I said, "But I care whether you do right or wrong. So what you're saying is that this isn't enough, and you also need to believe in something above humanity that cares whether you do right or wrong." So that stopped her, for a bit, because of course she'd never thought of it in those terms before. Just a standard application of the nonstandard toolbox.)

Later on, at one point, I was asking her if it would be good to do anything differently if there definitely was no God, and this time, she answered, "No."

"So," I said incredulously, "if God exists or doesn't exist, that has absolutely no effect on how it would be good for people to think or act? I think even a rabbi would look a little askance at that."

Her religion seems to now consist entirely of the worship of worship. As the true believers of older times might have believed that an all-seeing father would save them, she now believes that belief in God will save her.

After she said "I believe people are nicer than they are," I asked, "So, are you consistently surprised when people undershoot your expectations?" There was a long silence, and then, slowly: "Well... am I surprised when people... undershoot my expectations?"

I didn't understand this pause at the time. I'd intended it to suggest that if she was constantly disappointed by reality, then this was a downside of believing falsely. But she seemed, instead, to be taken aback at the implications of not being surprised.

I now realize that the whole essence of her philosophy was her belief that she had deceived herself, and the possibility that her estimates of other people were actually accurate, threatened the Dark Side Epistemology that she had built around beliefs such as "I benefit from believing people are nicer than they actually are."

She has taken the old idol off its throne, and replaced it with an explicit worship of the Dark Side Epistemology that was once invented to defend the idol; she worships her own attempt at self-deception. The attempt failed, but she is honestly unaware of this.

And so humanity's token guardians of sanity (motto: "pooping your deranged little party since Epicurus") must now fight the active worship of self-deception - the worship of the supposed benefits of faith, in place of God.

This actually explains a fact about myself that I didn't really understand earlier - the reason why I'm annoyed when people talk as if self-deception is easy, and why I write entire blog posts arguing that making a deliberate choice to believe the sky is green, is harder to get away with than people seem to think.

It's because - while you can't just choose to believe the sky is green - if you don't realize this fact, then you actually can fool yourself into believing that you've successfully deceived yourself.

And since you then sincerely expect to receive the benefits that you think come from self-deception, you get the same sort of placebo benefit that would actually come from a successful self-deception.

So by going around explaining how hard self-deception is, I'm actually taking direct aim at the placebo benefits that people get from believing that they've deceived themselves, and targeting the new sort of religion that worships only the worship of God.

Will this battle, I wonder, generate a new list of reasons why, not belief, but belief in belief, is itself a good thing? Why people derive great benefits from worshipping their worship? Will we have to do this over again with belief in belief in belief and worship of worship of worship? Or will intelligent theists finally just give up on that line of argument?

I wish I could believe that no one could possibly believe in belief in belief in belief, but the Zombie World argument in philosophy has gotten even more tangled than this and its proponents still haven't abandoned it.

I await the eager defenses of belief in belief in the comments, but I wonder if anyone would care to jump ahead of the game and defend belief in belief in belief? Might as well go ahead and get it over with.

Sequence: Against Doublethink

No, Really, I've Deceived Myself

Moore's Paradox

Referenced by: Moore's Paradox • Don't Believe You'll Self-Deceive • Striving to Accept • Incremental Progress and the Valley

Original with comments: Belief in Self-Deception

Posting now enabled

Eliezer Yudkowsky, 05 March 2009 03:56PM

Posting is now enabled with a minimum karma required of 20 - that is, you must have gotten at least 20 upvotes on your comments in order to publish a post. Or an adminstrator such as myself or Robin (by default you should bother me) can temporarily bless you with posting ability - in the long run this shouldn't happen much.

Original with comments: Posting now enabled

Slow down a little... maybe?

Eliezer Yudkowsky, 07 March 2009 01:34AM

I think that three posts a day over and above Yudkowsky and/or Hanson posts might be enough. Where anything that gets voted to 0 or below doesn't count, nor do quick links.

Say you differently, readers? I'm just trying to space things out so we don't get overloaded with everything, all at once... if it turns out that people just have more to say than this, sustainably in the long term, then we can raise the posting speed.

Original with comments: Slow down a little... maybe?

Moore's Paradox

Eliezer Yudkowsky, 08 March 2009 02:27AM

Followup to: Belief in Self-Deception

Moore's Paradox is the standard term for saying "It's raining outside but I don't believe that it is." HT to painquale on MetaFilter.

I think I understand Moore's Paradox a bit better now, after reading some of the comments on Less Wrong. Jimrandomh suggests:

Many people cannot distinguish between levels of indirection. To them, "I believe X" and "X" are the same thing, and therefore, reasons why it is beneficial to believe X are also reasons why X is true.

I don't think this is correct - relatively young children can understand the concept of having a false belief, which requires separate mental buckets for the map and the territory. But it points in the direction of a similar idea:

Many people may not consciously distinguish between believing something and endorsing it.

After all - "I believe in democracy" means, colloquially, that you endorse the concept of democracy, not that you believe democracy exists. The word "belief", then, has more than one meaning. We could be looking at a confused word that causes confused thinking (or maybe it just reflects pre-existing confusion).

So: in the original example, "I believe people are nicer than they are", she came up with some reasons why it would be good to believe people are nice - health benefits and such - and since she now had some warm affect on "believing people are nice", she introspected on this warm affect and concluded, "I believe people are nice". That is, she mistook the positive affect attached to the quoted belief, as signaling her belief in the proposition. At the same time, the world itself seemed like people weren't so nice. So she said, "I believe people are nicer than they are."

And that verges on being an honest mistake - sort of - since people are not taught explicitly how to know when they believe something. As in the parable of the dragon in the garage; the one who says "There is a dragon in my garage - but it's invisible", does not recognize his anticipation of seeing no dragon, as indicating that he possesses an (accurate) model with no dragon in it.

It's not as if people are trained to recognize when they believe something. It's not like they're ever taught in high school: "What it feels like to actually believe something - to have that statement in your belief pool - is that it just seems like the way the world is. You should recognize this feeling, which is actual (unquoted) belief, and distinguish it from having good feelings about a belief that you recognize as a belief (which means that it's in quote marks)."

This goes a long way toward making this real-life case of Moore's Paradox seem less alien, and providing another mechanism whereby people can be simultaneously right and wrong.

Likewise Kurige who wrote:

I believe that there is a God - and that He has instilled a sense of right and wrong in us by which we are able to evaluate the world around us. I also believe a sense of morality has been evolutionarily programmed into us - a sense of morality that is most likely a result of the formation of meta-political coalitions in Bonobo communities a very, very long time ago. These two beliefs are not contradictory, but the complexity lies in reconciling the two.

I suspect, Kurige, you have decided that you have reasons to endorse the quoted belief that God has instilled a sense of right and wrong in us. And also that you have reasons to endorse the verdict of science. They both seem like good communities to join, right? There are benefits to both sets of beliefs? You introspect and find that you feel good about both beliefs?

But you did not say:

"God instilled a sense of right and wrong in us, and also a sense of morality has been evolutionarily programmed into us. The two states of reality are not inconsistent, but the complexity lies in reconciling the two."

If you're reading this, Kurige, you should very quickly say the above out loud, so you can notice that it seems at least slightly harder to swallow - notice the subjective difference - before you go to the trouble of rerationalizing.

This is the subjective difference between having reasons to endorse two different beliefs, and your mental model of a single world, a single way-things-are.

Sequence: Against Doublethink

Belief in Self-Deception

Don't Believe You'll Self-Deceive

Referenced by: Don't Believe You'll Self-Deceive • Rationality is Systematized Winning

Original with comments: Moore's Paradox

Don't Believe You'll Self-Deceive

Eliezer Yudkowsky, 09 March 2009 08:03AM

Followup to: Moore's Paradox, Doublethink

I don't mean to seem like I'm picking on Kurige, but I think you have to expect a certain amount of questioning if you show up on Less Wrong and say:

One thing I've come to realize that helps to explain the disparity I feel when I talk with most other Christians is the fact that somewhere along the way my world-view took a major shift away from blind faith and landed somewhere in the vicinity of Orwellian double-think.

"If you know it's double-think...

...how can you still believe it?" I helplessly want to say.

Or:

I chose to believe in the existence of God - deliberately and consciously. This decision, however, has absolutely zero effect on the actual existence of God.

If you know your belief isn't correlated to reality, how can you still believe it?

Shouldn't the gut-level realization, "Oh, wait, the sky really isn't green" follow from the realization "My map that says 'the sky is green' has no reason to be correlated with the territory"?

Well... apparently not.

One part of this puzzle may be my explanation of Moore's Paradox ("It's raining, but I don't believe it is") - that people introspectively mistake positive affect attached to a quoted belief, for actual credulity.

But another part of it may just be that - contrary to the indignation I initially wanted to put forward - it's actually quite easy not to make the jump from "The map that reflects the territory would say 'X'" to actually believing "X". It takes some work to explain the ideas of minds as map-territory correspondence builders, and even then, it may take more work to get the implications on a gut level.

I realize now that when I wrote "You cannot make yourself believe the sky is green by an act of will", I wasn't just a dispassionate reporter of the existing facts. I was also trying to instill a self-fulfilling prophecy.

It may be wise to go around deliberately repeating "I can't get away with double-thinking! Deep down, I'll know it's not true! If I know my map has no reason to be correlated with the territory, that means I don't believe it!"

Because that way - if you're ever tempted to try - the thoughts "But I know this isn't really true!" and "I can't fool myself!" will always rise readily to mind; and that way, you will indeed be less likely to fool yourself successfully. You're more likely to get, on a gut level, that telling yourself X doesn't make X true: and therefore, really truly not-X.

If you keep telling yourself that you can't just deliberately choose to believe the sky is green - then you're less likely to succeed in fooling yourself on one level or another; either in the sense of really believing it, or of falling into Moore's Paradox, belief in belief, or belief in self-deception.

If you keep telling yourself that deep down you'll know -

If you keep telling yourself that you'd just look at your elaborately constructed false map, and just know that it was a false map without any expected correlation to the territory, and therefore, despite all its elaborate construction, you wouldn't be able to invest any credulity in it -

If you keep telling yourself that reflective consistency will take over and make you stop believing on the object level, once you come to the meta-level realization that the map is not reflecting -

Then when push comes to shove - you may, indeed, fail.

When it comes to deliberate self-deception, you must believe in your own inability!

Tell yourself the effort is doomed - and it will be!

Is that the power of positive thinking, or the power of negative thinking? Either way, it seems like a wise precaution.

Sequence: Against Doublethink

Moore's Paradox

Referenced by: Striving to Accept • Incremental Progress and the Valley

Original with comments: Don't Believe You'll Self-Deceive

Striving to Accept

Eliezer Yudkowsky, 09 March 2009 11:29PM

Reply to: The Mystery of the Haunted Rationalist
Followup to: Don't Believe You'll Self-Deceive

Should a rationalist ever find themselves trying hard to believe something?

You may be tempted to answer "No", because "trying to believe" sounds so stereotypical of Dark Side Epistemology. You may be tempted to reply, "Surely, if you have to try hard to believe something, it isn't worth believing."

But Yvain tells us that - even though he knows damn well, on one level, that spirits and other supernatural things are not to be found in the causal closure we name "reality" - and even though he'd bet $100 against $10,000 that an examination would find no spirits in a haunted house - he's pretty sure he's still scared of haunted houses.

Maybe it's okay for Yvain to try a little harder to accept that there are no ghosts, since he already knows that there are no ghosts?

In my very early childhood I was lucky enough to read a book from the children's section of a branch library, called "The Mystery of Something Hill" or something. In which one of the characters says, roughly: "There are two ways to believe in ghosts. One way is to fully believe in ghosts, to look for them and talk about them. But the other way is to half-believe - to make fun of the idea of ghosts, and talk scornfully of ghosts; but to break into a cold sweat when you hear a bump in the night, or be afraid to enter a graveyard."

I wish I remembered the book's name, or the exact quote, because this was one of those statements that sinks in during childhood and remains a part of you for the rest of the life. But all I remember was that the solution to the mystery had to do with hoofbeats echoing from a nearby road.

So whenever I found something that I knew I shouldn't believe, I also tried to avoid half-believing; and I soon noticed that this was the harder part of the problem. In my childhood, I cured myself of the fear of the dark by thinking: If I'm going to think magically anyway, then I'll pretend that all the black and shapeless, dark and shadowy things are my friends. Not quite the way I would do it nowadays, but it worked to counteract the half-belief, and in not much time I wasn't thinking about it at all.

Considerably later in my life, I realized that I was having a problem with half-believing in magical thinking - that I would sometimes try to avoid visualizing unpleasant things, from half-fear that they would happen. If, before walking through a door, I visualized that a maniac had chosen that exact moment to sneak into the room on the other side, and was armed and waiting with a knife - then I would be that little bit more scared, and look around more nervously, when entering the room.

So - being, at this point, a bit more sophisticated - I visualized a spread of probable worlds, in which - in some tiny fraction - a knife-wielding maniac had indeed chosen that moment to lurk behind the door; and I visualized the fact that my visualizing the knife-wielding maniac did not make him the tiniest bit more likely to be there - did not increase the total number of maniacs across the worlds. And that did cure me, and it was done; along with a good deal of other half-superstitions of the same pattern, like not thinking too loudly about other people in case they heard me.

Enforcing reflectivity - making ourselves accept what we already know - is, in general, an ongoing challenge for rationalists. I cite the example above because it's a very direct illustration of the genre: I actually went so far as to visualize the (non-)correlation of map to territory across possible worlds, in order to get my object-level map to realize that the maniac really really wasn't there.

It wouldn't be unusual for a rationalist to find themselves struggling to rid themselves of attachment to an unwanted belief. If we can get out of sync in that direction, why not the other direction? If it's okay to make ourselves try to disbelieve, why not make ourselves try to believe?

Well, because it really is one of the classic warning signs of Dark Side Epistemology that you have to struggle to make yourself believe something.

So let us then delimit, and draw sharp boundaries around the particular and rationalist version of striving for acceptance, as follows:

First, you should only find yourself doing this when you find yourself thinking, "Wait a minute, that really is actually true - why can't I get my mind to accept that?" Not Gloriously and Everlastingly True, mind you, but plain old mundanely true. This will be gameable in verbal arguments between people - "Wait, but I do believe it's really actually true!" - but if you're honestly trying, you should be able to tell the difference internally. If you can't find that feeling of frustration at your own inability to accept the obvious, then you should back up and ask whether or not it really is obvious, before trying to make your mind do anything. Can the fool say, "But I do think it's completely true and obvious" about random silly beliefs? Yes, they can. But as for you, just don't do that. This is to be understood as a technique for not shooting off your own foot, not as a way of proving anything to anyone.

Second, I call it "striving to accept", not "striving to believe", following billswift's suggestion. Why? Consider the difference between "I believe people are nicer than they are" and "I accept people are nicer than they are". You shouldn't be trying to raise desperate enthusiasm for a belief - if it doesn't seem like a plain old reality that you need to accept, then you're using the wrong technique.

Third and I think most importantly - you should always be striving to accept some particular argument that you feel isn't sinking in. Strive to accept "X implies Y", not just "Y". Strive to accept that there are no ghosts because spirits are only made of material neurons, or because the supernatural is incoherent. Strive to accept that there's no maniac behind the door because your thoughts don't change reality. Strive to accept that you won't win the lottery because you could make one distinct statement every second for a year with every one of them wrong, and not be so wrong as you would be by saying "I will win the lottery."

So there is my attempt to draw a line between the Dark Side and the Light Side versions of "trying to believe". Of course the Light Side also tends to be aimed a bit more heavily at accepting negative beliefs than positive beliefs, as is seen from the three examples. Trying to think of a positive belief-to-accept was difficult; the best I could come up with offhand was, "Strive to accept that your personal identity is preserved by diassembly and reassembly, because deep down, there just aren't any billiard balls down there." And even that, I suspect, is more a negative belief, that identity is not disrupted.

But to summarize - there should always be some particular argument, that has the feeling of being plain old actually true, and that you are only trying to accept, and are frustrated at your own trouble in accepting. Not a belief that you feel obligated to believe in more strongly and enthusiastically, apart from any particular argument.

Original with comments: Striving to Accept

Wanted: Python open source volunteers

Eliezer Yudkowsky, 11 March 2009 04:59AM

Less Wrong is a fork of the open Reddit codebase, written in Python. Less Wrong's code is online at Github (look in r2/r2 for the meat), the issues tracker is at code.google.com. See Contributing to Less Wrong for a gentle introduction to getting started.

According to Reddit's blog, we are the coolest use of reddit source code they've seen! But we've still got a long way to go before we're as cool as we want to be.

If anyone out there is fluent in Python and willing to donate a noticeable amount of time to a good cause, an extra hand or two might help us implement many Less Wrong features a lot sooner.

The Reddit codebase does not have unit tests or a whole lot of documentation. Contributors need to be able to wade through Reddit's code to grok it, and write unit tests for what they do (or better yet, write unit tests for existing code).

Items on the issues tracker marked "Contributions-Welcome" are those that look relatively easy to contribute. Items marked "Contributions-LeaveItToUs" are those that look big and complicated, or that Tricycle (the main developers) have strong opinions about how to design and implement. You could hack the big ones, but the developers might need to spend time talking to you - so please don't step up unless you're fluent in Python, have the necessary time, and are serious about it.

An example of a Welcome contribution would be having the registration page explain what is a valid username, or making sure that any HTML generated automatically in comments is also legal to enter directly (like <a href></a>).

An example of a LeaveItToUs contribution would be the future Tag and Sequence system - a next/prev tracking system for tags (so that you can navigate through via "next in self_deception" / "prev in self_deception" arrows); a system that lets authors create sequences which behave like tags but are owned by that author; and RSS feeds for tags and sequences that feed a specified number of posts per day to new users trying to catch up. An experienced volunteer Python dev with enough block hours free to run out and do this would be very welcome.

Items like making the site fluid-width or importing old posts from Overcoming Bias are probably best left to the current site designers.

Hopefully this gives you an idea of what kind of work needs doing. See the issues tracker for more.

Original with comments: Wanted: Python open source volunteers

Raising the Sanity Waterline

Eliezer Yudkowsky, 12 March 2009 04:28AM

To paraphrase the Black Belt Bayesian: Behind every exciting, dramatic failure, there is a more important story about a larger and less dramatic failure that made the first failure possible.

If every trace of religion was magically eliminated from the world tomorrow, then - however much improved the lives of many people would be - we would not even have come close to solving the larger failures of sanity that made religion possible in the first place.

We have good cause to spend some of our efforts on trying to eliminate religion directly, because it is a direct problem. But religion also serves the function of an asphyxiated canary in a coal mine - religion is a sign, a symptom, of larger problems that don't go away just because someone loses their religion.

Consider this thought experiment - what could you teach people that is not directly about religion, which is true and useful as a general method of rationality, which would cause them to lose their religions? In fact - imagine that we're going to go and survey all your students five years later, and see how many of them have lost their religions compared to a control group; if you make the slightest move at fighting religion directly, you will invalidate the experiment. You may not make a single mention of religion or any religious belief in your classroom, you may not even hint at it in any obvious way. All your examples must center about real-world cases that have nothing to do with religion.

If you can't fight religion directly, what do you teach that raises the general waterline of sanity to the point that religion goes underwater?

Here are some such topics I've already covered - not avoiding all mention of religion, but it could be done:

- Affective Death Spirals - plenty of non-supernaturalist examples.

- How to avoid cached thoughts and fake wisdom; the pressure of conformity.

- Evidence and Occam's Razor - the rules of probability.

- The Bottom Line / Engines of Cognition - the causal reasons why Reason works.

- Mysterious Answers to Mysterious Questions - and the whole associated sequence, like making beliefs pay rent and curiosity-stoppers - have excellent historical examples in vitalism and phlogiston.

- Non-existence of ontologically fundamental mental things - apply the Mind Projection Fallacy to probability, move on to reductionism versus holism, then brains and cognitive science.

- The many sub-arts of Crisis of Faith - though you'd better find something else to call this ultimate high master-level technique of actually updating on evidence.

- Dark Side Epistemology - teaching this with no mention of religion would be hard, but perhaps you could videotape the interrogation of some snake-oil sales agent as your real-world example.

- Fun Theory - teach as a literary theory of utopian fiction, without the direct application to theodicy.

- Joy in the Merely Real, naturalistic metaethics, etcetera etcetera etcetera and so on.

But to look at it another way -

Suppose we have a scientist who's still religious, either full-blown scriptural-religion, or in the sense of tossing around vague casual endorsements of "spirituality".

We now know this person is not applying any technical, explicit understanding of...

- ...what constitutes evidence and why;

- ...Occam's Razor;

- ...how the above two rules derive from the lawful and causal operation of minds as mapping engines, and do not switch off when you talk about tooth fairies;

- ...how to tell the difference between a real answer and a curiosity-stopper;

- ...how to rethink matters for themselves instead of just repeating things they heard;

- ...certain general trends of science over the last three thousand years;

- ...the difficult arts of actually updating on new evidence and relinquishing old beliefs;

- ...epistemology 101;

- ...self-honesty 201;

- ...etcetera etcetera etcetera and so on.

When you consider it - these are all rather basic matters of study, as such things go. A quick introduction to all of them (well, except naturalistic metaethics) would be... a four-credit undergraduate course with no prerequisites?

But there are Nobel laureates who haven't taken that course! Richard Smalley if you're looking for a cheap shot, or Robert Aumann if you're looking for a scary shot.

And they can't be isolated exceptions. If all of their professional compatriots had taken that course, then Smalley or Aumann would either have been corrected (as their colleagues kindly took them aside and explained the bare fundamentals) or else regarded with too much pity and concern to win a Nobel Prize. Could you - realistically speaking, regardless of fairness - win a Nobel while advocating the existence of Santa Claus?

That's what the dead canary, religion, is telling us: that the general sanity waterline is currently really ridiculously low. Even in the highest halls of science.

If we throw out that dead and rotting canary, then our mine may stink a bit less, but the sanity waterline may not rise much higher.

This is not to criticize the neo-atheist movement. The harm done by religion is clear and present danger, or rather, current and ongoing disaster. Fighting religion's directly harmful effects takes precedence over its use as a canary or experimental indicator. But even if Dawkins, and Dennett, and Harris, and Hitchens should somehow win utterly and absolutely to the last corner of the human sphere, the real work of rationalists will be only just beginning.

Sequence: The Craft and the Community

A Sense That More Is Possible

Referenced by: A Sense That More Is Possible • Rationality: Common Interest of Many Causes • The Craft and the Community • This Failing Earth • I'm Not Saying People Are Stupid

Original with comments: Raising the Sanity Waterline

A Sense That More Is Possible

Eliezer Yudkowsky, 13 March 2009 01:15AM

Previously in series: Raising the Sanity Waterline
Followup to: Teaching the Unteachable

To teach people about a topic you've labeled "rationality", it helps for them to be interested in "rationality". (There are less direct ways to teach people how to attain the map that reflects the territory, or optimize reality according to their values; but the explicit method is the course I tend to take.)

And when people explain why they're not interested in rationality, one of the most commonly proffered reasons tends to be like: "Oh, I've known a couple of rational people and they didn't seem any happier."

Who are they thinking of? Probably an Objectivist or some such. Maybe someone they know who's an ordinary scientist. Or an ordinary atheist.

That's really not a whole lot of rationality, as I have previously said.

Even if you limit yourself to people who can derive Bayes's Theorem - which is going to eliminate, what, 98% of the above personnel? - that's still not a whole lot of rationality. I mean, it's a pretty basic theorem.

Since the beginning I've had a sense that there ought to be some discipline of cognition, some art of thinking, the studying of which would make its students visibly more competent, more formidable: the equivalent of Taking a Level in Awesome.

But when I look around me in the real world, I don't see that. Sometimes I see a hint, an echo, of what I think should be possible, when I read the writings of folks like Robyn Dawes, Daniel Gilbert, Tooby & Cosmides. A few very rare and very senior researchers in psychological sciences, who visibly care a lot about rationality - to the point, I suspect, of making their colleagues feel uncomfortable, because it's not cool to care that much. I can see that they've found a rhythm, a unity that begins to pervade their arguments -

Yet even that... isn't really a whole lot of rationality either.

Even among those whose few who impress me with a hint of dawning formidability - I don't think that their mastery of rationality could compare to, say, John Conway's mastery of math. The base knowledge that we drew upon to build our understanding - if you extracted only the parts we used, and not everything we had to study to find it - it's probably not comparable to what a professional nuclear engineer knows about nuclear engineering. It may not even be comparable to what a construction engineer knows about bridges. We practice our skills, we do, in the ad-hoc ways we taught ourselves; but that practice probably doesn't compare to the training regimen an Olympic runner goes through, or maybe even an ordinary professional tennis player.

And the root of this problem, I do suspect, is that we haven't really gotten together and systematized our skills. We've had to create all of this for ourselves, ad-hoc, and there's a limit to how much one mind can do, even if it can manage to draw upon work done in outside fields.

The chief obstacle to doing this the way it really should be done, is the difficulty of testing the results of rationality training programs, so you can have evidence-based training methods. I will write more about this, because I think that recognizing successful training and distinguishing it from failure is the essential, blocking obstacle.

There are experiments done now and again on debiasing interventions for particular biases, but it tends to be something like, "Make the students practice this for an hour, then test them two weeks later." Not, "Run half the signups through version A of the three-month summer training program, and half through version B, and survey them five years later." You can see, here, the implied amount of effort that I think would go into a training program for people who were Really Serious about rationality, as opposed to the attitude of taking Casual Potshots That Require Like An Hour Of Effort Or Something.

Daniel Burfoot brilliantly suggests that this is why intelligence seems to be such a big factor in rationality - that when you're improvising everything ad-hoc with very little training or systematic practice, intelligence ends up being the most important factor in what's left.

Why aren't "rationalists" surrounded by a visible aura of formidability? Why aren't they found at the top level of every elite selected on any basis that has anything to do with thought? Why do most "rationalists" just seem like ordinary people, perhaps of moderately above-average intelligence, with one more hobbyhorse to ride?

Of this there are several answers; but one of them, surely, is that they have received less systematic training of rationality in a less systematic context than a first-dan black belt gets in hitting people.

I do not except myself from this criticism. I am no beisutsukai, because there are limits to how much Art you can create on your own, and how well you can guess without evidence-based statistics on the results. I know about a single use of rationality, which might be termed "reduction of confusing cognitions". This I asked of my brain, this it has given me. There are other arts, I think, that a mature rationality training program would not neglect to teach, which would make me stronger and happier and more effective - if I could just go through a standardized training program using the cream of teaching methods experimentally demonstrated to be effective. But the kind of tremendous, focused effort that I put into creating my single sub-art of rationality from scratch - my life doesn't have room for more than one of those.

I consider myself something more than a first-dan black belt, and less. I can punch through brick and I'm working on steel along my way to adamantine, but I have a mere casual street-fighter's grasp of how to kick or throw or block.

Why are there schools of martial arts, but not rationality dojos? (This was the first question I asked in my first blog post.) Is it more important to hit people than to think?

No, but it's easier to verify when you have hit someone. That's part of it, a highly central part.

But maybe even more importantly - there are people out there who want to hit, and who have the idea that there ought to be a systematic art of hitting that makes you into a visibly more formidable fighter, with a speed and grace and strength beyond the struggles of the unpracticed. So they go to a school that promises to teach that. And that school exists because, long ago, some people had the sense that more was possible. And they got together and shared their techniques and practiced and formalized and practiced and developed the Systematic Art of Hitting. They pushed themselves that far because they thought they should be awesome and they were willing to put some back into it.

Now - they got somewhere with that aspiration, unlike a thousand other aspirations of awesomeness that failed, because they could tell when they had hit someone; and the schools competed against each other regularly in realistic contests with clearly-defined winners.

But before even that - there was first the aspiration, the wish to become stronger, a sense that more was possible. A vision of a speed and grace and strength that they did not already possess, but could possess, if they were willing to put in a lot of work, that drove them to systematize and train and test.

Why don't we have an Art of Rationality?

Third, because current "rationalists" have trouble working in groups: of this I shall speak more.

Second, because it is hard to verify success in training, or which of two schools is the stronger.

But first, because people lack the sense that rationality is something that should be systematized and trained and tested like a martial art, that should have as much knowledge behind it as nuclear engineering, whose superstars should practice as hard as chess grandmasters, whose successful practitioners should be surrounded by an evident aura of awesome.

And conversely they don't look at the lack of visibly greater formidability, and say, "We must be doing something wrong."

"Rationality" just seems like one more hobby or hobbyhorse, that people talk about at parties; an adopted mode of conversational attire with few or no real consequences; and it doesn't seem like there's anything wrong about that, either.

Sequence: The Craft and the Community

Raising the Sanity Waterline

Epistemic Viciousness

Referenced by: Epistemic Viciousness • 3 Levels of Rationality Verification • Can Humanism Match Religion's Output? • Selecting Rationalist Groups • Rationality is Systematized Winning • Go Forth and Create the Art! • The Craft and the Community

Original with comments: A Sense That More Is Possible

Epistemic Viciousness

Eliezer Yudkowsky, 13 March 2009 11:33PM

Previously in series: A Sense That More Is Possible

Someone deserves a large hattip for this, but I'm having trouble remembering who; my records don't seem to show any email or OB comment which told me of this 12-page essay, "Epistemic Viciousness in the Martial Arts" by Gillian Russell. Maybe Anna Solomon?

We all lined up in our ties and sensible shoes (this was England) and copied him—left, right, left, right—and afterwards he told us that if we practised in the air with sufficient devotion for three years, then we would be able to use our punches to kill a bull with one blow.
I worshipped Mr Howard (though I would sooner have died than told him that) and so, as a skinny, eleven-year-old girl, I came to believe that if I practised, I would be able to kill a bull with one blow by the time I was fourteen.
This essay is about epistemic viciousness in the martial arts, and this story illustrates just that. Though the word ‘viciousness’ normally suggests deliberate cruelty and violence, I will be using it here with the more old-fashioned meaning, possessing of vices.

It all generalizes amazingly. To summarize some of the key observations for how epistemic viciousness arises:

- The art, the dojo, and the sensei are seen as sacred. "Having red toe-nails in the dojo is like going to church in a mini-skirt and halter-top... The students of other martial arts are talked about like they are practicing the wrong religion."

- If your teacher takes you aside and teaches you a special move and you practice it for 20 years, you have a large emotional investment in it, and you'll want to discard any incoming evidence against the move.

- Incoming students don't have much choice: a martial art can't be learned from a book, so they have to trust the teacher.

- Deference to famous historical masters. "Runners think that the contemporary staff of Runner's World know more about running than than all the ancient Greeks put together. And it's not just running, or other physical activities, where history is kept in its place; the same is true in any well-developed area of study. It is not considered disrespectful for a physicist to say that Isaac Newton's theories are false..." (Sound familiar?)

- "We martial artists struggle with a kind of poverty—data-poverty—which makes our beliefs hard to test... Unless you're unfortunate enough to be fighting a hand-to-hand war you cannot check to see how much force and exactly which angle a neck-break requires..."

- "If you can't test the effectiveness of a technique, then it is hard to test methods for improving the technique. Should you practice your nukite in the air, or will that just encourage you to overextend? ... Our inability to test our fighting methods restricts our ability to test our training methods."

- "But the real problem isn’t just that we live in data poverty—I think that’s true for some perfectly respectable disciplines, including theoretical physics—the problem is that we live in poverty but continue to act as though we live in luxury, as though we can safely afford to believe whatever we’re told..." (+10!)

One thing that I remembered being in this essay, but, on a second reading, wasn't actually there, was the degeneration of martial arts after the decline of real fights - by which I mean, fights where people were really trying to hurt each other and someone occasionally got killed.

In those days, you had some idea of who the real masters were, and which school could defeat others.

And then things got all civilized. And so things went downhill to the point that we have videos on Youtube of supposed Nth-dan black belts being pounded into the ground by someone with real fighting experience.

I had one case of this bookmarked somewhere (but now I can't find the bookmark) that was really sad; it was a master of a school who was convinced he could use ki techniques. His students would actually fall over when he used ki attacks, a strange and remarkable and frightening case of self-hypnosis or something... and the master goes up against a skeptic and of course gets pounded completely into the floor. Feel free to comment this link if you know where it is.

Truly is it said that "how to not lose" is more broadly applicable information than "how to win". Every single one of these risk factors transfers straight over to any attempt to start a "rationality dojo". I put to you the question: What can be done about it?

Sequence: The Craft and the Community

A Sense That More Is Possible

Schools Proliferating Without Evidence

Referenced by: Schools Proliferating Without Evidence • Mandatory Secret Identities • The Craft and the Community

Original with comments: Epistemic Viciousness

Schools Proliferating Without Evidence

Eliezer Yudkowsky, 15 March 2009 06:43AM

Previously in series: Epistemic Viciousness

Robyn Dawes, author of one of the original papers from Judgment Under Uncertainty and of the book Rational Choice in an Uncertain World - one of the few who tries really hard to import the results to real life - is also the author of House of Cards: Psychology and Psychotherapy Built on Myth.

From House of Cards, chapter 1:

The ability of these professionals has been subjected to empirical scrutiny - for example, their effectiveness as therapists (Chapter 2), their insight about people (Chapter 3), and the relationship between how well they function and the amount of experience they have had in their field (Chapter 4). Virtually all the research - and this book will reference more than three hundred empirical investigations and summaries of investigations - has found that these professionals' claims to superior intuitive insight, understanding, and skill as therapists are simply invalid...

Remember Rorschach ink-blot tests? It's such an appealing argument: the patient looks at the ink-blot and says what he sees, the psychotherapist interprets their psychological state based on this. There've been hundreds of experiments looking for some evidence that it actually works. Since you're reading this, you can guess the answer is simply "No." Yet the Rorschach is still in use. It's just such a good story that psychotherapists just can't bring themselves to believe the vast mounds of experimental evidence saying it doesn't work -

- which tells you what sort of field we're dealing with here.

And the experimental results on the field as a whole are commensurate. Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing. But there is no statistically discernible difference between the many schools of psychotherapy. There is no discernible gain from years of expertise.

And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It's just talking to anyone that helps you get better, apparently.

In the entire absence of the slightest experimental evidence for their effectiveness, psychotherapists became licensed by states, their testimony accepted in court, their teaching schools accredited, and their bills paid by health insurance.

And there was also a huge proliferation of "schools", of traditions of practice, in psychotherapy; despite - or perhaps because of - the lack of any experiments showing that one school was better than another...

I should really post more some other time on all the sad things this says about our world; about how the essence of medicine, as recognized by society and the courts, is not a repertoire of procedures with statistical evidence for their healing effectiveness; but, rather, the right air of authority.

But the subject today is the proliferation of traditions in psychotherapy. So far as I can discern, this was the way you picked up prestige in the field - not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own "school", supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.

This was probably, to no small extent, responsible for the existence and continuation of psychotherapy in the first place - the promise of making yourself a Master, like Freud who'd done it first (also without the slightest scrap of experimental evidence). That's the brass ring of success to chase - the prospect of being a guru and having your own adherents. It's the struggle for adherents that keeps the clergy vital.

That's what happens to a field when it unbinds itself from the experimental evidence - though there were other factors that also placed psychotherapists at risk, such as the deference shown them by their patients, the wish of society to believe that mental healing was possible, and, of course, the general dangers of telling people how to think.

The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness - that there was a family of measures that by golly did validate well against each other.

The act of creating a new measurement creates new science; if it's a good measurement, you get good science.

If you're going to create an organized practice of anything, you really do need some way of telling how well you're doing, and a practice of doing serious testing - that means a control group, an experimental group, and statistics - on plausible-sounding techniques that people come up with. You really need it.

Added: Dawes wrote in the 80s and I know that the Rorschach was still in use as recently as the 90s, but it's possible matters have improved since then (as one commenter states). I do remember hearing that there was positive evidence for the greater effectiveness of cognitive-behavioral therapy.

Sequence: The Craft and the Community

Epistemic Viciousness

3 Levels of Rationality Verification

Referenced by: 3 Levels of Rationality Verification • The Craft and the Community

Original with comments: Schools Proliferating Without Evidence

3 Levels of Rationality Verification

Eliezer Yudkowsky, 15 March 2009 05:19PM

Previously in series: Schools Proliferating Without Evidence
Followup to: A Sense That More Is Possible

I strongly suspect that there is a possible art of rationality (attaining the map that reflects the territory, choosing so as to direct reality into regions high in your preference ordering) which goes beyond the skills that are standard, and beyond what any single practitioner singly knows. I have a sense that more is possible.

The degree to which a group of people can do anything useful about this, will depend overwhelmingly on what methods we can devise to verify our many amazing good ideas.

I suggest stratifying verification methods into 3 levels of usefulness:

- Reputational

- Experimental

- Organizational

If your martial arts master occasionally fights realistic duels (ideally, real duels) against the masters of other schools, and wins or at least doesn't lose too often, then you know that the master's reputation is grounded in reality; you know that your master is not a complete poseur. The same would go if your school regularly competed against other schools. You'd be keepin' it real.

Some martial arts fail to compete realistically enough, and their students go down in seconds against real streetfighters. Other martial arts schools fail to compete at all - except based on charisma and good stories - and their masters decide they have chi powers. In this latter class we can also place the splintered schools of psychoanalysis.

So even just the basic step of trying to ground reputations in some realistic trial other than charisma and good stories, has tremendous positive effects on a whole field of endeavor.

But that doesn't yet get you a science. A science requires that you be able to test 100 applications of method A against 100 applications of method B and run statistics on the results. Experiments have to be replicable and replicated. This requires standard measurements that can be run on students who've been taught using randomly-assigned alternative methods, not just realistic duels fought between masters using all of their accumulated techniques and strength.

The field of happiness studies was created, more or less, by realizing that asking people "On a scale of 1 to 10, how good do you feel right now?" was a measure that statistically validated well against other ideas for measuring happiness. And this, despite all skepticism, looks like it's actually a pretty useful measure of some things, if you ask 100 people and average the results.

But suppose you wanted to put happier people in positions of power - pay happy people to train other people to be happier, or employ the happiest at a hedge fund? Then you're going to need some test that's harder to game than just asking someone "How happy are you?"

This question of verification methods good enough to build organizations, is a huge problem at all levels of modern human society. If you're going to use the SAT to control admissions to elite colleges, then can the SAT be defeated by studying just for the SAT in a way that ends up not correlating to other scholastic potential? If you give colleges the power to grant degrees, then do they have an incentive not to fail people? (I consider it drop-dead obvious that the task of verifying acquired skills and hence the power to grant degrees should be separated from the institutions that do the teaching, but let's not go into that.) If a hedge fund posts 20% returns, are they really that much better than the indices, or are they selling puts that will blow up in a down market?

If you have a verification method that can be gamed, the whole field adapts to game it, and loses its purpose. Colleges turn into tests of whether you can endure the classes. High schools do nothing but teach to statewide tests. Hedge funds sell puts to boost their returns.

On the other hand - we still manage to teach engineers, even though our organizational verification methods aren't perfect. So what perfect or imperfect methods could you use for verifying rationality skills, that would be at least a little resistant to gaming?

(Added: Measurements with high noise can still be used experimentally, if you randomly assign enough subjects to have an expectation of washing out the variance. But for the organizational purpose of verifying particular individuals, you need low-noise measurements.)

So I now put to you the question - how do you verify rationality skills? At any of the three levels? Brainstorm, I beg you; even a difficult and expensive measurement can become a gold standard to verify other metrics. Feel free to email me at sentience@pobox.com to suggest any measurements that are better off not being publicly known (though this is of course a major disadvantage of that method). Stupid ideas can suggest good ideas, so if you can't come up with a good idea, come up with a stupid one.

Reputational, experimental, organizational:

- Something the masters and schools can do to keep it real (realistically real);

- Something you can do to measure each of a hundred students;

- Something you could use as a test even if people have an incentive to game it.

Finding good solutions at each level determines what a whole field of study can be useful for - how much it can hope to accomplish. This is one of the Big Important Foundational Questions, so -

Think!

(PS: And ponder on your own before you look at the other comments; we need breadth of coverage here.)

Sequence: The Craft and the Community

Schools Proliferating Without Evidence

Why Our Kind Can't Cooperate

Referenced by: Why Our Kind Can't Cooperate • Selecting Rationalist Groups • Incremental Progress and the Valley • The Craft and the Community

Original with comments: 3 Levels of Rationality Verification

What Do We Mean By "Rationality"?

Eliezer Yudkowsky, 16 March 2009 10:33PM

We mean:

1. Epistemic rationality: believing, and updating on evidence, so as to systematically improve the correspondence between your map and the territory. The art of obtaining beliefs that correspond to reality as closely as possible. This correspondence is commonly termed "truth" or "accuracy", and we're happy to call it that.

2. Instrumental rationality: achieving your values. Not necessarily "your values" in the sense of being selfish values or unshared values: "your values" means anything you care about. The art of choosing actions that steer the future toward outcomes ranked higher in your preferences. On LW we sometimes refer to this as "winning".

If that seems like a perfectly good definition, you can stop reading here; otherwise continue.

Sometimes experimental psychologists uncover human reasoning that seems very strange - for example, someone rates the probability "Bill plays jazz" as less than the probability "Bill is an accountant who plays jazz". This seems like an odd judgment, since any particular jazz-playing accountant is obviously a jazz player. But to what higher vantage point do we appeal in saying that the judgment is wrong?

Experimental psychologists use two gold standards: probability theory, and decision theory. Since it is a universal law of probability theory that P(A) ≥ P(A & B), the judgment P("Bill plays jazz") < P("Bill plays jazz" & "Bill is accountant") is labeled incorrect.

To keep it technical, you would say that this probability judgment is non-Bayesian. Beliefs that conform to a coherent probability distribution, and decisions that maximize the probabilistic expectation of a coherent utility function, are called "Bayesian".

This does not quite exhaust the problem of what is meant in practice by "rationality", for two major reasons:

First, the Bayesian formalisms in their full form are computationally intractable on most real-world problems. No one can actually calculate and obey the math, any more than you can predict the stock market by calculating the movements of quarks.

This is why we have a whole site called "Less Wrong", rather than simply stating the formal axioms and being done. There's a whole further art to finding the truth and accomplishing value from inside a human mind: we have to learn our own flaws, overcome our biases, prevent ourselves from self-deceiving, get ourselves into good emotional shape to confront the truth and do what needs doing, etcetera etcetera and so on.

Second, sometimes the meaning of the math itself is called into question. The exact rules of probability theory are called into question by e.g. anthropic problems in which the number of observers is uncertain. The exact rules of decision theory are called into question by e.g. Newcomblike problems in which other agents may predict your decision before it happens.

In cases like these, it is futile to try to settle the problem by coming up with some new definition of the word "rational", and saying, "Therefore my preferred answer, by definition, is what is meant by the word 'rational'." This simply begs the question of why anyone should pay attention to your definition. We aren't interested in probability theory because it is the holy word handed down from Laplace. We're interested in Bayesian-style belief-updating (with Occam priors) because we expect that this style of thinking gets us systematically closer to, you know, accuracy, the map that reflects the territory. (More on the futility of arguing "by definition" here and here.)

And then there are questions of "How to think" that seem not quite answered by either probability theory or decision theory - like the question of how to feel about the truth once we have it. Here again, trying to define "rationality" a particular way doesn't support an answer, merely presume it.

From the Twelve Virtues of Rationality:

How can you improve your conception of rationality? Not by saying to yourself, “It is my duty to be rational.”  By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, “The sky is green,” and you look up at the sky and see blue. If you think: “It may look like the sky is blue, but rationality is to believe the words of the Great Teacher,” you lose a chance to discover your mistake.

Do not ask whether it is “the Way” to do this or that. Ask whether the sky is blue or green. If you speak overmuch of the Way you will not attain it.

You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory”. But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.

We are not here to argue the meaning of a word, not even if that word is "rationality". The point of attaching sequences of letters to particular concepts is to let two people communicate - to help transport thoughts from one mind to another. You cannot change reality, or prove the thought, by manipulating which meanings go with which words.

So if you understand what concept we are generally getting at with this word "rationality", and with the sub-terms "epistemic rationality" and "instrumental rationality", we have communicated: we have accomplished everything there is to accomplish by talking about how to define "rationality". What's left to discuss is not what meaning to attach to the syllables "ra-tio-na-li-ty"; what's left to discuss is what is a good way to think.

With that said, you should be aware that many of us will regard as controversial - at the very least - any construal of "rationality" that makes it non-normative:

For example, if you say, "The rational belief is X, but the true belief is Y" then you are probably using the word "rational" in a way that means something other than what most of us have in mind. (E.g. some of us expect "rationality" to be consistent under reflection - "rationally" looking at the evidence, and "rationally" considering how your mind processes the evidence, shouldn't lead to two different conclusions.) Similarly, if you find yourself saying "The rational thing to do is X, but the right thing to do is Y" then you are almost certainly using one of the words "rational" or "right" in a way that a huge chunk of readers won't agree with.

In this case - or in any other case where controversy threatens - you should substitute more specific language: "The self-benefiting thing to do is to run away, but I hope I would at least try to drag the girl off the railroad tracks" or "Causal decision theory as usually formulated says you should two-box on Newcomb's Problem, but I'd rather have a million dollars."

"X is rational!" is usually just a more strident way of saying "I think X is true" or "I think X is good". So why have an additional word for "rational" as well as "true" and "good"? Because we want to talk about systematic methods for obtaining truth and winning.

The word "rational" has potential pitfalls, but there are plenty of non-borderline cases where "rational" works fine to communicate what one is getting at, likewise "irrational". In these cases we're not afraid to use it.

Yet one should also be careful not to overuse that word. One receives no points merely for pronouncing it loudly. If you speak overmuch of the Way you will not attain it.

Sequence: Map and Territory

Why truth? And...

Referenced by: Comments for "Rationality" • Why Our Kind Can't Cooperate • Of Gender and Rationality • Of Exclusionary Speech and Gender Politics

Original with comments: What Do We Mean By "Rationality"?

Comments for "Rationality"

Eliezer Yudkowsky, 16 March 2009 10:34PM

I wrote an Admin page for "What do we mean by 'Rationality'?" since this has risen to the status of a FAQ. Comments can go here.

Original with comments: Comments for "Rationality"

Rationalist Fiction

Eliezer Yudkowsky, 19 March 2009 08:22AM

Followup to: Lawrence Watt-Evans's Fiction
Reply to: On Juvenile Fiction

MBlume asked us to remember what childhood stories might have influenced us toward rationality; and this was given such excellent answers as Norton Juster's The Phantom Tollbooth. So now I'd like to ask a related question, expanding the purview to all novels (adult or child, SF&F or literary): Where can we find explicitly rationalist fiction?

Now of course there are a great many characters who claim to be using logic. The whole genre of mystery stories with seemingly logical detectives, starting from Sherlock Holmes, would stand in witness of that.

But when you look at what Sherlock Holmes does - you can't go out and do it at home. Sherlock Holmes is not really operating by any sort of reproducible method. He is operating by magically finding the right clues and carrying out magically correct complicated chains of deduction. Maybe it's just me, but it seems to me that reading Sherlock Holmes does not inspire you to go and do likewise. Holmes is a mutant superhero. And even if you did try to imitate him, it would never work in real life.

Contrast to A. E. van Vogt's Null-A novels, starting with The World of Null-A. Now let it first be admitted that Van Vogt had a number of flaws as an author. With that said, it is probably a historical fact about my causal origins, that the Null-A books had an impact on my mind that I didn't even realize until years later. It's not the sort of book that I read over and over again, I read it and then put it down, but -

- but this is where I was first exposed to such concepts as "The map is not the territory" and "rose_1 is not rose_2".

Null-A stands for "Non-Aristotelian", and the premise of the ficton is that studying Korzybski's General Semantics makes you a superhero. Let's not really go into that part. But in the Null-A ficton:

1) The protagonist, Gilbert Gosseyn, is not a mutant. He has studied rationality techniques that have been systematized and are used by other members of his society, not just him.

2) Van Vogt tells us what (some of) these principles are, rather than leaving them mysteriously blank - we can't be Gilbert Gosseyn, but we can at least use some of this stuff.

3) Van Vogt conveys the experience, shows Gosseyn in action using the principles, rather than leaving them to triumphant explanation afterward. We are put into Gosseyn's shoes at the moment of his e.g. making a conscious distinction between two different things referred to by the same name.

This is a high standard to meet.

But Marc Stiegler's David's Sling (quoted in e.g. this post) meets this same standard: The Zetetics derive their abilities from training in a systematized tradition; we get to see the actual principles the Zetetics are using, and they're ones we could try to apply in real life; and we're put into their shoes at the moments of their use.

I mention this to show that it isn't only van Vogt who's ever done this.

However...

...those two examples actually exhaust my knowledge of the science fiction and fantasy literature, so far as I can remember.

It really is a very high standard we're setting here. To realistically show your characters using an interesting technique of rationality, you have to know an interesting technique of rationality. Van Vogt was inspired by Korzybski, who - I discovered when I looked this up, just now - actually invented the phrase "The map is not the territory". Marc Stiegler was inspired by, among other sources, Eric Drexler and Robin Hanson. (Stiegler has another novel called Earthweb about using prediction markets to defend the Earth from invading aliens, which was my introduction to the concept of prediction markets.)

If I relax the standard to focus mainly on item (3), fiction that transmits a powerful experience of using rationality, then I could add in Greg Egan's Distress, some of Lawrence Watt-Evans's strange little novels, the travails of Salvor Hardin in the first Foundation novel, and probably any number of others.

But what I'm really interested in is whether there's any full-blown Rationalist Fiction that I've missed - or maybe just haven't remembered. Failing that, I'm interested in stories that merely do a good job of conveying a rationalist experience. (Please specify which of these cases is true, if you make a recommendation.)

Referenced by: Whining-Based Communities

Original with comments: Rationalist Fiction

Why Our Kind Can't Cooperate

Eliezer Yudkowsky, 20 March 2009 08:37AM

Previously in series: Rationality Verification

From when I was still forced to attend, I remember our synagogue's annual fundraising appeal. It was a simple enough format, if I recall correctly. The rabbi and the treasurer talked about the shul's expenses and how vital this annual fundraise was, and then the synagogue's members called out their pledges from their seats.

Straightforward, yes?

Let me tell you about a different annual fundraising appeal. One that I ran, in fact; during the early years of a nonprofit organization that may not be named. One difference was that the appeal was conducted over the Internet. And another difference was that the audience was largely drawn from the atheist/libertarian/technophile/sf-fan/early-adopter/programmer/etc crowd. (To point in the rough direction of an empirical cluster in personspace. If you understood the phrase "empirical cluster in personspace" then you know who I'm talking about.)

I crafted the fundraising appeal with care. By my nature I'm too proud to ask other people for help; but I've gotten over around 60% of that reluctance over the years. The nonprofit needed money and was growing too slowly, so I put some force and poetry into that year's annual appeal. I sent it out to several mailing lists that covered most of our potential support base.

And almost immediately, people started posting to the mailing lists about why they weren't going to donate. Some of them raised basic questions about the nonprofit's philosophy and mission. Others talked about their brilliant ideas for all the other sources that the nonprofit could get funding from, instead of them. (They didn't volunteer to contact any of those sources themselves, they just had ideas for how we could do it.)

Now you might say, "Well, maybe your mission and philosophy did have basic problems - you wouldn't want to censor that discussion, would you?"

Hold on to that thought.

Because people were donating. We started getting donations right away, via Paypal. We even got congratulatory notes saying how the appeal had finally gotten them to start moving. A donation of $111.11 was accompanied by a message saying, "I decided to give **** a little bit more. One more hundred, one more ten, one more single, one more dime, and one more penny. All may not be for one, but this one is trying to be for all."

But none of those donors posted their agreement to the mailing list. Not one.

So far as any of those donors knew, they were alone. And when they tuned in the next day, they discovered not thanks, but arguments for why they shouldn't have donated. The criticisms, the justifications for not donating - only those were displayed proudly in the open.

As though the treasurer had finished his annual appeal, and everyone not making a pledge had proudly stood up to call out justifications for refusing; while those making pledges whispered them quietly, so that no one could hear.

I know someone with a rationalist cause who goes around plaintively asking, "How come the Raelian flying-saucer cult can get tens of thousands of members [probably around 40,000] interested in complete nonsense, but we can't even get a thousand people working on this?"

The obvious wrong way to finish this thought is to say, "Let's do what the Raelians do! Let's add some nonsense to this meme!" For the benefit of those not immediately stopped by their ethical inhibitions, I will observe that there may be a hundred failed flying-saucer cults for every one that becomes famous. And the Dark Side may require non-obvious skills, which you, yes you, do not have: Not everyone can be a Sith Lord. In particular, if you talk about your planned lies on the public Internet, you fail. I'm no master criminal, but even I can tell certain people are not cut out to be crooks.

So it's probably not a good idea to cultivate a sense of violated entitlement at the thought that some other group, who you think ought to be inferior to you, has more money and followers. That path leads to - pardon the expression - the Dark Side.

But it probably does make sense to start asking ourselves some pointed questions, if supposed "rationalists" can't manage to coordinate as well as a flying-saucer cult.

How do things work on the Dark Side?

The respected leader speaks, and there comes a chorus of pure agreement: if there are any who harbor inward doubts, they keep them to themselves. So all the individual members of the audience see this atmosphere of pure agreement, and they feel more confident in the ideas presented - even if they, personally, harbored inward doubts, why, everyone else seems to agree with it.

("Pluralistic ignorance" is the standard label for this.)

If anyone is still unpersuaded after that, they leave the group (or in some places, are executed) - and the remainder are more in agreement, and reinforce each other with less interference.

(I call that "evaporative cooling of groups".)

The ideas themselves, not just the leader, generate unbounded enthusiasm and praise. The halo effect is that perceptions of all positive qualities correlate - e.g. telling subjects about the benefits of a food preservative made them judge it as lower-risk, even though the quantities were logically uncorrelated. This can create a positive feedback effect that makes an idea seem better and better and better, especially if criticism is perceived as traitorous or sinful.

(Which I term the "affective death spiral".)

So these are all examples of strong Dark Side forces that can bind groups together.

And presumably we would not go so far as to dirty our hands with such...

Therefore, as a group, the Light Side will always be divided and weak. Atheists, libertarians, technophiles, nerds, science-fiction fans, scientists, or even non-fundamentalist religions, will never be capable of acting with the fanatic unity that animates radical Islam. Technological advantage can only go so far; your tools can be copied or stolen, and used against you. In the end the Light Side will always lose in any group conflict, and the future inevitably belongs to the Dark.

I think that one's reaction to this prospect says a lot about their attitude towards "rationality".

Some "Clash of Civilizations" writers seem to accept that the Enlightenment is destined to lose out in the long run to radical Islam, and sigh, and shake their heads sadly. I suppose they're trying to signal their cynical sophistication or something.

For myself, I always thought - call me loony - that a true rationalist ought to be effective in the real world.

So I have a problem with the idea that the Dark Side, thanks to their pluralistic ignorance and affective death spirals, will always win because they are better coordinated than us.

You would think, perhaps, that real rationalists ought to be more coordinated? Surely all that unreason must have its disadvantages? That mode can't be optimal, can it?

And if current "rationalist" groups cannot coordinate - if they can't support group projects so well as a single synagogue draws donations from its members - well, I leave it to you to finish that syllogism.

There's a saying I sometimes use: "It is dangerous to be half a rationalist."

For example, I can think of ways to sabotage someone's intelligence by selectively teaching them certain methods of rationality. Suppose you taught someone a long list of logical fallacies and cognitive biases, and trained them to spot those fallacies in biases in other people's arguments. But you are careful to pick those fallacies and biases that are easiest to accuse others of, the most general ones that can easily be misapplied. And you do not warn them to scrutinize arguments they agree with just as hard as they scrutinize incongruent arguments for flaws. So they have acquired a great repertoire of flaws of which to accuse only arguments and arguers who they don't like. This, I suspect, is one of the primary ways that smart people end up stupid. (And note, by the way, that I have just given you another Fully General Counterargument against smart people whose arguments you don't like.)

Similarly, if you wanted to ensure that a group of "rationalists" never accomplished any task requiring more than one person, you could teach them only techniques of individual rationality, without mentioning anything about techniques of coordinated group rationality.

I'll write more later (tomorrow?) on how I think rationalists might be able to coordinate better. But today I want to focus on what you might call the culture of disagreement, or even, the culture of objections, which is one of the two major forces preventing the atheist/libertarian/technophile crowd from coordinating.

Imagine that you're at a conference, and the speaker gives a 30-minute talk. Afterward, people line up at the microphones for questions. The first questioner objects to the graph used in slide 14 using a logarithmic scale; he quotes Tufte on The Visual Display of Quantitative Information. The second questioner disputes a claim made in slide 3. The third questioner suggests an alternative hypothesis that seems to explain the same data...

Perfectly normal, right? Now imagine that you're at a conference, and the speaker gives a 30-minute talk. People line up at the microphone.

The first person says, "I agree with everything you said in your talk, and I think you're brilliant." Then steps aside.

The second person says, "Slide 14 was beautiful, I learned a lot from it. You're awesome." Steps aside.

The third person -

Well, you'll never know what the third person at the microphone had to say, because by this time, you've fled screaming out of the room, propelled by a bone-deep terror as if Cthulhu had erupted from the podium, the fear of the impossibly unnatural phenomenon that has invaded your conference.

Yes, a group which can't tolerate disagreement is not rational. But if you tolerate only disagreement - if you tolerate disagreement but not agreement - then you also are not rational. You're only willing to hear some honest thoughts, but not others. You are a dangerous half-a-rationalist.

We are as uncomfortable together as flying-saucer cult members are uncomfortable apart. That can't be right either. Reversed stupidity is not intelligence.

Let's say we have two groups of soldiers. In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy. In group 2, everyone at all levels knows all about tactics and strategy.

Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.

Doing worse with more knowledge means you are doing something very wrong. You should always be able to at least implement the same strategy you would use if you are ignorant, and preferably do better. You definitely should not do worse. If you find yourself regretting your "rationality" then you should reconsider what is rational.

On the other hand, if you are only half-a-rationalist, you can easily do worse with more knowledge. I recall a lovely experiment which showed that politically opinionated students with more knowledge of the issues reacted less to incongruent evidence, because they had more ammunition with which to counter-argue only incongruent evidence.

We would seem to be stuck in an awful valley of partial rationality where we end up more poorly coordinated than religious fundamentalists, able to put forth less effort than flying-saucer cultists. True, what little effort we do manage to put forth may be better-targeted at helping people rather than the reverse - but that is not an acceptable excuse.

If I were setting forth to systematically train rationalists, there would be lessons on how to disagree and lessons on how to agree, lessons intended to make the trainee more comfortable with dissent, and lessons intended to make them more comfortable with conformity. One day everyone shows up dressed differently, another day they all show up in uniform. You've got to cover both sides, or you're only half a rationalist.

Can you imagine training prospective rationalists to wear a uniform and march in lockstep, and practice sessions where they agree with each other and applaud everything a speaker on a podium says? It sounds like unspeakable horror, doesn't it, like the whole thing has admitted outright to being an evil cult? But why is it not okay to practice that, while it is okay to practice disagreeing with everyone else in the crowd? Are you never going to have to agree with the majority?

Our culture puts all the emphasis on heroic disagreement and heroic defiance, and none on heroic agreement or heroic group consensus. We signal our superior intelligence and our membership in the nonconformist community by inventing clever objections to others' arguments. Perhaps that is why the atheist/libertarian/technophile/sf-fan/Silicon-Valley/programmer/early-adopter crowd stays marginalized, losing battles with less nonconformist factions in larger society. No, we're not losing because we're so superior, we're losing because our exclusively individualist traditions sabotage our ability to cooperate.

The other major component that I think sabotages group efforts in the atheist/libertarian/technophile/etcetera community, is being ashamed of strong feelings. We still have the Spock archetype of rationality stuck in our heads, rationality as dispassion. Or perhaps a related mistake, rationality as cynicism - trying to signal your superior world-weary sophistication by showing that you care less than others. Being careful to ostentatiously, publicly look down on those so naive as to show they care strongly about anything.

Wouldn't it make you feel uncomfortable if the speaker at the podium said that he cared so strongly about, say, fighting aging, that he would willingly die for the cause?

But it is nowhere written in either probability theory or decision theory that a rationalist should not care. I've looked over those equations and, really, it's not in there.

The best informal definition I've ever heard of rationality is "That which can be destroyed by the truth should be." We should aspire to feel the emotions that fit the facts, not aspire to feel no emotion. If an emotion can be destroyed by truth, we should relinquish it. But if a cause is worth striving for, then let us by all means feel fully its importance.

Some things are worth dying for. Yes, really! And if we can't get comfortable with admitting it and hearing others say it, then we're going to have trouble caring enough - as well as coordinating enough - to put some effort into group projects. You've got to teach both sides of it, "That which can be destroyed by the truth should be," and "That which the truth nourishes should thrive."

I've heard it argued that the taboo against emotional language in, say, science papers, is an important part of letting the facts fight it out without distraction. That doesn't mean the taboo should apply everywhere. I think that there are parts of life where we should learn to applaud strong emotional language, eloquence, and poetry. When there's something that needs doing, poetic appeals help get it done, and, therefore, are themselves to be applauded.

We need to keep our efforts to expose counterproductive causes and unjustified appeals, from stomping on tasks that genuinely need doing. You need both sides of it - the willingness to turn away from counterproductive causes, and the willingness to praise productive ones; the strength to be unswayed by ungrounded appeals, and the strength to be swayed by grounded ones.

I think the synagogue at their annual appeal had it right, really. They weren't going down row by row and putting individuals on the spot, staring at them and saying, "How much will you donate, Mr. Schwartz?" People simply announced their pledges - not with grand drama and pride, just simple announcements - and that encouraged others to do the same. Those who had nothing to give, stayed silent; those who had objections, chose some later or earlier time to voice them. That's probably about the way things should be in a sane human community - taking into account that people often have trouble getting as motivated as they wish they were, and can be helped by social encouragement to overcome this weakness of will.

But even if you disagree with that part, then let us say that both supporting and countersupporting opinions should have been publicly voiced. Supporters being faced by an apparently solid wall of objections and disagreements - even if it resulted from their own uncomfortable self-censorship - is not group rationality. It is the mere mirror image of what Dark Side groups do to keep their followers. Reversed stupidity is not intelligence.

Sequence: The Craft and the Community

3 Levels of Rationality Verification

Tolerate Tolerance

Referenced by: Tolerate Tolerance • You're Calling *Who* A Cult Leader? • Your Price for Joining • Can Humanism Match Religion's Output? • Church vs. Taskforce • Bystander Apathy • Bayesians vs. Barbarians • Of Gender and Rationality • The Craft and the Community • The End (of Sequences) • The Correct Contrarian Cluster

Original with comments: Why Our Kind Can't Cooperate

Tolerate Tolerance

Eliezer Yudkowsky, 21 March 2009 07:34AM

Followup to: Why Our Kind Can't Cooperate

One of the likely characteristics of someone who sets out to be a "rationalist" is a lower-than-usual tolerance for flaws in reasoning. This doesn't strictly follow. You could end up, say, rejecting your religion, just because you spotted more or deeper flaws in the reasoning, not because you were, by your nature, more annoyed at a flaw of fixed size. But realistically speaking, a lot of us probably have our level of "annoyance at all these flaws we're spotting" set a bit higher than average.

That's why it's so important for us to tolerate others' tolerance if we want to get anything done together.

For me, the poster case of tolerance I need to tolerate is Ben Goertzel, who among other things runs an annual AI conference, and who has something nice to say about everyone. Ben even complimented the ideas of M*nt*f*x, the most legendary of all AI crackpots. (M*nt*f*x apparently started adding a link to Ben's compliment in his email signatures, presumably because it was the only compliment he'd ever gotten from a bona fide AI academic.) (Please do not pronounce his True Name correctly or he will be summoned here.)

But I've come to understand that this is one of Ben's strengths - that he's nice to lots of people that others might ignore, including, say, me - and every now and then this pays off for him.

And if I subtract points off Ben's reputation for finding something nice to say about people and projects that I think are hopeless - even M*nt*f*x - then what I'm doing is insisting that Ben dislike everyone I dislike before I can work with him.

Is that a realistic standard? Especially if different people are annoyed in different amounts by different things?

But it's hard to remember that when Ben is being nice to so many idiots.

Cooperation is unstable, in both game theory and evolutionary biology, without some kind of punishment for defection. So it's one thing to subtract points off someone's reputation for mistakes they make themselves, directly. But if you also look askance at someone for refusing to castigate a person or idea, then that is punishment of non-punishers, a far more dangerous idiom that can lock an equilibrium in place even if it's harmful to everyone involved.

The danger of punishing nonpunishers is something I remind myself of, say, every time Robin Hanson points out a flaw in some academic trope and yet modestly confesses he could be wrong (and he's not wrong). Or every time I see Michael Vassar still considering the potential of someone who I wrote off as hopeless within 30 seconds of being introduced to them. I have to remind myself, "Tolerate tolerance! Don't demand that your allies be equally extreme in their negative judgments of everything you dislike!"

By my nature, I do get annoyed when someone else seems to be giving too much credit. I don't know if everyone's like that, but I suspect that at least some of my fellow aspiring rationalists are. I wouldn't be surprised to find it a human universal; it does have an obvious evolutionary rationale - one which would make it a very unpleasant and dangerous adaptation.

I am not generally a fan of "tolerance". I certainly don't believe in being "intolerant of intolerance", as some inconsistently hold. But I shall go on trying to tolerate people who are more tolerant than I am, and judge them only for their own un-borrowed mistakes.

Oh, and it goes without saying that if the people of Group X are staring at you demandingly, waiting for you to hate the right enemies with the right intensity, and ready to castigate you if you fail to castigate loudly enough, you may be hanging around the wrong group.

Just don't demand that everyone you work with be equally intolerant of behavior like that. Forgive your friends if some of them suggest that maybe Group X wasn't so awful after all...

Sequence: The Craft and the Community

Why Our Kind Can't Cooperate

You're Calling *Who* A Cult Leader?

Referenced by: The Craft and the Community • Why I'm Staying On Bloggingheads.tv

Original with comments: Tolerate Tolerance

You're Calling *Who* A Cult Leader?

Eliezer Yudkowsky, 22 March 2009 06:57AM

Followup to: Why Our Kind Can't Cooperate, Cultish Countercultishness

I used to be a lot more worried that I was a cult leader before I started reading Hacker News. (WARNING: Do not click that link if you do not want another addictive Internet habit.)

From time to time, on a mailing list or IRC channel or blog which I ran, someone would start talking about "cults" and "echo chambers" and "coteries". And it was a scary accusation, because no matter what kind of epistemic hygeine I try to practice myself, I can't look into other people's minds. I don't know if my long-time readers are agreeing with me because I'm making sense, or because I've developed creepy mind-control powers. My readers are drawn from the nonconformist crowd - the atheist/libertarian/technophile/sf-reader/Silicon-Valley/early-adopter cluster - and so they certainly wouldn't admit to worshipping me even if they were.

And then I ran into Hacker News, where accusations in exactly the same tone were aimed at the site owner, Paul Graham.

Hold on. Paul Graham gets the same flak I do?

- Paul Graham has written a word or two about rationality... in a much more matter-of-fact style.

- Paul Graham does not ask his readers for donations. He is independently wealthy.

- Paul Graham is not dabbling in mad-science-grade AI. He runs Y Combinator, a seed-stage venture fund.

- Paul Graham is not trying to save the world. He's trying to help a new generation of entrepreneurs.

I've never heard of Paul Graham saying or doing a single thing that smacks of cultishness. Not one.

He just wrote some great essays (that appeal especially to the nonconformist crowd), and started an online forum where some people who liked those essays hang out (among others who just wandered into that corner of the Internet).

So when I read someone:

1. Comparing the long hours worked by Y Combinator startup founders to the sleep-deprivation tactic used in cults;

2. Claiming that founders were asked to move to the Bay Area startup hub as a cult tactic of separation from friends and family;

...well, that outright broke my suspension of disbelief.

Something is going on here which has more to do with the behavior of nonconformists in packs than whether or not you can make a plausible case for cultishness or even cultishness risk factors.

But there are aspects of this phenomenon that I don't understand, because I'm not feeling what they're feeling.

Behold the following, which is my true opinion:

"Gödel, Escher, Bach" by Douglas R. Hofstadter is the most awesome book that I have ever read. If there is one book that emphasizes the tragedy of Death, it is this book, because it's terrible that so many people have died without reading it.

I know people who would never say anything like that, or even think it: admiring anything that much would mean they'd joined a cult (note: Hofstadter does not have a cult). And I'm pretty sure that this negative reaction to strong admiration is what's going on with Paul Graham and his essays, and I begin to suspect that not a single thing more is going on with me.

But I'm having trouble understanding this phenomenon, because I myself feel no barrier against admiring Gödel, Escher, Bach that highly.

In fact, I would say that by far the most cultish-looking behavior on Hacker News is people trying to show off how willing they are to disagree with Paul Graham. Let me try to explain how this feels when you're the target of it:

It's like going to a library, and when you walk in the doors, everyone looks at you, staring. Then you walk over to a certain row of bookcases - say, you're looking for books on writing - and at once several others, walking with stiff, exaggerated movements, select a different stack to read in. When you reach the bookshelves for Dewey decimal 808, there are several other people present, taking quick glances out of the corner of their eye while pretending not to look at you. You take out a copy of The Poem's Heartbeat: A Manual of Prosody.

At once one of the others present reaches toward a different bookcase and proclaims, "I'm not reading The Poem's Heartbeat! In fact, I'm not reading anything about poetry! I'm reading The Elements of Style, which is much more widely recommended by many mainstream writers." Another steps in your direction and nonchalantly takes out a second copy of The Poem's Heartbeat, saying, "I'm not reading this book just because you're reading it, you know; I think it's a genuinely good book, myself."

Meanwhile, a teenager who just happens to be there, glances over at the book. "Oh, poetry," he says.

"Not exactly," you say. "I just thought that if I knew more about how words sound - the rhythm - it might make me a better writer."

"Oh!" he says, "You're a writer?"

You pause, trying to calculate whether the term does you too much credit, and finally say, "Well, I have a lot of readers, so I must be a writer."

"I plan on being a writer," he says. "Got any tips?"

"Start writing now," you say immediately. "I once read that every writer has a million words of bad writing inside them, and you have to get it out before you can write anything good. Yes, one million. The sooner you start, the sooner you finish."

The teenager nods, looking very serious. "Any of these books," gesturing around, "that you'd recommend?"

"If you're interested in fiction, then definitely Jack Bickham's Scene and Structure," you say, "though I'm still struggling with the form myself. I need to get better at description."

"Thanks," he says, and takes a copy of Scene and Structure.

"Hold on!" says the holder of The Elements of Style in a tone of shock. "You're going to read that book just because he told you to?"

The teenager furrows his brow. "Well, sure."

There's an audible gasp, coming not just from the local stacks but from several other stacks nearby.

"Well," says the one who took the other copy of The Poem's Heartbeat, "of course you mean that you're taking into account his advice about which books to read, but really, you're perfectly capable of deciding for yourself which books to read, and would never allow yourself to be swayed by arguments without adequate support. Why, I bet you can think of several book recommendations that you've rejected, thus showing your independence. Certainly, you would never go so far as to lose yourself in following someone else's book recommendations -"

"What?" says the teenager.

If there's an aspect of the whole thing that annoys me, it's that it's hard to get that innocence back, once you even start thinking about whether you're independent of someone. I recently downvoted one of PG's comments on HN (for the first time - a respondent had pointed out that the comment was wrong, and it was). And I couldn't help thinking, "Gosh, I'm downvoting one of PG's comments" - no matter how silly that is in context - because the cached thought had been planted in my mind from reading other people arguing over whether or not HN was a "cult" and defending their own freedom to disagree with PG.

You know, there might be some other things that I admire highly besides Gödel, Escher, Bach, and I might or might not disagree with some things Douglas Hofstadter once said, but I'm not even going to list them, because GEB doesn't need that kind of moderation. It is okay for GEB to be awesome. In this world there are people who have created awesome things and it is okay to admire them highly! Let this Earth have at least a little of its pride!

I've been flipping through ideas that might explain the anti-admiration phenomenon. One of my first thoughts was that I evaluate my own potential so highly (rightly or wrongly is not relevant here) that praising Gödel, Escher, Bach to the stars doesn't feel like making myself inferior to Douglas Hofstadter. But upon reflection, I strongly suspect that I would feel no barrier to praising GEB even if I weren't doing anything much interesting with my life. There's some fear I don't feel, or some norm I haven't acquired.

So rather than guess any further, I'm going to turn this over to my readers. I'm hoping in particular that someone used to feel this way - shutting down an impulse to praise someone else highly, or feeling that it was cultish to praise someone else highly - and then had some kind of epiphany after which it felt, not allowed, but rather, quite normal.

Sequence: The Craft and the Community

Tolerate Tolerance

On Things that are Awesome

Referenced by: On Things that are Awesome • The Craft and the Community

Original with comments: You're Calling *Who* A Cult Leader?

BHTV: Yudkowsky & Adam Frank on "religious experience"

Eliezer Yudkowsky, 23 March 2009 01:33AM

BHTV episode with myself and Adam Frank, author of "The Constant Fire", on whether or not religious experience is compatible with the scientific experience, or worth trying to salvage.

Referenced by: The Sacred Mundane • Why I'm Staying On Bloggingheads.tv

Original with comments: BHTV: Yudkowsky & Adam Frank on "religious experience"

On Things that are Awesome

Eliezer Yudkowsky, 24 March 2009 03:24AM

This post, which touched on the allowedness of admiration, started me thinking about the nature of things that are awesome.

The first thing one does in such a situation is generate examples. And my brain, asked to enumerate things that are awesome, said: "Douglas Hofstadter, E. T. Jaynes, Greg Egan..."

Upon that initial output of my brain, I had many other thoughts:

(1) My brain was able to list more than one thing that is awesome. I am not going to dwell on this, because I think it needless to go around saying, "Douglas Hofstadter is awesome, but E. T. Jaynes is awesome too," as though to deliberately moderate or subtract from the admiration of Hofstadter. The enjoyment of things that are awesome is an important part of life, and I don't think a healthy mind should have to hold back. But the more things you know that are awesome, the more there is to enjoy - this doesn't mean you should artificially inflate your estimations of awesomeness, but it does mean that if you can think of only one awesome thing, you must be missing out on a lot of life. And some awesome things, but not all, are compatible enough with yourself that you can draw upon the awesome - Hofstadter and Jaynes are both like this for me, but Greg Egan is not. So even leaving aside certain mental health risks from having only one awesome thing - it is both enjoyable, and strengthening, to know of many things that are awesome.

(2) I can think of many places where I disagree with statements emitted by Douglas Hofstadter and Greg Egan, and even one or two places where I would want to pencil in a correction to Jaynes (his interpretation of quantum mechanics being the most obvious). In fact, when my brain says "Greg Egan" it is really referring to two novels, Permutation City and Quarantine, which overshadow all his other works in my book. And when my brain says "Hofstadter" it is referring to Gödel, Escher, Bach with a small side order of some essays in Metamagical Themas. For most people their truly awesome work is usually only a slice of their total output, from some particular years (I find that scary as hell, by the way).

(3) Once you realize that you're only admiring someone's peak work, you also realize that the work is not the person: I don't actually know Hofstadter, or Greg Egan, or E. T. Jaynes. I have no idea what they are (were) like in their personal lives, or whether their daily deeds had any trace of the awesome that is in their books. If you start thinking that a person is supposed to be as universally and consistently awesome as their best work, so that every word from their lips is supposed to be as good as the best book they ever wrote, that's probably some kind of failure mode. This is not to try to moderate or diminish the awesomeness: for their best work is that awesome, and so there must have been a moment of their life, a time-slice out of their worldline, which was also that awesome. But what the symbol "Douglas Hofstadter" stands for, in my mind, is not all his works, or all his life.

(4) This made me realize a strange thing: Whenever someone compliments "Eliezer Yudkowsky", they are really complimenting "Eliezer Yudkowsky's writing" or "Eliezer Yudkowsky's best writing that stands out most in my mind". People who met me in person were often shocked at how much my in-person impression departed from the picture they had in their minds. I think this mostly had to do with imagining me as being the sort of actor who would be chosen to play me in the movie version of my life - they imagined way too much dignity. That forms a large part of the reason why I occasionally toss in the deliberate anime reference, which does seem to have fixed the divergence a bit. And these days I have videos of myself online. But then the inside of my head is something different again. It's an odd thought to realize that everyone else who uses the symbol 'Eliezer Yudkowsky' uses it to refer to a quite different thing than you do.

(5) What chiefly conveys to me the experience of the awesome is to see someone - pardon me, see someone's work - that is way above me. My most recent experience of the awesome was reading the third book in Jacqueline Carey's Kushiel series, and realizing that although I want to write with that kind of emotional depth, I can't, and may never be able to in this world. I looked back at all my own tries in (unpublished) fiction, and it paled to grey by comparison. It was the same way with reading Hofstadter the first time, and thinking that I could never, ever write as well as Gödel, Escher, Bach; or reading Permutation City, and seeing how far above me Greg Egan was as an idea-based science fiction writer. And it would have been the same way with Jaynes, if that time I hadn't been thinking to myself, "No, I must become this good." This is also something of a reply to Carl's comment that we may feel freer to admire those who do not compete with us - for me, the experience of the awesome is most strongly created by seeing someone (or rather their work) outdoing me overwhelmingly, in some place where I have tried my hand. I don't think there's anything unhealthy about making this a basis of admiration.

(6) My brain did not immediately enumerate all sorts of things that are too much a part of my background world to be salient: Science, space travel, the human brain, and the universe are all awesome. But the latter two are not human works, and you can't draw power from them the same way you can from a human work that is awesome and at least partly imitable. And the virtue of narrowness seems to play an important part here: an awesome thing that can be viewed in one small chunk and understood in detail will seem more awesome than something big and diffusely awesome. I would probably admire the space shuttle far more if I knew about it in more detail!

(7) One of the reasons why I object to Adam Frank's attempt to salvage the concept of "sacredness" from religion, instead of reinventing it from scratch, is that e.g. being contaminated by religious experience makes you more likely to think that sacredness should only be about stars or something - those works that were once thought to be of God - whereas there is often a lot more awesomeness stored up in a human work that you know is human. If I want to canonize something as sacred, I'll take Gödel, Escher, Bach over a mountain any day.

Sequence: The Craft and the Community

You're Calling *Who* A Cult Leader?

Your Price for Joining

Referenced by: Can Humanism Match Religion's Output? • The Craft and the Community

Original with comments: On Things that are Awesome

Hyakujo's Fox

Eliezer Yudkowsky, 24 March 2009 10:14AM

From "Hyakujo's Fox", #2 of the 49 koans in The Gateless Gate:

Once when Hyakujo delivered some Zen lectures an old man attended them, unseen by the monks. At the end of each talk when the monks left so did he. But one day he remained after the had gone, and Hyakujo asked him: `Who are you?'

The old man replied: `I am not a human being, but I was a human being when the Kashapa Buddha preached in this world. I was a Zen master and lived on this mountain. At that time one of my students asked me whether the enlightened man is subject to the law of causation. I answered him: "The enlightened man is not subject to the law of causation." For this answer evidencing a clinging to absoluteness I became a fox for five hundred rebirths, and I am still a fox. Will you save me from this condition with your Zen words and let me get out of a fox's body? Now may I ask you: Is the enlightened man subject to the law of causation?'

Hyakujo said: `The enlightened man is one with the law of causation.'

At the words of Hyakujo the old man was enlightened.

Mumon's poem:

Controlled or not controlled?
The same dice shows two faces.
Not controlled or controlled,
Both are a grievous error.

It really makes you wonder how the hell they got that far while still believing that the wrong answer could turn you into a fox.

Original with comments: Hyakujo's Fox

The Sacred Mundane

Eliezer Yudkowsky, 25 March 2009 09:53AM

Followup to: Is Humanism a Religion-Substitute?

So I was reading (around the first half of) Adam Frank's The Constant Fire, in preparation for my Bloggingheads dialogue with him. Adam Frank's book is about the experience of the sacred. I might not usually call it that, but of course I know the experience Frank is talking about. It's what I feel when I watch a video of a space shuttle launch; or what I feel - to a lesser extent, because in this world it is too common - when I look up at the stars at night, and think about what they mean. Or the birth of a child, say. That which is significant in the Unfolding Story.

Adam Frank holds that this experience is something that science holds deeply in common with religion. As opposed to e.g. being a basic human quality which religion corrupts.

The Constant Fire quotes William James's The Varieties of Religious Experience as saying:

Religion... shall mean for us the feelings, acts, and experiences of individual men in their solitude; so far as they apprehend themselves to stand in relation to whatever they may consider the divine.

And this theme is developed further: Sacredness is something intensely private and individual.

Which completely nonplussed me. Am I supposed to not have any feeling of sacredness if I'm one of many people watching the video of SpaceShipOne winning the X-Prize? Why not? Am I supposed to think that my experience of sacredness has to be somehow different from that of all the other people watching? Why, when we all have the same brain design? Indeed, why would I need to believe I was unique? (But "unique" is another word Adam Frank uses; so-and-so's "unique experience of the sacred".) Is the feeling private in the same sense that we have difficulty communicating any experience? Then why emphasize this of sacredness, rather than sneezing?

The light came on when I realized that I was looking at a trick of Dark Side Epistemology - if you make something private, that shields it from criticism. You can say, "You can't criticize me, because this is my private, inner experience that you can never access to question it."

But the price of shielding yourself from criticism is that you are cast into solitude - the solitude that William James admired as the core of religious experience, as if loneliness were a good thing.

Such relics of Dark Side Epistemology are key to understanding the many ways that religion twists the experience of sacredness:

Mysteriousness - why should the sacred have to be mysterious? A space shuttle launch gets by just fine without being mysterious. How much less would I appreciate the stars if I did not know what they were, if they were just little points in the night sky? But if your religious beliefs are questioned - if someone asks, "Why doesn't God heal amputees?" - then you take refuge and say, in a tone of deep profundity, "It is a sacred mystery!" There are questions that must not be asked, and answers that must not be acknowledged, to defend the lie. Thus unanswerability comes to be associated with sacredness. And the price of shielding yourself from criticism is giving up the true curiosity that truly wishes to find answers. You will worship your own ignorance of the temporarily unanswered questions of your own generation - probably including ones that are already answered.

Faith - in the early days of religion, when people were more naive, when even intelligent folk actually believed that stuff, religions staked their reputation upon the testimony of miracles in their scriptures. And Christian archaeologists set forth truly expecting to find the ruins of Noah's Ark. But when no such evidence was forthcoming, then religion executed what William Bartley called the retreat to commitment, "I believe because I believe!" Thus belief without good evidence came to be associated with the experience of the sacred. And the price of shielding yourself from criticism is that you sacrifice your ability to think clearly about that which is sacred, and to progress in your understanding of the sacred, and relinquish mistakes.

Experientialism - if before you thought that the rainbow was a sacred contract of God with humanity, and then you begin to realize that God doesn't exist, then you may execute a retreat to pure experience - to praise yourself just for feeling such wonderful sensations when you think about God, whether or not God actually exists. And the price of shielding yourself from criticism is solipsism: your experience is stripped of its referents. What a terrible hollow feeling it would be to watch a space shuttle rising on a pillar of flame, and say to yourself, "But it doesn't really matter whether the space shuttle actually exists, so long as I feel."

Separation - if the sacred realm is not subject to ordinary rules of evidence or investigable by ordinary means, then it must be different in kind from the world of mundane matter: and so we are less likely to think of a space shuttle as a candidate for sacredness, because it is a work of merely human hands. Keats lost his admiration of the rainbow and demoted it to the "dull catalogue of mundane things" for the crime of its woof and texture being known. And the price of shielding yourself from all ordinary criticism is that you lose the sacredness of all merely real things.

Privacy - of this I have already spoken.

Such distortions are why we had best not to try to salvage religion. No, not even in the form of "spirituality". Take away the institutions and the factual mistakes, subtract the churches and the scriptures, and you're left with... all this nonsense about mysteriousness, faith, solipsistic experience, private solitude, and discontinuity.

The original lie is only the beginning of the problem. Then you have all the ill habits of thought that have evolved to defend it. Religion is a poisoned chalice, from which we had best not even sip. Spirituality is the same cup after the original pellet of poison has been taken out, and only the dissolved portion remains - a little less directly lethal, but still not good for you.

When a lie has been defended for ages upon ages, the true origin of the inherited habits lost in the mists, with layer after layer of undocumented sickness; then the wise, I think, will start over from scratch, rather than trying to selectively discard the original lie while keeping the habits of thought that protected it. Just admit you were wrong, give up entirely on the mistake, stop defending it at all, stop trying to say you were even a little right, stop trying to save face, just say "Oops!" and throw out the whole thing and begin again.

That capacity - to really, really, without defense, admit you were entirely wrong - is why religious experience will never be like scientific experience. No religion can absorb that capacity without losing itself entirely and becoming simple humanity...

...to just look up at the distant stars. Believable without strain, without a constant distracting struggle to fend off your awareness of the counterevidence. Truly there in the world, the experience united with the referent, a solid part of that unfolding story. Knowable without threat, offering true meat for curiosity. Shared in togetherness with the many other onlookers, no need to retreat to privacy. Made of the same fabric as yourself and all other things. Most holy and beautiful, the sacred mundane.

Sequence: Against Rationalization

Dark Side Epistemology

Referenced by: Church vs. Taskforce • My Way • Why I'm Staying On Bloggingheads.tv

Original with comments: The Sacred Mundane

Your Price for Joining

Eliezer Yudkowsky, 26 March 2009 07:16AM

Previously in series: Why Our Kind Can't Cooperate

In the Ultimatum Game, the first player chooses how to split $10 between themselves and the second player, and the second player decides whether to accept the split or reject it - in the latter case, both parties get nothing. So far as conventional causal decision theory goes (two-box on Newcomb's Problem, defect in Prisoner's Dilemma), the second player should prefer any non-zero amount to nothing. But if the first player expects this behavior - accept any non-zero offer - then they have no motive to offer more than a penny. As I assume you all know by now, I am no fan of conventional causal decision theory. Those of us who remain interested in cooperating on the Prisoner's Dilemma, either because it's iterated, or because we have a term in our utility function for fairness, or because we use an unconventional decision theory, may also not accept an offer of one penny.

And in fact, most Ultimatum "deciders" offer an even split; and most Ultimatum "accepters" reject any offer less than 20%. A 100 USD game played in Indonesia (average per capita income at the time: 670 USD) showed offers of 30 USD being turned down, although this equates to two week's wages. We can probably also assume that the players in Indonesia were not thinking about the academic debate over Newcomblike problems - this is just the way people feel about Ultimatum Games, even ones played for real money.

There's an analogue of the Ultimatum Game in group coordination. (Has it been studied? I'd hope so...) Let's say there's a common project - in fact, let's say that it's an altruistic common project, aimed at helping mugging victims in Canada, or something. If you join this group project, you'll get more done than you could on your own, relative to your utility function. So, obviously, you should join.

But wait! The anti-mugging project keeps their funds invested in a money market fund! That's ridiculous; it won't earn even as much interest as US Treasuries, let alone a dividend-paying index fund.

Clearly, this project is run by morons, and you shouldn't join until they change their malinvesting ways.

Now you might realize - if you stopped to think about it - that all things considered, you would still do better by working with the common anti-mugging project, than striking out on your own to fight crime. But then - you might perhaps also realize - if you too easily assent to joining the group, why, what motive would they have to change their malinvesting ways?

Well... Okay, look. Possibly because we're out of the ancestral environment where everyone knows everyone else... and possibly because the nonconformist crowd tries to repudiate normal group-cohering forces like conformity and leader-worship...

...It seems to me that people in the atheist/libertarian/technophile/sf-fan/etcetera cluster often set their joining prices way way way too high. Like a 50-way split Ultimatum game, where every one of 50 players demands at least 20% of the money.

If you think how often situations like this would have arisen in the ancestral environment, then it's almost certainly a matter of evolutionary psychology. System 1 emotions, not System 2 calculation. Our intuitions for when to join groups, versus when to hold out for more concessions to our own preferred way of doing things, would have been honed for hunter-gatherer environments of, e.g., 40 people all of whom you knew personally.

And if the group is made up of 1000 people? Then your hunter-gatherer instincts will underestimate the inertia of a group so large, and demand an unrealistically high price (in strategic shifts) for you to join. There's a limited amount of organizational effort, and a limited number of degrees of freedom, that can go into doing things any one's person way.

And if the strategy is large and complex, the sort of thing that takes e.g. ten people doing paperwork for a week, rather than being hammered out over a half-hour of negotiation around a campfire? Then your hunter-gatherer instincts will underestimate the inertia of the group, relative to your own demands.

And if you live in a wider world than a single hunter-gatherer tribe, so that you only see the one group representative who negotiates with you, and not the hundred other negotiations that have taken place already? Then your instincts will tell you that it is just one person, a stranger at that, and the two of you are equals; whatever ideas they bring to the table are equal with whatever ideas you bring to the table, and the meeting point ought to be about even.

And if you suffer from any weakness of will or akrasia, or if you are influenced by motives other than those you would admit to yourself that you are influenced by, then any group-altruistic project which does not offer you the rewards of status and control, may perhaps find itself underserved by your attentions.

Now I do admit that I speak here primarily from the perspective of someone who goes around trying to herd cats; and not from the other side as someone who spends most of their time withholding their energies in order to blackmail those damned morons already on the project. Perhaps I am a little prejudiced.

But it seems to me that a reasonable rule of thumb might be as follows:

If, on the whole, joining your efforts to a group project would still have a net positive effect according to your utility function -

(or a larger positive effect than any other marginal use to which you could otherwise put those resources, although this latter mode of thinking seems little-used and humanly-unrealistic, for reasons I may post about some other time)

- and the awful horrible annoying issue is not so important that you personally will get involved deeply enough to put in however many hours, weeks, or years may be required to get it fixed up -

- then the issue is not worth you withholding your energies from the project; either instinctively until you see that people are paying attention to you and respecting you, or by conscious intent to blackmail the group into getting it done.

And if the issue is worth that much to you... then by all means, join the group and do whatever it takes to get things fixed up.

Now, if the existing contributors refuse to let you do this, and a reasonable third party would be expected to conclude that you were competent enough to do it, and there is no one else whose ox is being gored thereby, then, perhaps, we have a problem on our hands. And it may be time for a little blackmail, if the resources you can conditionally commit are large enough to get their attention.

Is this rule a little extreme? Oh, maybe. There should be a motive for the decision-making mechanism of a project to be responsible to its supporters; unconditional support would create its own problems.

But usually... I observe that people underestimate the costs of what they ask for, or perhaps just act on instinct, and set their prices way way way too high. If the nonconformist crowd ever wants to get anything done together, we need to move in the direction of joining groups and staying there at least a little more easily. Even in the face of annoyances and imperfections! Even in the face of unresponsiveness to our own better ideas!

In the age of the Internet and in the company of nonconformists, it does get a little tiring reading the 451st public email from someone saying that the Common Project isn't worth their resources until the website has a sans-serif font.

Of course this often isn't really about fonts. It may be about laziness, akrasia, or hidden rejections. But in terms of group norms... in terms of what sort of public statements we respect, and which excuses we publicly scorn... we probably do want to encourage a group norm of:

If the issue isn't worth your personally fixing by however much effort it takes, and it doesn't arise from outright bad faith, it's not worth refusing to contribute your efforts to a cause you deem worthwhile.

Sequence: The Craft and the Community

On Things that are Awesome

Can Humanism Match Religion's Output?

Referenced by: Can Humanism Match Religion's Output? • Helpless Individuals • The Unfinished Mystery of the Shangri-La Diet • Bayesians vs. Barbarians • The Craft and the Community

Original with comments: Your Price for Joining

Can Humanism Match Religion's Output?

Eliezer Yudkowsky, 27 March 2009 11:32AM

Previously in series: Your Price for Joining

Perhaps the single largest voluntary institution of our modern world - bound together not by police and taxation, not by salaries and managers, but by voluntary donations flowing from its members - is the Catholic Church.

It's too large to be held together by individual negotiations, like a group task in a hunter-gatherer band. But in a larger world with more people to be infected and faster transmission, we can expect more virulent memes. The Old Testament doesn't talk about Hell, but the New Testament does. The Catholic Church is held together by affective death spirals - around the ideas, the institutions, and the leaders. By promises of eternal happiness and eternal damnation - theologians don't really believe that stuff, but many ordinary Catholics do. By simple conformity of people meeting in person at a Church and being subjected to peer pressure. &c.

We who have the temerity to call ourselves "rationalists", think ourselves too good for such communal bindings.

And so anyone with a simple and obvious charitable project - responding with food and shelter to a tidal wave in Thailand, say - would be better off by far pleading with the Pope to mobilize the Catholics, rather than with Richard Dawkins to mobilize the atheists.

For so long as this is true, any increase in atheism at the expense of Catholicism will be something of a hollow victory, regardless of all other benefits.

True, the Catholic Church also goes around opposing the use of condoms in AIDS-ravaged Africa. True, they waste huge amounts of the money they raise on all that religious stuff. Indulging in unclear thinking is not harmless, prayer comes with a price.

To refrain from doing damaging things, is a true victory for a rationalist...

Unless it is your only victory, in which case it seems a little empty.

If you discount all harm done by the Catholic Church, and look only at the good... then does the average Catholic do more gross good than the average atheist, just by virtue of being more active?

Perhaps if you are wiser but less motivated, you can search out interventions of high efficiency and purchase utilons on the cheap... But there are few of us who really do that, as opposed to planning to do it someday.

Now you might at this point throw up your hands, saying: "For so long as we don't have direct control over our brain's motivational circuitry, it's not realistic to expect a rationalist to be as strongly motivated as someone who genuinely believes that they'll burn eternally in hell if they don't obey."

This is a fair point. Any folk theorem to the effect that a rational agent should do at least as well as a non-rational agent will rely on the assumption that the rational agent can always just implement whatever "irrational" policy is observed to win. But if you can't choose to have unlimited mental energy, then it may be that some false beliefs are, in cold fact, more strongly motivating than any available true beliefs. And if we all generally suffer from altruistic akrasia, being unable to bring ourselves to help as much as we think we should, then it is possible for the God-fearing to win the contest of altruistic output.

But though it is a motivated continuation, let us consider this question a little further.

Even the fear of hell is not a perfect motivator. Human beings are not given so much slack on evolution's leash; we can resist motivation for a short time, but then we run out of mental energy (HT: infotropism). Even believing that you'll go to hell does not change this brute fact about brain circuitry. So the religious sin, and then are tormented by thoughts of going to hell, in much the same way that smokers reproach themselves for being unable to quit.

If a group of rationalists cared a lot about something... who says they wouldn't be able to match the real, de-facto output of a believing Catholic? The stakes might not be "infinite" happiness or "eternal" damnation, but of course the brain can't visualize 3^^^3, let alone infinity. Who says that the actual quantity of caring neurotransmitters discharged by the brain (as 'twere) has to be so much less for "the growth and flowering of humankind" or even "tidal-wave-stricken Thais", than for "eternal happiness in Heaven"? Anything involving more than 100 people is going to involve utilities too large to visualize. And there are all sorts of other standard biases at work here; knowing about them might be good for a bonus as well, one hopes?

Cognitive-behavioral therapy and Zen meditation are two mental disciplines experimentally shown to yield real improvements. It is not the area of the art I've focused on developing, but then I don't have a real martial art of rationality in back of me. If you combine a purpose genuinely worth caring about, with discipline extracted from CBT and Zen meditation, then who says rationalists can't keep up? Or even more generally: if we have an evidence-based art of fighting akrasia, with experiments to see what actually works, then who says we've got to be less motivated than some disorganized mind that fears God's wrath?

Still... that's a further-future speculation that it might be possible to develop an art that doesn't presently exist. It's not a technique I can use right now. I present it just to illustrate the idea of not giving up so fast on rationality: Understanding what's going wrong, trying intelligently to fix it, and gathering evidence on whether it worked - this is a powerful idiom, not to be lightly dismissed upon sighting the first disadvantage.

Really, I suspect that what's going on here has less to do with the motivating power of eternal damnation, and a lot more to do with the motivating power of physically meeting other people who share your cause. The power, in other words, of being physically present at church and having religious neighbors.

This is a problem for the rationalist community in its present stage of growth, because we are rare and geographically distributed way the hell all over the place. If all the readers of this blog lived within a 5-mile radius of each other, I bet we'd get a lot more done, not for reasons of coordination but just sheer motivation.

I'll post tomorrow about some long-term, starry-eyed, idealistic thoughts on this particular problem. Shorter-term solutions that don't rely on our increasing our numbers by a factor of 100 would be better. I wonder in particular whether the best modern videoconferencing software would provide some of the motivating effect of meeting someone in person; I suspect the answer is "no" but it might be worth trying.

Meanwhile... in the short-term, we're stuck fighting akrasia mostly without the reinforcing physical presense of other people who care. I want to say something like "This is difficult, but it can be done" except I'm not sure that's even true.

I suspect that the largest step rationalists could take toward matching the per-capita power output of the Catholic Church would be to have regular physical meetings of people contributing to the same task - not for purposes of coordination, just for purposes of of motivation.

In the absence of that...

We could try for a group norm of being openly allowed - nay, applauded - for caring strongly about something. And a group norm of being expected to do something useful with your life - contribute your part to cleaning up this world. Religion doesn't really emphasize the getting-things-done aspect as much.

And if rationalists could match just half the average altruistic effort output per Catholic, then I don't think it's remotely unrealistic to suppose that with better targeting on more efficient causes, the modal rationalist could get twice as much done.

How much of its earnings does the Catholic Church spend on all that useless religious stuff instead of actually helping people? More than 50%, I would venture. So then we could say - with a certain irony, though that's not quite the spirit in which we should be doing things - that we should try to propagate a group norm of donating a minimum of 5% of income to real causes. (10% being the usual suggested minimum religious tithe.) And then there's the art of picking causes for which expected utilons are orders of magnitude cheaper (for so long as the inefficient market in utilons lasts).

But long before we can begin to dream of any such boast, we secular humanists need to work on at least matching the per capita benevolent output of the worshippers.

Sequence: The Craft and the Community

Your Price for Joining

Church vs. Taskforce

Referenced by: Church vs. Taskforce • The Craft and the Community

Original with comments: Can Humanism Match Religion's Output?

Church vs. Taskforce

Eliezer Yudkowsky, 28 March 2009 09:23AM

Previously in series: Can Humanism Match Religion's Output?
Followup to: Is Humanism a Religion-Substitute?

I am generally suspicious of envying crazy groups or trying to blindly copycat the rhythm of religion - what I called "hymns to the nonexistence of God", replying, "A good 'atheistic hymn' is simply a song about anything worth singing about that doesn't happen to be religious."

But religion does fill certain holes in people's minds, some of which are even worth filling. If you eliminate religion, you have to be aware of what gaps are left behind.

If you suddenly deleted religion from the world, the largest gap left would not be anything of ideals or morals; it would be the church, the community. Among those who now stay religious without quite really believing in God - how many are just sticking to it from wanting to stay with their neighbors at the church, and their family and friends? How many would convert to atheism, if all those others deconverted, and that were the price of staying in the community and keeping its respect? I would guess... probably quite a lot.

In truth... this is probably something I don't understand all that well, myself. "Brownies and babysitting" were the first two things that came to mind. Do churches lend helping hands in emergencies? Or just a shoulder to cry on? How strong is a church community? It probably depends on the church, and in any case, that's not the correct question. One should start by considering what a hunter-gatherer band gives its people, and ask what's missing in modern life - if a modern First World church fills only some of that, then by all means let us try to do better.

So without copycatting religion - without assuming that we must gather every Sunday morning in a building with stained-glass windows while the children dress up in formal clothes and listen to someone sing - let's consider how to fill the emotional gap, after religion stops being an option.

To help break the mold to start with - the straitjacket of cached thoughts on how to do this sort of thing - consider that some modern offices may also fill the same role as a church. By which I mean that some people are fortunate to receive community from their workplaces: friendly coworkers who bake brownies for the office, whose teenagers can be safely hired for babysitting, and maybe even help in times of catastrophe...? But certainly not everyone is lucky enough to find a community at the office.

Consider further - a church is ostensibly about worship, and a workplace is ostensibly about the commercial purpose of the organization. Neither has been carefully optimized to serve as a community.

Looking at a typical religious church, for example, you could suspect - although all of these things would be better tested experimentally, than just suspected -

- That getting up early on a Sunday morning is not optimal;

- That wearing formal clothes is not optimal, especially for children;

- That listening to the same person give sermons on the same theme every week ("religion") is not optimal;

- That the cost of supporting a church and a pastor is expensive, compared to the number of different communities who could time-share the same building for their gatherings;

- That they probably don't serve nearly enough of a matchmaking purpose, because churches think they're supposed to enforce their medieval moralities;

- That the whole thing ought to be subject to experimental data-gathering to find out what works and what doesn't.

By using the word "optimal" above, I mean "optimal under the criteria you would use if you were explicitly building a community qua community". Spending lots of money on a fancy church with stained-glass windows and a full-time pastor makes sense if you actually want to spend money on religion qua religion.

I do confess that when walking past the churches of my city, my main thought is "These buildings look really, really expensive, and there are too many of them." If you were doing it over from scratch... then you might have a big building that could be used for the occasional wedding, but it would be time-shared for different communities meeting at different times on the weekend, and it would also have a nice large video display that could be used for speakers giving presentations, lecturers teaching something, or maybe even showing movies. Stained glass? Not so high a priority.

Or to the extent that the church membership lends a helping hand in times of trouble - could that be improved by an explicit rainy-day fund or contracting with an insurer, once you realized that this was an important function? Possibly not; dragging explicit finance into things changes their character oddly. Conversely, maybe keeping current on some insurance policies should be a requirement for membership, lest you rely too much on the community... But again, to the extent that churches provide community, they're trying to do it without actually admitting that this nearly all of what people get out of it. Same thing with the corporations whose workplaces are friendly enough to serve as communities; it's still something of an accidental function.

Once you start thinking explicitly about how to give people a hunter-gatherer band to belong to, you can see all sorts of things that sound like good ideas. Should you welcome the newcomer in your midst? The pastor may give a sermon on that sometime, if you think church is about religion. But if you're explicitly setting out to build community - then right after a move is when someone most lacks community, when they most need your help. It's also an opportunity for the band to grow. If anything, tribes ought to be competing at quarterly exhibitions to capture newcomers.

But can you really have a community that's just a community - that isn't also an office or a religion? A community with no purpose beyond itself?

Maybe you can. After all, hunter-gatherer tribes have any purposes beyond themselves? - well, there was survival and feeding yourselves, that was a purpose.

But anything that people have in common, especially any goal they have in common, tends to want to define a community. Why not take advantage of that?

Though in this age of the Internet, alas, too many binding factors have supporters too widely distributed to form a decent band - if you're the only member of the Church of the Subgenius in your city, it may not really help much. It really is different without the physical presence; the Internet does not seem to be an acceptable substitute at the current stage of the technology.

So to skip right to the point -

Should the Earth last so long, I would like to see, as the form of rationalist communities, taskforces focused on all the work that needs doing to fix up this world. Communities in any geographic area would form around the most specific cluster that could support a decent-sized band. If your city doesn't have enough people in it for you to find 50 fellow Linux programmers, you might have to settle for 15 fellow open-source programmers... or in the days when all of this is only getting started, 15 fellow rationalists trying to spruce up the Earth in their assorted ways.

That's what I think would be a fitting direction for the energies of communities, and a common purpose that would bind them together. Tasks like that need communities anyway, and this Earth has plenty of work that needs doing, so there's no point in waste. We have so much that needs doing - let the energy that was once wasted into the void of religious institutions, find an outlet there. And let purposes admirable without need for delusion, fill any void in the community structure left by deleting religion and its illusionary higher purposes.

Strong communities built around worthwhile purposes: That would be the shape I would like to see for the post-religious age, or whatever fraction of humanity has then gotten so far in their lives.

Although... as long as you've got a building with a nice large high-resolution screen anyway, I wouldn't mind challenging the idea that all post-adulthood learning has to take place in distant expensive university campuses with teachers who would rather be doing something else. And it's empirically the case that colleges seem to support communities quite well. So in all fairness, there are other possibilities for things you could build a post-theistic community around.

Is all of this just a dream? Maybe. Probably. It's not completely devoid of incremental implementability, if you've got enough rationalists in a sufficiently large city who have heard of the idea. But on the off-chance that rationality should catch on so widely, or the Earth should last so long, and that my voice should be heard, then that is the direction I would like to see things moving in - as the churches fade, we don't need artificial churches, but we do need new idioms of community.

Sequence: The Craft and the Community

Can Humanism Match Religion's Output?

Rationality: Common Interest of Many Causes

Referenced by: Rationality: Common Interest of Many Causes • Helpless Individuals • Of Gender and Rationality • The Craft and the Community

Original with comments: Church vs. Taskforce

Rationality: Common Interest of Many Causes

Eliezer Yudkowsky, 29 March 2009 10:49AM

Previously in series: Church vs. Taskforce

It is a non-so-hidden agenda of this site, Less Wrong, that there are many causes which benefit from the spread of rationality - because it takes a little more rationality than usual to see their case, as a supporter, or even just a supportive bystander. Not just the obvious causes like atheism, but things like marijuana legalization - where you could wish that people were a bit more self-aware about their motives and the nature of signaling, and a bit more moved by inconvenient cold facts. The Institute Which May Not Be Named was merely an unusually extreme case of this, wherein it got to the point that after years of bogging down I threw up my hands and explicitly recursed on the job of creating rationalists.

But of course, not all the rationalists I create will be interested in my own project - and that's fine. You can't capture all the value you create, and trying can have poor side effects.

If the supporters of other causes are enlightened enough to think similarly...

Then all the causes which benefit from spreading rationality, can, perhaps, have something in the way of standardized material to which to point their supporters - a common task, centralized to save effort - and think of themselves as spreading a little rationality on the side. They won't capture all the value they create. And that's fine. They'll capture some of the value others create. Atheism has very little to do directly with marijuana legalization, but if both atheists and anti-Prohibitionists are willing to step back a bit and say a bit about the general, abstract principle of confronting a discomforting truth that interferes with a fine righteous tirade, then both atheism and marijuana legalization pick up some of the benefit from both efforts.

But this requires - I know I'm repeating myself here, but it's important - that you be willing not to capture all the value you create. It requires that, in the course of talking about rationality, you maintain an ability to temporarily shut up about your own cause even though it is the best cause ever. It requires that you don't regard those other causes, and they do not regard you, as competing for a limited supply of rationalists with a limited capacity for support; but, rather, creating more rationalists and increasing their capacity for support. You only reap some of your own efforts, but you reap some of others' efforts as well.

If you and they don't agree on everything - especially priorities - you have to be willing to agree to shut up about the disagreement. (Except possibly in specialized venues, out of the way of the mainstream discourse, where such disagreements are explicitly prosecuted.)

A certain person who was taking over as the president of a certain organization once pointed out that the organization had not enjoyed much luck with its message of "This is the best thing you can do", as compared to e.g. the X-Prize Foundation's tremendous success conveying to rich individuals of "Here is a cool thing you can do."

This is one of those insights where you blink incredulously and then grasp how much sense it makes. The human brain can't grasp large stakes and people are not anything remotely like expected utility maximizers, and we are generally altruistic akrasics. Saying, "This is the best thing" doesn't add much motivation beyond "This is a cool thing". It just establishes a much higher burden of proof. And invites invidious motivation-sapping comparison to all other good things you know (perhaps threatening to diminish moral satisfaction already purchased).

If we're operating under the assumption that everyone by default is an altruistic akrasic (someone who wishes they could choose to do more) - or at least, that most potential supporters of interest fit this description - then fighting it out over which cause is the best to support, may have the effect of decreasing the overall supply of altruism.

"But," you say, "dollars are fungible; a dollar you use for one thing indeed cannot be used for anything else!" To which I reply: But human beings really aren't expected utility maximizers, as cognitive systems. Dollars come out of different mental accounts, cost different amounts of willpower (the true limiting resource) under different circumstances, people want to spread their donations around as an act of mental accounting to minimize the regret if a single cause fails, and telling someone about an additional cause may increase the total amount they're willing to help.

There are, of course, limits to this principle of benign tolerance. If someone has a project to help stray puppies get warm homes, then it's probably best to regard them as trying to exploit bugs in human psychology for their personal gain, rather than a worthy sub-task of the great common Neo-Enlightenment project of human progress.

But to the extent that something really is a task you would wish to see done on behalf of humanity... then invidious comparisons of that project to Your-Favorite-Project, may not help your own project as much as you might think. We may need to learn to say, by habit and in nearly all forums, "Here is a cool rationalist project", not, "Mine alone is the highest-return in expected utilons per marginal dollar project." If someone cold-blooded enough to maximize expected utility of fungible money without regard to emotional side effects explicitly asks, we could perhaps steer them to a specialized subforum where anyone willing to make the claim of top priority fights it out. Though if all goes well, those projects that have a strong claim to this kind of underserved-ness will get more investment and their marginal returns will go down, and the winner of the competing claims will no longer be clear.

If there are many rationalist projects that benefit from raising the sanity waterline, then their mutual tolerance and common investment in spreading rationality could conceivably exhibit a commons problem. But this doesn't seem too hard to deal with: if there's a group that's not willing to share the rationalists they create or mention to them that other Neo-Enlightenment projects might exist, then any common, centralized rationalist resources could remove the mention of their project as a cool thing to do.

Though all this is an idealistic and future-facing thought, the benefits - for all of us - could be finding some important things we're missing right now. So many rationalist projects have few supporters and far-flung; if we could all identify as elements of the Common Project of human progress, the Neo-Enlightenment, there would be a substantially higher probability of finding ten of us in any given city. Right now, a lot of these projects are just a little lonely for their supporters. Rationality may not be the most important thing in the world - that, of course, is the thing that we protect - but it is a cool thing that more of us have in common. We might gain much from identifying ourselves also as rationalists.

Sequence: The Craft and the Community

Church vs. Taskforce

Helpless Individuals

Referenced by: Helpless Individuals • Money: The Unit of Caring • The Craft and the Community • The End (of Sequences)

Original with comments: Rationality: Common Interest of Many Causes

Helpless Individuals

Eliezer Yudkowsky, 30 March 2009 11:10AM

Previously in series: Rationality: Common Interest of Many Causes

When you consider that our grouping instincts are optimized for 50-person hunter-gatherer bands where everyone knows everyone else, it begins to seem miraculous that modern-day large institutions survive at all.

Well - there are governments with specialized militaries and police, which can extract taxes. That's a non-ancestral idiom which dates back to the invention of sedentary agriculture and extractible surpluses; humanity is still struggling to deal with it.

There are corporations in which the flow of money is controlled by centralized management, a non-ancestral idiom dating back to the invention of large-scale trade and professional specialization.

And in a world with large populations and close contact, memes evolve far more virulent than the average case of the ancestral environment; memes that wield threats of damnation, promises of heaven, and professional priest classes to transmit them.

But by and large, the answer to the question "How do large institutions survive?" is "They don't!" The vast majority of large modern-day institutions - some of them extremely vital to the functioning of our complex civilization - simply fail to exist in the first place.

I first realized this as a result of grasping how Science gets funded: namely, not by individual donations.

Science traditionally gets funded by governments, corporations, and large foundations. I've had the opportunity to discover firsthand that it's amazingly difficult to raise money for Science from individuals. Not unless it's science about a disease with gruesome victims, and maybe not even then.

Why? People are, in fact, prosocial; they give money to, say, puppy pounds. Science is one of the great social interests, and people are even widely aware of this - why not Science, then?

Any particular science project - say, studying the genetics of trypanotolerance in cattle - is not a good emotional fit for individual charity. Science has a long time horizon that requires continual support. The interim or even final press releases may not sound all that emotionally arousing. You can't volunteer; it's a job for specialists. Being shown a picture of the scientist you're supporting at or somewhat below the market price for their salary, lacks the impact of being shown the wide-eyed puppy that you helped usher to a new home. You don't get the immediate feedback and the sense of immediate accomplishment that's required to keep an individual spending their own money.

Ironically, I finally realized this, not from my own work, but from thinking "Why don't Seth Roberts's readers come together to support experimental tests of Roberts's hypothesis about obesity? Why aren't individual philanthropists paying to test Bussard's polywell fusor?" These are examples of obviously ridiculously underfunded science, with applications (if true) that would be relevant to many, many individuals. That was when it occurred to me that, in full generality, Science is not a good emotional fit for people spending their own money.

In fact very few things are, with the individuals we have now. It seems to me that this is key to understanding how the world works the way it does - why so many individual interests are poorly protected - why 200 million adult Americans have such tremendous trouble supervising the 535 members of Congress, for example.

So how does Science actually get funded? By governments that think they ought to spend some amount of money on Science, with legislatures or executives deciding to do so - it's not quite their own money they're spending. Sufficiently large corporations decide to throw some amount of money at blue-sky R&D. Large grassroots organizations built around affective death spirals may look at science that suits their ideals. Large private foundations, based on money block-allocated by wealthy individuals to their reputations, spend money on Science which promises to sound very charitable, sort of like allocating money to orchestras or modern art. And then the individual scientists (or individual scientific task-forces) fight it out for control of that pre-allocated money supply, given into the hands of grant committee members who seem like the sort of people who ought to be judging scientists.

You rarely see a scientific project making a direct bid for some portion of society's resource flow; rather, it first gets allocated to Science, and then scientists fight over who actually gets it. Even the exceptions to this rule are more likely to be driven by politicians (moonshot) or military purposes (Manhattan project) than by the appeal of scientists to the public.

Now I'm sure that if the general public were in the habit of funding particular science by individual donations, a whole lotta money would be wasted on e.g. quantum gibberish - assuming that the general public somehow acquired the habit of funding science without changing any other facts about the people or the society.

But it's still an interesting point that Science manages to survive not because it is in our collective individual interest to see Science get done, but rather, because Science has fastened itself as a parasite onto the few forms of large organization that can exist in our world. There are plenty of other projects that simply fail to exist in the first place.

It seems to me that modern humanity manages to put forth very little in the way of coordinated effort to serve collective individual interests. It's just too non-ancestral a problem when you scale to more than 50 people. There are only big taxers, big traders, supermemes, occasional individuals of great power; and a few other organizations, like Science, that can fasten parasitically onto them.

Sequence: The Craft and the Community

Rationality: Common Interest of Many Causes

Money: The Unit of Caring

Referenced by: Money: The Unit of Caring • Collective Apathy and the Internet • Bayesians vs. Barbarians • The Craft and the Community

Original with comments: Helpless Individuals

Money: The Unit of Caring

Eliezer Yudkowsky, 31 March 2009 12:35PM

Previously in series: Helpless Individuals

Steve Omohundro has suggested a folk theorem to the effect that, within the interior of any approximately rational, self-modifying agent, the marginal benefit of investing additional resources in anything ought to be about equal. Or, to put it a bit more exactly, shifting a unit of resource between any two tasks should produce no increase in expected utility, relative to the agent's utility function and its probabilistic expectations about its own algorithms.

This resource balance principle implies that - over a very wide range of approximately rational systems, including even the interior of a self-modifying mind - there will exist some common currency of expected utilons, by which everything worth doing can be measured.

In our society, this common currency of expected utilons is called "money". It is the measure of how much society cares about something.

This is a brutal yet obvious point, which many are motivated to deny.

With this audience, I hope, I can simply state it and move on. It's not as if you thought "society" was intelligent, benevolent, and sane up until this point, right?

I say this to make a certain point held in common across many good causes. Any charitable institution you've ever had a kind word for, certainly wishes you would appreciate this point, whether or not they've ever said anything out loud. For I have listened to others in the nonprofit world, and I know that I am not speaking only for myself here...

Many people, when they see something that they think is worth doing, would like to volunteer a few hours of spare time, or maybe mail in a five-year-old laptop and some canned goods, or walk in a march somewhere, but at any rate, not spend money.

Believe me, I understand the feeling. Every time I spend money I feel like I'm losing hit points. That's the problem with having a unified quantity describing your net worth: Seeing that number go down is not a pleasant feeling, even though it has to fluctuate in the ordinary course of your existence. There ought to be a fun-theoretic principle against it.

But, well...

There is this very, very old puzzle/observation in economics about the lawyer who spends an hour volunteering at the soup kitchen, instead of working an extra hour and donating the money to hire someone to work for five hours at the soup kitchen.

There's this thing called "Ricardo's Law of Comparative Advantage". There's this idea called "professional specialization". There's this notion of "economies of scale". There's this concept of "gains from trade". The whole reason why we have money is to realize the tremendous gains possible from each of us doing what we do best.

This is what grownups do. This is what you do when you want something to actually get done. You use money to employ full-time specialists.

Yes, people are sometimes limited in their ability to trade time for money (underemployed), so that it is better for them if they can directly donate that which they would usually trade for money. If the soup kitchen needed a lawyer, and the lawyer donated a large contiguous high-priority block of lawyering, then that sort of volunteering makes sense - that's the same specialized capability the lawyer ordinarily trades for money. But "volunteering" just one hour of legal work, constantly delayed, spread across three weeks in casual minutes between other jobs? This is not the way something gets done when anyone actually cares about it, or to state it near-equivalently, when money is involved.

To the extent that individuals fail to grasp this principle on a gut level, they may think that the use of money is somehow optional in the pursuit of things that merely seem morally desirable - as opposed to tasks like feeding ourselves, whose desirability seems to be treated oddly differently. This factor may be sufficient by itself to prevent us from pursuing our collective common interest in groups larger than 40 people.

Economies of trade and professional specialization are not just vaguely good yet unnatural-sounding ideas, they are the only way that anything ever gets done in this world. Money is not pieces of paper, it is the common currency of caring.

Hence the old saying: "Money makes the world go 'round, love barely keeps it from blowing up."

Now, we do have the problem of akrasia - of not being able to do what we've decided to do - which is a part of the art of rationality that I hope someone else will develop; I specialize more in the impossible questions business. And yes, spending money is more painful than volunteering, because you can see the bank account number go down, whereas the remaining hours of our span are not visibly numbered. But when it comes time to feed yourself, do you think, "Hm, maybe I should try raising my own cattle, that's less painful than spending money on beef?" Not everything can get done without invoking Ricardo's Law; and on the other end of that trade are people who feel just the same pain at the thought of having less money.

It does seem to me offhand that there ought to be things doable to diminish the pain of losing hit points, and to increase the felt strength of the connection from donating money to "I did a good thing!" Some of that I am trying to accomplish right now, by emphasizing the true nature and power of money; and by inveighing against the poisonous meme saying that someone who gives mere money must not care enough to get personally involved. This is a mere reflection of a mind that doesn't understand the post-hunter-gatherer concept of a market economy. The act of donating money is not the momentary act of writing the check, it is the act of every hour you spent to earn the money to write that check - just as though you worked at the charity itself in your professional capacity, at maximum, grownup efficiency.

If the lawyer needs to work an hour at the soup kitchen to keep himself motivated and remind himself why he's doing what he's doing, that's fine. But he should also be donating some of the hours he worked at the office, because that is the power of professional specialization and it is how grownups really get things done. One might consider the check as buying the right to volunteer at the soup kitchen, or validating the time spent at the soup kitchen. I may post more about this later.

To a first approximation, money is the unit of caring up to a positive scalar factor - the unit of relative caring. Some people are frugal and spend less money on everything; but if you would, in fact, spend $5 on a burrito, then whatever you will not spend $5 on, you care about less than you care about the burrito. If you don't spend two months salary on a diamond ring, it doesn't mean you don't love your Significant Other. ("De Beers: It's Just A Rock.") But conversely, if you're always reluctant to spend any money on your SO, and yet seem to have no emotional problems with spending $1000 on a flat-screen TV, then yes, this does say something about your relative values.

Yes, frugality is a virtue. Yes, spending money hurts. But in the end, if you are never willing to spend any units of caring, it means you don't care.

Sequence: The Craft and the Community

Helpless Individuals

Purchase Fuzzies and Utilons Separately

Referenced by: Existential Angst Factory • Helpless Individuals • Purchase Fuzzies and Utilons Separately • Collective Apathy and the Internet • The Craft and the Community • The End (of Sequences)

Original with comments: Money: The Unit of Caring

Purchase Fuzzies and Utilons Separately

Eliezer Yudkowsky, 01 April 2009 09:51AM

Previously in series: Money: The Unit of Caring

Yesterday:

There is this very, very old puzzle/observation in economics about the lawyer who spends an hour volunteering at the soup kitchen, instead of working an extra hour and donating the money to hire someone...

If the lawyer needs to work an hour at the soup kitchen to keep himself motivated and remind himself why he's doing what he's doing, that's fine. But he should also be donating some of the hours he worked at the office, because that is the power of professional specialization and it is how grownups really get things done. One might consider the check as buying the right to volunteer at the soup kitchen, or validating the time spent at the soup kitchen.

I hold open doors for little old ladies. I can't actually remember the last time this happened literally (though I'm sure it has, sometime in the last year or so). But within the last month, say, I was out on a walk and discovered a station wagon parked in a driveway with its trunk completely open, giving full access to the car's interior. I looked in to see if there were packages being taken out, but this was not so. I looked around to see if anyone was doing anything with the car. And finally I went up to the house and knocked, then rang the bell. And yes, the trunk had been accidentally left open.

Under other circumstances, this would be a simple act of altruism, which might signify true concern for another's welfare, or fear of guilt for inaction, or a desire to signal trustworthiness to oneself or others, or finding altruism pleasurable. I think that these are all perfectly legitimate motives, by the way; I might give bonus points for the first, but I wouldn't deduct any penalty points for the others. Just so long as people get helped.

But in my own case, since I already work in the nonprofit sector, the further question arises as to whether I could have better employed the same sixty seconds in a more specialized way, to bring greater benefit to others. That is: can I really defend this as the best use of my time, given the other things I claim to believe?

The obvious defense - or perhaps, obvious rationalization - is that an act of altruism like this one acts as an willpower restorer, much more efficiently than, say, listening to music. I also mistrust my ability to be an altruist only in theory; I suspect that if I walk past problems, my altruism will start to fade. I've never pushed that far enough to test it; it doesn't seem worth the risk.

But if that's the defense, then my act can't be defended as a good deed, can it? For these are self-directed benefits that I list.

Well - who said that I was defending the act as a selfless good deed? It's a selfish good deed. If it restores my willpower, or if it keeps me altruistic, then there are indirect other-directed benefits from that (or so I believe). You could, of course, reply that you don't trust selfish acts that are supposed to be other-benefiting as an "ulterior motive"; but then I could just as easily respond that, by the same principle, you should just look directly at the original good deed rather than its supposed ulterior motive.

Can I get away with that? That is, can I really get away with calling it a "selfish good deed", and still derive willpower restoration therefrom, rather than feeling guilt about it being selfish? Apparently I can. I'm surprised it works out that way, but it does. So long as I knock to tell them about the open trunk, and so long as the one says "Thank you!", my brain feels like it's done its wonderful good deed for the day.

Your mileage may vary, of course. The problem with trying to work out an art of willpower restoration is that different things seem to work for different people. (That is: We're probing around on the level of surface phenomena without understanding the deeper rules that would also predict the variations.)

But if you find that you are like me in this aspect - that selfish good deeds still work - then I recommend that you purchase warm fuzzies and utilons separately. Not at the same time. Trying to do both at the same time just means that neither ends up done well. If status matters to you, purchase status separately too!

If I had to give advice to some new-minted billionaire entering the realm of charity, my advice would go something like this:

- To purchase warm fuzzies, find some hard-working but poverty-stricken woman who's about to drop out of state college after her husband's hours were cut back, and personally, but anonymously, give her a cashier's check for $10,000. Repeat as desired.

- To purchase status among your friends, donate $100,000 to the current sexiest X-Prize, or whatever other charity seems to offer the most stylishness for the least price. Make a big deal out of it, show up for their press events, and brag about it for the next five years.

- Then - with absolute cold-blooded calculation - without scope insensitivity or ambiguity aversion - without concern for status or warm fuzzies - figuring out some common scheme for converting outcomes to utilons, and trying to express uncertainty in percentage probabilitiess - find the charity that offers the greatest expected utilons per dollar. Donate up to however much money you wanted to give to charity, until their marginal efficiency drops below that of the next charity on the list.

I would furthermore advise the billionaire that what they spend on utilons should be at least, say, 20 times what they spend on warm fuzzies - 5% overhead on keeping yourself altruistic seems reasonable, and I, your dispassionate judge, would have no trouble validating the warm fuzzies against a multiplier that large. Save that the original, fuzzy act really should be helpful rather than actively harmful.

(Purchasing status seems to me essentially unrelated to altruism. If giving money to the X-Prize gets you more awe from your friends than an equivalently priced speedboat, then there's really no reason to buy the speedboat. Just put the money under the "impressing friends" column, and be aware that this is not the "altruism" column.)

But the main lesson is that all three of these things - warm fuzzies, status, and expected utilons - can be bought far more efficiently when you buy separately, optimizing for only one thing at a time. Writing a check for $10,000,000 to a breast-cancer charity - while far more laudable than spending the same $10,000,000 on, I don't know, parties or something - won't give you the concentrated euphoria of being present in person when you turn a single human's life around, probably not anywhere close. It won't give you as much to talk about at parties as donating to something sexy like an X-Prize - maybe a short nod from the other rich. And if you threw away all concern for warm fuzzies and status, there are probably at least a thousand underserved existing charities that could produce orders of magnitude more utilons with ten million dollars. Trying to optimize for all three criteria in one go only ensures that none of them end up optimized very well - just vague pushes along all three dimensions.

Of course, if you're not a millionaire or even a billionaire - then you can't be quite as efficient about things, can't so easily purchase in bulk. But I would still say - for warm fuzzies, find a relatively cheap charity with bright, vivid, ideally in-person and direct beneficiaries. Volunteer at a soup kitchen. Or just get your warm fuzzies from holding open doors for little old ladies. Let that be validated by your other efforts to purchase utilons, but don't confuse it with purchasing utilons. Status is probably cheaper to purchase by buying nice clothes.

And when it comes to purchasing expected utilons - then, of course, shut up and multiply.

Sequence: The Craft and the Community

Money: The Unit of Caring

Selecting Rationalist Groups

Referenced by: Selecting Rationalist Groups • The Craft and the Community • The End (of Sequences)

Original with comments: Purchase Fuzzies and Utilons Separately

Wrong Tomorrow

Eliezer Yudkowsky, 02 April 2009 08:18AM

Wrong Tomorrow by Maciej CegĹ‚owski is a very simple site for listing pundit predictions and tracking them [FAQ]. It doesn't come with prices and active betting... but a simple registry of this kind can scale much faster than a market, and right now we're in a situation where no one is bothering to track pundit predictions or report on pundit track records. Predictions are produced as simple entertainment or as simple political theater, without the slightest fear of accountability.

This site is missing some features, but it looks to me like a starting attempt at what's needed - a Wikipedia-like, user-contributed, low-barrier-to-entry database of all pundit predictions, past and present.

Original with comments: Wrong Tomorrow

Selecting Rationalist Groups

Eliezer Yudkowsky, 02 April 2009 04:21PM

Previously in series: Purchase Fuzzies and Utilons Separately
Followup to: Conjuring an Evolution To Serve You

GreyThumb.blog offered an interesting comparison of poor animal breeding practices and the fall of Enron, which I previously posted on in some detail. The essential theme was that individual selection on chickens for the chicken in each generation who laid the most eggs, produced highly competitive chickens - the most dominant chickens that pecked their way to the top of the pecking order at the expense of other chickens. The chickens subjected to this individual selection for egg-laying prowess needed their beaks clipped, or housing in individual cages, or they would peck each other to death.

Which is to say: individual selection is selecting on the wrong criterion, because what the farmer actually wants is high egg production from groups of chickens.

While group selection is nearly impossible in ordinary biology, it is easy to impose in the laboratory: and breeding the best groups, rather than the best individuals, increased average days of hen survival from 160 to 348, and egg mass per bird from 5.3 to 13.3 kg.

The analogy being to the way that Enron evaluated its employees every year, fired the bottom 10%, and gave the top individual performers huge raises and bonuses. Jeff Skilling fancied himself as exploiting the wondrous power of evolution, it seems.

If you look over my accumulated essays, you will observe that the art contained therein is almost entirely individual in nature... for around the same reason that it all focuses on confronting impossibly tricky questions: That's what I was doing when I thought up all this stuff, and for the most part I worked in solitude. But this is not inherent in the Art, not reflective of what a true martial art of rationality would be like if many people had contributed to its development along many facets.

Case in point: At the recent LW / OB meetup, we played Paranoid Debating, a game that tests group rationality. As is only appropriate, this game was not the invention of any single person, but was collectively thought up in a series of suggestions by Nick Bostrom, Black Belt Bayesian, Tom McCabe, and steven0461.

In the game's final form, Robin Gane-McCalla asked us questions like "How many Rhode Islands would fit into Alaska?" and a group of (in this case) four rationalists tried to pool their knowledge and figure out the answer... except that before the round started, we each drew facedown from a set of four cards, containing one spade card and one red card. Whoever drew the red card got the job of trying to mislead the group. Whoever drew the spade showed the card and became the spokesperson, who had to select the final answer. It was interesting, trying to play this game, and realizing how little I'd practiced basic skills like trying to measure the appropriateness of another's confidence or figure out who was lying.

A bit further along, at the suggestion of Steve Rayhawk, and slightly simplified by myself, we named 60% confidence intervals for the quantity with lower and upper bounds; Steve fit a Cauchy distribution to the interval ("because it has a fatter tail than a Gaussian") and we were scored according to the log of our probability density on the true answer, except for the red-card drawer, who got the negative of this number.

The Paranoid Debating game worked surprisingly well - at least I had fun, despite somehow managing to draw the red card three out of four times. I can totally visualize doing this at some corporate training event or even at parties. The red player is technically acting as an individual and learning to practice deception, but perhaps practicing deception (in this controlled, ethically approved setting) might help you be a little less gullible in turn. As Zelazny observes, there is a difference in the arts of discovering lies and finding truth.

In a real institution... you would probably want to optimize less for fun, and more for work-relevance: something more like Black Belt Bayesian's original suggestion of The Aumann Game, no red cards. But where both B^3 and Tom McCabe originally thought in terms of scoring individuals, I would suggest forming people into groups and scoring the groups. An institution's performance is the sum of its groups more directly than it is the sum of its individuals - though of course there are interactions between groups as well. Find people who, in general, seem to have a statistical tendency to belong to high-performing groups - these are the ones who contribute much to the group, who are persuasive with good arguments.

I wonder if there are any hedge funds that practice "trio trading", by analogy with pair programming?

Hal Finney called Aumann's Agreement Theorem "the most interesting, surprising, and challenging result in the field of human bias: that mutually respectful, honest, and rational debaters cannot disagree on any factual matter once they know each other's opinions". It is not just my own essays that are skewed toward individual application; the whole trope of Traditional Rationality seems to me skewed the same way. It's the individual heretic who is the hero, and Authority the untrustworthy villain whose main job is not to resist the heretic too much, to be properly defeated. Science is cast as a competition between theories in an arena with rules designed to let the strongest contender win. Of course, it may be that I am selective in my memory, and that if I went back and read my childhood books again, I would notice more on group tactics that originally slipped my attention... but really, Aumann's Agreement Theorem doesn't get enough attention.

Of course most Bayesian math is not widely known - the Agreement Theorem is no exception here. But even the intuitively obvious counterpart of the Agreement Theorem, the treatment of others' beliefs as evidence, receives little shrift in Traditional Rationality. This may have something to do with Science developing in the midst of insanity and in defiance of Authority; that is a historical fact about how Science developed. But if the high performers of a rationality dojo need to practice the same sort of lonely dissent... well, that must not be a very effective rationality dojo.

Sequence: The Craft and the Community

Purchase Fuzzies and Utilons Separately

Incremental Progress and the Valley

Referenced by: Whining-Based Communities • The Craft and the Community

Original with comments: Selecting Rationalist Groups

Rationality is Systematized Winning

Eliezer Yudkowsky, 03 April 2009 02:41PM

Followup to: Newcomb's Problem and Regret of Rationality

"Rationalists should win," I said, and I may have to stop saying it, for it seems to convey something other than what I meant by it.

Where did the phrase come from originally? From considering such cases as Newcomb's Problem: The superbeing Omega sets forth before you two boxes, a transparent box A containing $1000 (or the equivalent in material wealth), and an opaque box B that contains either $1,000,000 or nothing. Omega tells you that It has already put $1M in box B if and only if It predicts that you will take only box B, leaving box A behind. Omega has played this game many times before, and has been right 99 times out of 100. Do you take both boxes, or only box B?

A common position - in fact, the mainstream/dominant position in modern philosophy and decision theory - is that the only reasonable course is to take both boxes; Omega has already made Its decision and gone, and so your action cannot affect the contents of the box in any way (they argue). Now, it so happens that certain types of unreasonable individuals are rewarded by Omega - who moves even before they make their decisions - but this in no way changes the conclusion that the only reasonable course is to take both boxes, since taking both boxes makes you $1000 richer regardless of the unchanging and unchangeable contents of box B.

And this is the sort of thinking that I intended to reject by saying, "Rationalists should win!"

Said Miyamoto Musashi: "The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him."

Said I: "If you fail to achieve a correct answer, it is futile to protest that you acted with propriety."

This is the distinction I had hoped to convey by saying, "Rationalists should win!"

There is a meme which says that a certain ritual of cognition is the paragon of reasonableness and so defines what the reasonable people do. But alas, the reasonable people often get their butts handed to them by the unreasonable ones, because the universe isn't always reasonable. Reason is just a way of doing things, not necessarily the most formidable; it is how professors talk to each other in debate halls, which sometimes works, and sometimes doesn't. If a hoard of barbarians attacks the debate hall, the truly prudent and flexible agent will abandon reasonableness.

No. If the "irrational" agent is outcompeting you on a systematic and predictable basis, then it is time to reconsider what you think is "rational".

For I do fear that a "rationalist" will clutch to themselves the ritual of cognition they have been taught, as loss after loss piles up, consoling themselves: "I have behaved virtuously, I have been so reasonable, it's just this awful unfair universe that doesn't give me what I deserve. The others are cheating by not doing it the rational way, that's how they got ahead of me."

It is this that I intended to guard against by saying: "Rationalists should win!" Not whine, win. If you keep on losing, perhaps you are doing something wrong. Do not console yourself about how you were so wonderfully rational in the course of losing. That is not how things are supposed to go. It is not the Art that fails, but you who fails to grasp the Art.

Likewise in the realm of epistemic rationality, if you find yourself thinking that the reasonable belief is X (because a majority of modern humans seem to believe X, or something that sounds similarly appealing) and yet the world itself is obviously Y.

But people do seem to be taking this in some other sense than I meant it - as though any person who declared themselves a rationalist would in that moment be invested with an invincible spirit that enabled them to obtain all things without effort and without overcoming disadvantages, or something, I don't know.

Maybe there is an alternative phrase to be found again in Musashi, who said: "The Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size."

"Rationality is the spirit of winning"? "Rationality is the Way of winning"? "Rationality is systematized winning"? If you have a better suggestion, post it in the comments.

Referenced by: Incremental Progress and the Valley • Whining-Based Communities • The Sin of Underconfidence • The Craft and the Community

Original with comments: Rationality is Systematized Winning

Incremental Progress and the Valley

Eliezer Yudkowsky, 04 April 2009 04:42PM

Followup to: Rationality is Systematized Winning

Yesterday I said: "Rationality is systematized winning."

"But," you protest, "the reasonable person doesn't always win!"

What do you mean by this? Do you mean that every week or two, someone who bought a lottery ticket with negative expected value, wins the lottery and becomes much richer than you? That is not a systematic loss; it is selective reporting by the media. From a statistical standpoint, lottery winners don't exist - you would never encounter one in your lifetime, if it weren't for the selective reporting.

Even perfectly rational agents can lose. They just can't know in advance that they'll lose. They can't expect to underperform any other performable strategy, or they would simply perform it.

"No," you say, "I'm talking about how startup founders strike it rich by believing in themselves and their ideas more strongly than any reasonable person would. I'm talking about how religious people are happier -"

Ah. Well, here's the the thing: An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory. There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance. This has not yet been proven, because it is not, in fact, true.

"So," you say, "what point is there then in striving to be more rational? We won't reach the perfect ideal. So we have no guarantee that our steps forward are helping."

You have no guarantee that a step backward will help you win, either. Guarantees don't exist in the world of flesh; but contrary to popular misconceptions, judgment under uncertainty is what rationality is all about.

"But we have several cases where, based on either vaguely plausible-sounding reasoning, or survey data, it looks like an incremental step forward in rationality is going to make us worse off. If it's really all about winning - if you have something to protect more important than any ritual of cognition - then why take that step?"

Ah, and now we come to the meat of it.

I can't necessarily answer for everyone, but...

My first reason is that, on a professional basis, I deal with deeply confused problems that make huge demands on precision of thought. One small mistake can lead you astray for years, and there are worse penalties waiting in the wings. An unimproved level of performance isn't enough; my choice is to try to do better, or give up and go home.

"But that's just you. Not all of us lead that kind of life. What if you're just trying some ordinary human task like an Internet startup?"

My second reason is that I am trying to push some aspects of my art further than I have seen done. I don't know where these improvements lead. The loss of failing to take a step forward is not that one step, it is all the other steps forward you could have taken, beyond that point. Robin Hanson has a saying: The problem with slipping on the stairs is not falling the height of the first step, it is that falling one step leads to falling another step. In the same way, refusing to climb one step up forfeits not the height of that step but the height of the staircase.

"But again - that's just you. Not all of us are trying to push the art into uncharted territory."

My third reason is that once I realize I have been deceived, I can't just shut my eyes and pretend I haven't seen it. I have already taken that step forward; what use to deny it to myself? I couldn't believe in God if I tried, any more than I could believe the sky above me was green while looking straight at it. If you know everything you need to know in order to know that you are better off deceiving yourself, it's much too late to deceive yourself.

"But that realization is unusual; other people have an easier time of doublethink because they don't realize it's impossible. You go around trying to actively sponsor the collapse of doublethink. You, from a higher vantage point, may know enough to expect that this will make them unhappier. So is this out of a sadistic desire to hurt your readers, or what?"

Then I finally reply that my experience so far - even in this realm of merely human possibility - does seem to indicate that, once you sort yourself out a bit and you aren't doing quite so many other things wrong, striving for more rationality actually will make you better off. The long road leads out of the valley and higher than before, even in the human lands.

The more I know about some particular facet of the Art, the more I can see this is so. As I've previously remarked, my essays may be unreflective of what a true martial art of rationality would be like, because I have only focused on answering confusing questions - not fighting akrasia, coordinating groups, or being happy. In the field of answering confusing questions - the area where I have most intensely practiced the Art - it now seems massively obvious that anyone who thought they were better off "staying optimistic about solving the problem" would get stomped into the ground. By a casual student.

When it comes to keeping motivated, or being happy, I can't guarantee that someone who loses their illusions will be better off - because my knowledge of these facets of rationality is still crude. If these parts of the Art have been developed systematically, I do not know of it. But even here I have gone to some considerable pains to dispel half-rational half-mistaken ideas that could get in a beginner's way, like the idea that rationality opposes feeling, or the idea that rationality opposes value, or the idea that sophisticated thinkers should be angsty and cynical.

And if, as I hope, someone goes on to develop the art of fighting akrasia or achieving mental well-being as thoroughly as I have developed the art of answering impossible questions, I do fully expect that those who wrap themselves in their illusions will not begin to compete. Meanwhile - others may do better than I, if happiness is their dearest desire, for I myself have invested little effort here.

I find it hard to believe that the optimally motivated individual, the strongest entrepreneur a human being can become, is still wrapped up in a blanket of comforting overconfidence. I think they've probably thrown that blanket out the window and organized their mind a little differently. I find it hard to believe that the happiest we can possibly live, even in the realms of human possibility, involves a tiny awareness lurking in the corner of your mind that it's all a lie. I'd rather stake my hopes on neurofeedback or Zen meditation, though I've tried neither.

But it cannot be denied that this is a very real issue in very real life. Consider this pair of comments from Less Wrong:

I'll be honest -- my life has taken a sharp downturn since I deconverted. My theist girlfriend, with whom I was very much in love, couldn't deal with this change in me, and after six months of painful vacillation, she left me for a co-worker. That was another six months ago, and I have been heartbroken, miserable, unfocused, and extremely ineffective since.

Perhaps this is an example of the valley of bad rationality of which PhilGoetz spoke, but I still hold my current situation higher in my preference ranking than happiness with false beliefs.

And:

My empathies: that happened to me about 6 years ago (though thankfully without as much visible vacillation).

My sister, who had some Cognitive Behaviour Therapy training, reminded me that relationships are forming and breaking all the time, and given I wasn't unattractive and hadn't retreated into monastic seclusion, it wasn't rational to think I'd be alone for the rest of my life (she turned out to be right). That was helpful at the times when my feelings hadn't completely got the better of me.

So - in practice, in real life, in sober fact - those first steps can, in fact, be painful. And then things can, in fact, get better. And there is, in fact, no guarantee that you'll end up higher than before. Even if in principle the path must go further, there is no guarantee that any given person will get that far.

If you don't prefer truth to happiness with false beliefs...

Well... and if you are not doing anything especially precarious or confusing... and if you are not buying lottery tickets... and if you're already signed up for cryonics, a sudden ultra-high-stakes confusing acid test of rationality that illustrates the Black Swan quality of trying to bet on ignorance in ignorance...

Then it's not guaranteed that taking all the incremental steps toward rationality that you can find, will leave you better off. But the vaguely plausible-sounding arguments against losing your illusions, generally do consider just one single step, without postulating any further steps, without suggesting any attempt to regain everything that was lost and go it one better. Even the surveys are comparing the average religious person to the average atheist, not the most advanced theologians to the most advanced rationalists.

But if you don't care about the truth - and you have nothing to protect - and you're not attracted to the thought of pushing your art as far as it can go - and your current life seems to be going fine - and you have a sense that your mental well-being depends on illusions you'd rather not think about -

Then you're probably not reading this. But if you are, then, I guess... well... (a) sign up for cryonics, and then (b) stop reading Less Wrong before your illusions collapse! RUN AWAY!

Sequence: The Craft and the Community

Selecting Rationalist Groups

Whining-Based Communities

Referenced by: Real-Life Anthropic Weirdness • The Craft and the Community

Original with comments: Incremental Progress and the Valley

Real-Life Anthropic Weirdness

Eliezer Yudkowsky, 05 April 2009 10:26PM

In passing, I said:

From a statistical standpoint, lottery winners don't exist - you would never encounter one in your lifetime, if it weren't for the selective reporting.

And lo, CronoDAS said:

Well... one of my grandmothers' neighbors, whose son I played with as a child, did indeed win the lottery. (AFAIK, it was a relatively modest jackpot, but he did win!)

To which I replied:

Well, yes, some of the modest jackpots are statistically almost possible, in the sense that on a large enough web forum, someone else's grandmother's neighbor will have won it. Just not your own grandmother's neighbor.

Sorry about your statistical anomalatude, CronoDAS - it had to happen to someone, just not me.

There's a certain resemblance here - though not an actual analogy - to the strange position your friend ends up in, after you test the Quantum Theory of Immortality.

For those unfamiliar with QTI, it's a simple simultaneous test of many-worlds plus a particular interpretation of anthropic observer-selection effects: You put a gun to your head and wire up the trigger to a quantum coinflipper. After flipping a million coins, if the gun still hasn't gone off, you can be pretty sure of the simultaneous truth of MWI+QTI.

But what is your watching friend supposed to think? Though his predicament is perfectly predictable to you - that is, you expected before starting the experiment to see his confusion - from his perspective it is just a pure 100% unexplained miracle. What you have reason to believe and what he has reason to believe would now seem separated by an uncrossable gap, which no amount of explanation can bridge. This is the main plausible exception I know to Aumann's Agreement Theorem.

Pity those poor folk who actually win the lottery! If the hypothesis "this world is a holodeck" is normatively assigned a calibrated confidence well above 10-8, the lottery winner now has incommunicable good reason to believe they are in a holodeck. (I.e. to believe that the universe is such that most conscious observers observe ridiculously improbable positive events.)

It's a sad situation to be in - but don't worry: it will always happen to someone else, not you.

Original with comments: Real-Life Anthropic Weirdness

Newcomb's Problem standard positions

Eliezer Yudkowsky, 06 April 2009 05:05PM

Marion Ledwig's dissertation summarizes much of the existing thinking that's gone into Newcomb's Problem.

(For the record, I myself am neither an evidential decision theorist, nor a causal decision theorist in the current sense. My view is not easily summarized, but it is reflectively consistent without need of precommitment or similar dodges; my agents see no need to modify their own source code or invoke abnormal decision procedures on Newcomblike problems.)

Original with comments: Newcomb's Problem standard positions

Extenuating Circumstances

Eliezer Yudkowsky, 06 April 2009 10:57PM

Followup to: Tsuyoku Naritai

"Just remember, there but for a massive genetic difference, environmental factors, and conscious choices, go you or I." -- Justin Corwin

Failures don't have single causes. We choose single causes to focus on, but nothing in the universe emerges from a single parent event. Every assassination ever committed is the fault of every asteroid that wasn't in the right place to hit the assassin.

What good, then, does it do to blame circumstances for your failure? What good does it do? - to look over a huge causal lattice in which your own decisions played a part, and point to something you can't control, and say: "There is where it failed." It might be that a surgical intervention on the past, altering some node outside yourself, would have let you succeed instead of fail. But what good does this counterfactual do you? Will you choose that outside reality be different on your next try?

And yet... when I look at other people, not myself, I find myself taking "extenuating circumstances" into account a great deal. I go to great lengths to "save the world" (as I believe from my epistemic vantage point). When I consider doing less, I consider that this would make me a horrible awful unforgivable person. And then I cheerfully shake hands with others who aren't trying at all to save the world. I seem to want to have my cake and eat it too - to instantiate Goetz's Paradox: "Society tells you to work to make yourself more valuable. Then it tells you that when you reason morally, you must assume that all lives are equally valuable. You can't have it both ways."

Is this an inherent subjective asymmetry - does morality just look different from the outside than inside? If so, is that okay, or is it a sign of self-contradiction? Or is it condescension on my part - that I think less of others and so hold them to lower standards?

I've pondered this question for a while, and this is the main defense I can offer against the charge of condescension:

I wouldn't tell others to take into account "extenuating circumstances" in judging themselves.

Indeed, that would feel like an act of sabotage - like slashing their tires. Too much of life consists of holding ourselves to a high enough standard.

There are people who blame themselves too easily - people depressed, falling into despair and not moving forward, because they blame themselves for things they couldn't help.

But you really want to be very careful with applying this kind of reasoning to yourself, because a whole whack of a lot of people who were successful in life got there by driving straight through problems that couldn't be helped. I'm minded of a recent comment on Hacker News (not sure where) about someone who wanted to work at a certain game company, only there were no jobs available and no H.R. contact listed... so they looked at the numbers listed and deduced the corporate phone prefix, then systematically dialed telephone numbers until they got the CEO's office, and then pled their case to be hired. They did not, in fact, get that job; but, not surprisingly, did eventually end up employed in their chosen industry. Contrast to someone who reasons, "I won't be able to get a job now - there's a recession!"

For yea, I have watched some people be stopped in their tracks without trying by "obstacles" that other people I know, Silicon Valley entrepreneur types, would roll over like a steamroller flattening out a speedbump. That difference probably accounts for a lot of real life performance, and it probably has a great deal to do with what, exactly, you regard as a valid excuse - a condition that makes a failure not reflect badly on you.

A lot of people would regard being 14 years old as a valid excuse for not starting your own company. Not Ben Casnocha, though.

If someone has advanced to the point of explicitly pleading some excuse, then that's probably the point at which I do begin to hold them accountable. When someone says to me, "I haven't signed up my teenage son for cryonics because I'm religious so I must not believe in that sort of thing," I think, Well, I can't blame them, they don't know anything about rationality. But if they say, "I haven't signed up my teenage son for cryonics, because I don't know anything about rationality, so you can't expect me to give the correct answer to this dilemma," then at that point I really might start blaming them.

The way a real extenuating circumstance looks from the inside is not that you think "I have an extenuating circumstance, so I can be excused for failing to do X", but rather, that X just doesn't seem like an available option at all, or X seems like it would have so many penalties attached that it's not in fact the best option which you could and should perform but aren't performing.

Similarly, if ignorance is your extenuating circumstance, you just don't realize that X is a good idea, rather than thinking to yourself, "I am ignorant of the fact that X is a good idea, therefore I can be excused for failing to choose it."

Like "to believe falsely", if there were a verb meaning "to forgive due to extenuating circumstances", it would have no first-person, present-tense indicative.

So I would advise others, like myself, not to think in terms of "extenuating circumstances" at all; I would advise people to hold themselves accountable for every dilemma they have advanced to the point of explicitly perceiving as a dilemma - the same rule I use internally. This is my defense against the charge of condescension / Goetz's Paradox - that at this sufficiently meta level, I would tell others to use the same rule as I.

If I hold myself responsible for doing certain things, it is because I perceive them as morally-good, prosocially-obligatory options... which I may nonetheless have some difficulty in doing... but which I nonetheless could realistically do without overspending my mental energy budget.

That's my own main limit, incidentally, my mental energy budget. I am constantly wrestling with the fact of its reality, because it sounds like such a hideously wonderful excuse that my reflexes keep on doubting it. On any given occasion I'm never sure if it's a valid justification. And so when I do run up against my limits hard enough that there's no mistaking them, there's a certain guilty pleasure of validation, that I can feel less guilty about having done less on other occasions...

Well, that's my own demon to wrestle with. My intended point here is that I could be wrong about my mental energy budget. I could be doing too little, relative to what I can do without overspending myself. But I couldn't possibly say-in-the-moment, "I'm failing to choose the right option, but as an extenuating circumstance, I underestimated my own mental energy." If I am wrong, some forgiving other might look upon it as an "extenuating circumstance". But I cannot myself say, "And if I'm wrong, then that's an extenuating circumstance." From my own perspective, rather, I am obliged to not be wrong.

"There are no outs. Even if someone else would call it an extenuating circumstance and forgive me for giving up, I'll just get it done anyway." I'm not a venture capitalist, but if I were, that's the attitude I'd want to see in a startup founder before I invested money.

If after all that the project failed anyway, then I might really believe that everything that could be done, had been done...

...so long as it wasn't my project, which is supposed by golly to succeed and not fail due to "extenuating circumstances".

A true rationalist should win, after all.

Referenced by: Whining-Based Communities • The Craft and the Community • The End (of Sequences)

Original with comments: Extenuating Circumstances

Of Lies and Black Swan Blowups

Eliezer Yudkowsky, 07 April 2009 06:26PM

Followup to: Entangled Truths, Contagious Lies

Judge Marcus Einfeld, age 70, Queens Counsel since 1977, Australian Living Treasure 1997, United Nations Peace Award 2002, founding president of Australia's Human Rights and Equal Opportunities Commission, retired a few years back but routinely brought back to judge important cases...

...is going to jail for at least two years over a series of perjuries and lies that started with a £36, 6mph-over speeding ticket.

That whole suspiciously virtuous-sounding theory about honest people not being good at lying, and entangled traces being left somewhere, and the entire thing blowing up in a Black Swan epic fail, actually does have a certain number of exemplars in real life, though obvious selective reporting is at work in our hearing about this one.

Sequence: Against Rationalization

Entangled Truths, Contagious Lies

Dark Side Epistemology

Original with comments: Of Lies and Black Swan Blowups

Whining-Based Communities

Eliezer Yudkowsky, 07 April 2009 08:31PM

Previously in series: Selecting Rationalist Groups
Followup to: Rationality is Systematized Winning, Extenuating Circumstances

Why emphasize the connection between rationality and winning? Well... that is what decision theory is for. But also to place a Go stone to block becoming a whining-based community.

Let's be fair to Ayn Rand: There were legitimate messages in Atlas Shrugged that many readers had never heard before, and this lent the book a part of its compelling power over them. The message that it's all right to excel - that it's okay to be, not just good, but better than others - of this the Competitive Conspiracy would approve.

But this is only part of Rand's message, and the other part is the poison pill, a deadlier appeal: It's those looters who don't approve of excellence who are keeping you down. Surely you would be rich and famous and high-status like you deserve if not for them, those unappreciative bastards and their conspiracy of mediocrity.

If you consider the reasonableness-based conception of rationality rather than the winning-based conception of rationality - well, you can easily imagine some community of people congratulating themselves on how reasonable they were, while blaming the surrounding unreasonable society for keeping them down. Wrapping themselves up in their own bitterness for reality refusing to comply with the greatness they thought they should have.

But this is not how decision theory works - the "rational" strategy adapts to the other players' strategies, it does not depend on the other players being rational. If a rational agent believes the other players are irrational then it takes that expectation into account in maximizing expected utility. Van Vogt got this one right: his rationalist protagonists are formidable from accepting reality swiftly and adapting to it swiftly, without reluctance or attachment.

Self-handicapping (hat-tip Yvain) is when people who have been made aware of their own incompetence or probable future failure, deliberately impose handicaps on themselves - on the standard model, in order to give themselves an excuse for failure. To make sure they had an excuse, subjects reduced preparation times for athletic events, studied less, exerted less effort, gave opponents an advantage, lowered their own expectations, even took a drug they had been told was performance-inhibiting...

So you can see how much people value having an excuse - how much they'll pay to make sure they have something outside themselves to blame, in case of failure. And this is a need which many belief systems fill - they provide an excuse.

It's the government's fault, that taxes you and suppresses the economy - if it weren't for that, you would be a great entrepreneur. It's the fault of those less competent who envy your excellence and slander you - if not for that, the whole world would pilgrimage to admire you. It's racism, or sexism, that keeps you down - if it weren't for that, you would have gotten so much further in life with the same effort. Your rival Bob got the promotion by bootlicking. Those you call sinners may be much wealthier than you, but that's because God set up the system to reward the good deeds of the wicked in this world and punish them for their sins in the next, vice versa for the virtuous: "A boor cannot know, nor can a fool understand this: when the wicked bloom like grass and all the doers of iniquity blossom - it is to destroy them till eternity."

And maybe it's all true. The government does impose taxes and barriers to new businesses. There is racism and sexism. Scientists don't run out and embrace new ideas without huge amounts of work to evangelize them. Loyalty is a huge factor in promotions and flattery does signify loyalty. I can't back religions on that divine plan thing, but still, those wealthier than you may have gotten there by means more vile than you care to use...

And so what? In other countries there are those with far greater obstacles and less opportunity than you. There are those born with Down's Syndrome. There's not a one of us in this world, even the luckiest, whose path is entirely straight and without obstacles. In this unfair world, the test of your existence is how well you do in this unfair world.

I earlier suggested that we view our parents and environment and genes as having determined which person makes a decision - plucking you out of Platonic person-space to agonize in front of the burning orphanage, rather than someone else - but you determine what that particular person decides. If, counterfactually, your genes or environment had been different, then it would not so much change your decision as determine that someone else would make that decision.

In the same sense, I would suggest that a baby with your genes, born into a universe entirely fair, would by now be such a different person that as to be nowhere close to "you", your point in Platonic person-space. You are defined by the particular unfair challenges that you face; and the test of your existence is how well you do with them.

And in that unfair challenge, the art of rationality (if you can find it) is there to help you deal with the horrible unfair challenge and by golly win anyway, not to provide fellow bitter losers to hang out with. Even if the government does tax you and people do slander you and racists do discriminate against you and others smarm their way to success while you keep your ethics... still, this whole business of rationality is there to help you win anyway, if you can find the art you need. Find the art together, win together, if we can. And if we can't win, it means we weren't such good rationalists as we thought, and ought to try something different the next time around. (If it's one of those challenges where you get more than one try.)

From within that project - what good does a sense of violated entitlement do? At all? Ever? What good does it do to tell ourselves that we did everything right and deserved better, and that someone or something else is to blame? Is that the key thing we need to change, to do better next time?

Immediate adaptation to the realities of the situation! Followed by winning!

That is how I would cast down the gauntlet, just to make really, really sure we don't go down the utterly, completely, pointlessly unhelpful, surprisingly common path of mutual bitterness and consolation.

Sequence: The Craft and the Community

Incremental Progress and the Valley

Mandatory Secret Identities

Referenced by: Mandatory Secret Identities • The Craft and the Community

Original with comments: Whining-Based Communities

Zero-based karma coming through

Eliezer Yudkowsky, 08 April 2009 01:19AM

Our friends at Tricycle will push through a zero-based karma system (no self-voting possible) sometime this evening. At present this will only cover future posts/comments - they may go back and revise history some time in the indefinite future (or not), but apparently that would be overly complicated for now. We'll see how this works.

Original with comments: Zero-based karma coming through

Mandatory Secret Identities

Eliezer Yudkowsky, 08 April 2009 06:10PM

Previously in series: Whining-Based Communities

"But there is a reason why many of my students have achieved great things; and by that I do not mean high rank in the Bayesian Conspiracy. I expected much of them, and they came to expect much of themselves." -- Jeffreyssai

Among the failure modes of martial arts dojos, I suspect, is that a sufficiently dedicated martial arts student, will dream of...

...becoming a teacher and having their own martial arts dojo someday.

To see what's wrong with this, imagine going to a class on literary criticism, falling in love with it, and dreaming of someday becoming a famous literary critic just like your professor, but never actually writing anything. Writers tend to look down on literary critics' understanding of the art form itself, for just this reason. (Orson Scott Card uses the analogy of a wine critic who listens to a wine-taster saying "This wine has a great bouquet", and goes off to tell their students "You've got to make sure your wine has a great bouquet". When the student asks, "How? Does it have anything to do with grapes?" the critic replies disdainfully, "That's for grape-growers! I teach wine.")

Similarly, I propose, no student of rationality should study with the purpose of becoming a rationality instructor in turn. You do that on Sundays, or full-time after you retire.

And to place a go stone blocking this failure mode, I propose a requirement that all rationality instructors must have secret identities. They must have a life outside the Bayesian Conspiracy, which would be worthy of respect even if they were not rationality instructors. And to enforce this, I suggest the rule:

Rationality_Respect_1(Instructor) = min(Rationality_Respect_0(Instructor), Non_Rationality_Respect_0(Instructor))

That is, you can't respect someone as a rationality instructor, more than you would respect them if they were not rationality instructors.

Some notes:

• This doesn't set Rationality_Respect_1equal to Non_Rationality_Respect_0. It establishes an upper bound. This doesn't mean you can find random awesome people and expect them to be able to teach you. Explicit, abstract, cross-domain understanding of rationality and the ability to teach it to others is, unfortunately, an additional discipline on top of domain-specific life success. Newton was a Christian etcetera. I'd rather hear what Laplace had to say about rationality - Laplace wasn't as famous as Newton, but Laplace was a great mathematician, physicist, and astronomer in his own right, and he was the one said "I have no need of that hypothesis" (when Napoleon asked why Laplace's works on celestial mechanics did not mention God). So I would respect Laplace as a rationality instructor well above Newton, by the min() function given above.

• We should be generous about what counts as a secret identity outside the Bayesian Conspiracy. If it's something that outsiders do in fact see as impressive, then it's "outside" regardless of how much Bayesian content is in the job. An experimental psychologist who writes good papers on heuristics and biases, a successful trader who uses Bayesian algorithms, a well-selling author of a general-audiences popular book on atheism - all of these have worthy secret identities. None of this contradicts the spirit of being good at something besides rationality - no, not even the last, because writing books that sell is a further difficult skill! At the same time, you don't want to be too lax and start respecting the instructor's ability to put up probability-theory equations on the blackboard - it has to be visibly outside the walls of the dojo and nothing that could be systematized within the Conspiracy as a token requirement.

• Apart from this, I shall not try to specify what exactly is worthy of respect. A creative mind may have good reason to depart from any criterion I care to describe. I'll just stick with the idea that "Nice rationality instructor" should be bounded above by "Nice secret identity".

• But if the Bayesian Conspiracy is ever to populate itself with instructors, this criterion should not be too strict. A simple test to see whether you live inside an elite bubble is to ask yourself whether the percentage of PhD-bearers in your apparent world exceeds the 0.25% rate at which they are found in the general population. Being a math professor at a small university who has published a few original proofs, or a successful day trader who retired after five years to become an organic farmer, or a serial entrepreneur who lived through three failed startups before going back to a more ordinary job as a senior programmer - that's nothing to sneeze at. The vast majority of people go through their whole lives without being that interesting. Any of these three would have some tales to tell of real-world use, on Sundays at the small rationality dojo where they were instructors. What I'm trying to say here is: don't demand that everyone be Robin Hanson in their secret identity, that is setting the bar too high. Selective reporting makes it seem that fantastically high-achieving people have a far higher relative frequency than their real occurrence. So if you ask for your rationality instructor to be as interesting as the sort of people you read about in the newspapers - and a master rationalist on top of that - and a good teacher on top of that - then you're going to have to join one of three famous dojos in New York, or something. But you don't want to be too lax and start respecting things that others wouldn't respect if they weren't specially looking for reasons to praise the instructor. "Having a good secret identity" should require way more effort than anything that could become a token requirement.

Now I put to you: If the instructors all have real-world anecdotes to tell of using their knowledge, and all of the students know that the desirable career path can't just be to become a rationality instructor, doesn't that sound healthier?

Sequence: The Craft and the Community

Whining-Based Communities

Beware of Other-Optimizing

Referenced by: Beware of Other-Optimizing • Collective Apathy and the Internet • Go Forth and Create the Art! • The Craft and the Community

Original with comments: Mandatory Secret Identities

Beware of Other-Optimizing

Eliezer Yudkowsky, 10 April 2009 01:58AM

Previously in series: Mandatory Secret Identities

I've noticed a serious problem in which aspiring rationalists vastly overestimate their ability to optimize other people's lives. And I think I have some idea of how the problem arises.

You read nineteen different webpages advising you about personal improvement - productivity, dieting, saving money. And the writers all sound bright and enthusiastic about Their Method, they tell tales of how it worked for them and promise amazing results...

But most of the advice rings so false as to not even seem worth considering. So you sigh, mournfully pondering the wild, childish enthusiasm that people can seem to work up for just about anything, no matter how silly. Pieces of advice #4 and #15 sound interesting, and you try them, but... they don't... quite... well, it fails miserably. The advice was wrong, or you couldn't do it, and either way you're not any better off.

And then you read the twentieth piece of advice - or even more, you discover a twentieth method that wasn't in any of the pages - and STARS ABOVE IT ACTUALLY WORKS THIS TIME.

At long, long last you have discovered the real way, the right way, the way that actually works. And when someone else gets into the sort of trouble you used to have - well, this time you know how to help them. You can save them all the trouble of reading through nineteen useless pieces of advice and skip directly to the correct answer. As an aspiring rationalist you've already learned that most people don't listen, and you usually don't bother - but this person is a friend, someone you know, someone you trust and respect to listen.

And so you put a comradely hand on their shoulder, look them straight in the eyes, and tell them how to do it.

I, personally, get quite a lot of this. Because you see... when you've discovered the way that really works... well, you know better by now than to run out and tell your friends and family. But you've got to try telling Eliezer Yudkowsky. He needs it, and there's a pretty good chance that he'll understand.

It actually did take me a while to understand. One of the critical events was when someone on the Board of the Institute Which May Not Be Named, told me that I didn't need a salary increase to keep up with inflation - because I could be spending substantially less money on food if I used an online coupon service. And I believed this, because it was a friend I trusted, and it was delivered in a tone of such confidence. So my girlfriend started trying to use the service, and a couple of weeks later she gave up.

Now here's the the thing: if I'd run across exactly the same advice about using coupons on some blog somewhere, I probably wouldn't even have paid much attention, just read it and moved on. Even if it were written by Scott Aaronson or some similar person known to be intelligent, I still would have read it and moved on. But because it was delivered to me personally, by a friend who I knew, my brain processed it differently - as though I were being told the secret; and that indeed is the tone in which it was told to me. And it was something of a delayed reaction to realize that I'd simply been told, as personal advice, what otherwise would have been just a blog post somewhere; no more and no less likely to work for me, than a productivity blog post written by any other intelligent person.

And because I have encountered a great many people trying to optimize me, I can attest that the advice I get is as wide-ranging as the productivity blogosphere. But others don't see this plethora of productivity advice as indicating that people are diverse in which advice works for them. Instead they see a lot of obviously wrong poor advice. And then they finally discover the right way - the way that works, unlike all those other blog posts that don't work - and then they want to use it to optimize Eliezer Yudkowsky too, because it's important.

Don't get me wrong. Sometimes the advice is helpful. Sometimes it works. "Stuck In The Middle With Bruce" - that resonated, for me. It may prove to be the most helpful thing I've read on Less Wrong so far, though that has yet to be determined.

It's just that your earnest personal advice, that amazing thing you've found to actually work by golly, is no more and no less likely to work for me than a random personal improvement blog post written by an intelligent author is likely to work for you.

"Different things work for different people." That sentence may give you a squicky feeling; I know it gives me one. Because this sentence is a tool wielded by Dark Side Epistemology to shield from criticism, used in a way closely akin to "Different things are true for different people" (which is simply false).

But until you grasp the laws that are near-universal generalizations, sometimes you do end up messing around with surface tricks that work for one person and not another, without your understanding why, because you don't know the general laws that would dictate what works for who. And the best you can do is remember that, and be willing to take "No" for an answer.

You especially had better be willing to take "No" for an answer, if you have power over the Other. Power is, in general, a very dangerous thing, which is tremendously easy to abuse, without your being aware that you're abusing it. Oh, yes, even if you consider yourself a rationalist; there are things you can do to prevent yourself from abusing power, but you have to do them. There was a post on OB on how being in a position of power has been shown to decrease our ability to empathize with and understand the other, though I can't seem to locate it now. I have seen a rationalist who did not think he had power, and so did not think he needed to be cautious, who was amazed to learn that he might be feared...

It's even worse when their discovery that works for them, requires a little willpower. Then if you say it doesn't work for you, the answer is clear and obvious: you're just being lazy, and they need to exert some pressure on you to get you to do the correct thing, the advice they've found that actually works.

Sometimes - I suppose - people are being lazy. But be very, very, very careful before you assume that's the case and wield power over others to "get them moving". Bosses who can tell when something actually is in your capacity if you're a little more motivated, without it burning you out or making your life incredibly painful - these are the bosses who are a pleasure to work under. That ability is extremely rare, and the bosses who have it are worth their weight in silver. It's a high-level interpersonal technique that most people do not have. I surely don't have it. Do not assume you have it, because your intentions are good. Do not assume you have it, because you'd never do anything to others that you didn't want done to yourself. Do not assume you have it, because no one has ever complained to you. Maybe they're just scared. That rationalist of whom I spoke - who did not think he held power and threat, though it was certainly obvious enough to me - he did not realize that anyone could be scared of him.

Be careful even when you hold leverage, when you hold an important decision in your hand, or a threat, or something that the other person needs, and all of a sudden the temptation to optimize them seems overwhelming.

Consider, if you would, that Ayn Rand's whole reign of terror over Objectivists can be seen in just this light - that she found herself with power and leverage, and could not resist the temptation to optimize.

We underestimate the distance between ourselves and others. Not just inferential distance, but distances of temperament and ability, distances of situation and resource, distances of unspoken knowledge and unnoticed skills and luck, distances of interior landscape.

Even I am often surprised to find that X, which worked so well for me, doesn't work for someone else. But with so many others having tried to optimize me, I can at least recognize distance when I'm hit over the head with it.

Maybe being pushed on does work... for you. Maybe you don't get sick to the stomach when someone with power over you starts helpfully trying to reorganize your life the correct way. I don't know, I don't know what makes you tick. In the realm of willpower and akrasia and productivity, as in other realms, I don't know the generalizations deep enough to hold almost always. I don't possess the deep keys that would tell me when and why and for who a technique works or doesn't work. All I can do is be willing to accept it, when someone tells me it doesn't work... and go on looking for the deeper generalizations that will hold everywhere, the deeper laws governing both the rule and the exception, waiting to be found, someday.

Sequence: The Craft and the Community

Mandatory Secret Identities

Akrasia and Shangri-La

Referenced by: The Unfinished Mystery of the Shangri-La Diet • Akrasia and Shangri-La • Collective Apathy and the Internet • My Way • Go Forth and Create the Art! • The Craft and the Community

Original with comments: Beware of Other-Optimizing

That Crisis thing seems pretty useful

Eliezer Yudkowsky, 10 April 2009 05:10PM

Since there's been much questioning of late over "What good is advanced rationality in the real world?", I'd like to remind everyone that it isn't all about post-doctoral-level reductionism.

In particular, as a technique that seems like it ought to be useful in the real world, I exhibit the highly advanced, difficult, multi-component Crisis of Faith aka Reacting To The Damn Evidence aka Actually Changing Your Mind.

Scanning through this post and the list of sub-posts at the bottom (EDIT: copied to below the fold) should certainly qualify it as "extreme rationality" or "advanced rationality" or "x-rationality" or "Bayescraft" or whatever you want to distinguish from "traditional rationality as passed down from Richard Feynman".

An actual sit-down-for-an-hour Crisis of Faith might be something you'd only use once or twice in every year or two, but on important occasions. And the components are often things that you could practice day in and day out, also to positive effect.

I think this is the strongest foot that I could put forward for "real-world" uses of my essays. (Anyone care to nominate an alternative?)

Below the fold, I copy and paste the list of components from the original post, so that we have them at hand:

- Avoiding Your Belief's Real Weak Points - One of the first temptations in a crisis of faith is to doubt the strongest points of your belief, so that you can rehearse your good answers. You need to seek out the most painful spots, not the arguments that are most reassuring to consider.

- The Meditation on Curiosity - Roger Zelazny once distinguished between "wanting to be an author" versus "wanting to write", and there is likewise a distinction between wanting to have investigated and wanting to investigate. It is not enough to say "It is my duty to criticize my own beliefs"; you must be curious, and only uncertainty can create curiosity. Keeping in mind Conservation of Expected Evidence may help you Update Yourself Incrementally: For every single point that you consider, and each element of new argument and new evidence, you should not expect your beliefs to shift more (on average) in one direction than another - thus you can be truly curious each time about how it will go.

- Cached Thoughts and Pirsig's Original Seeing, to prevent standard thoughts from rushing in and completing the pattern.

- The Litany of Gendlin and the Litany of Tarski: People can stand what is true, for they are already enduring it. If a belief is true you will be better off believing it, and if it is false you will be better off rejecting it. You would advise a religious person to try to visualize fully and deeply the world in which there is no God, and to, without excuses, come to the full understanding that if there is no God then they will be better off believing there is no God. If one cannot come to accept this on a deep emotional level, they will not be able to have a crisis of faith. So you should put in a sincere effort to visualize the alternative to your belief, the way that the best and highest skeptic would want you to visualize it. Think of the effort a religionist would have to put forth to imagine, without corrupting it for their own comfort, an atheist's view of the universe.

- Make an Extraordinary Effort, for the concept of isshokenmei, the desperate convulsive effort to be rational that it would take to surpass the level of Robert Aumann and all the great scientists throughout history who never let go of their religions.

- The Genetic Heuristic: You should be extremely suspicious if you have many ideas suggested by a source that you now know to be untrustworthy, but by golly, it seems that all the ideas still ended up being right. (E.g., the one concedes that the Bible was written by human hands, but still clings to the idea that it contains indispensable ethical wisdom.)

- The Importance of Saying "Oops" - it really is less painful to swallow the entire bitter pill in one terrible gulp.

- Singlethink, the opposite of doublethink. See the thoughts you flinch away from, that appear in the corner of your mind for just a moment before you refuse to think them. If you become aware of what you are not thinking, you can think it.

- Affective Death Spirals and Resist the Happy Death Spiral. Affective death spirals are prime generators of false beliefs that it will take a Crisis of Faith to shake loose. But since affective death spirals can also get started around real things that are genuinely nice, you don't have to admit that your belief is a lie, to try and resist the halo effect at every point - refuse false praise even of genuinely nice things. Policy debates should not appear one-sided.

- Hold Off On Proposing Solutions until the problem has been discussed as thoroughly as possible without proposing any; make your mind hold off from knowing what its answer will be; and try for five minutes before giving up, both generally, and especially when pursuing the devil's point of view.

- The sequence on The Bottom Line and Rationalization, which explains why it is always wrong to selectively argue one side of a debate.

- Positive Bias and motivated skepticism and motivated stopping, lest you selectively look for support, selectively look for counter-counterarguments, and selectively stop the argument before it gets dangerous. Missing alternatives are a special case of stopping. A special case of motivated skepticism is fake humility where you bashfully confess that no one can know something you would rather not know. Don't selectively demand too much authority of counterarguments.

- Beware of Semantic Stopsigns, Applause Lights, and the choice to Explain/Worship/Ignore.

- Feel the weight of Burdensome Details; each detail a separate burden, a point of crisis.

Original with comments: That Crisis thing seems pretty useful

The Unfinished Mystery of the Shangri-La Diet

Eliezer Yudkowsky, 10 April 2009 08:30PM

Followup to: Beware of Other-Optimizing

Once upon a time, Seth Roberts (a professor of psychology at Berkeley, on the editorial board of Nutrition) noticed that he'd started losing weight while on vacation in Europe. For no apparent reason, he'd stopped wanting to eat.

Some time later, The Shangri-La Diet swept... the econoblogosphere, anyway. People including some respectable economists tried it, found that it actually seemed to work, and told their friends.

The Shangri-La Diet is unfortunately named - I would have called it "the set-point diet". And even worse, the actual procedure sounds like the wackiest fad diet imaginable:

Just drink two tablespoons of extra-light olive oil early in the morning... don't eat anything else for at least an hour afterward... and in a few days it will no longer take willpower to eat less; you'll feel so full all the time, you'll have to remind yourself to eat.

Why? I'm tempted to say "No one knows" just to see what kind of comments would show up, but that would be cheating. Roberts does have a theory motivating the diet, an elegant combination of pieces individually backed by previous experiments:

- Your metabolism has a set point, like the setting on a thermostat: when your weight is below the set point, you feel hungry; when your weight is above the set point, you feel full.

- But the set point is not a constant; it is raised and lowered by what you eat.

- This mechanism in turn seems to be regulated by a flavor-calorie association. (Possibly as a famine-storage mechanism that tries to store more resources when dense food sources are available.) If you eat something with flavor X, which is followed by your metabolism detecting a large source of calories, flavor X will (a) seem more appealing and taste better, and (b) will raise your set point whenever you eat items with flavor X.

- Your set point is always naturally dropping, but is raised by eating; usually these forces are in dynamic balance and your weight stays constant.

I'm not going to go into all the existing evidence that backs up each step of this theory, but the theory is very beautiful and elegant. The actual Shangri-La Diet is painfully simple by comparison: consume nearly tasteless extra-light olive oil, being careful not to associate it with any flavors before or after, to raise your body weight a little without raising your set point. Your body weight goes above your set point, and you stop feeling hungry. Then you eat less... and your weight drops... and your set point drops a little less than that... but then next morning it's time for your next dose of extra-light olive oil, which once again puts your (decreased) weight a bit above the set point. The regular dose of almost flavorless calories tilts the dynamic balance downward. That's the theory.

Many people, including some trustworthy econblogger types, have reported losing 1-2 pounds/week by implementing the actual actions of the Shangri-La Diet, up to 30 pounds or even more in some cases. Without expending willpower.

I tried it. It didn't work for me.

Now here's the frustrating thing: The Shangri-La Diet does not contain an obvious exception for Eliezer Yudkowsky. On the theory as stated, it should just work. But I am not the only person who reports trying this diet (and a couple of variations that Roberts recommended) without anything happening, except possibly some weight gain due to the added calories.

And here's the more frustrating thing: Roberts's explanation felt right. It's one of those insights that you grasp and then so much else snaps into place.

It explained that frustrating experience I'd often had, wherein I would try a new food and it would fill me up for a whole day - and then, as I kept on eating this amazing food in an effort to keep my total intake down, the satiation effect would go away.

It explained why I'd lost on the order of 50-60 pounds - with what, in retrospect, was very little effort - when I first moved out of my parents' house and to a new city and started eating non-Jewish food. In retrospect, I was eating an amazingly little amount each day, like 1200 calories, but without any feeling of hunger. And then my weight slowly started creeping up again, and no amount of exercise - to which (ha!) I'd originally attributed the weight loss - seemed able to stop it.

It's always hard to pick reality out of the gigantic morass of competing dietary theories. One of the elegant charms of Robert's hypothesis is that it helps explain why this is so - the mess of incoherent results. Any new diet will seem to work for a few months or weeks, you're losing weight and everything seems wonderful, you tell all your friends and they buy the same diet book, and then bam the flavor-calorie association kicks back in and you're back to hell. The number-one result of weight-loss science is that 95% of people who lose weight regain it.

(I haven't heard any complaints from people regaining weight they lost on the Shangri-La Diet, however - if it works for you at all, it seems to go on working. Most of the complaints on the forums are from people who suddenly plateau after losing 30 pounds, but who want to lose more. Or people like me, who try it, and find that it doesn't seem to do anything, or that we're gaining weight with no apparent loss of appetite.)

I have a pretty strong feeling - I don't know if I should trust it, since I'm not a dietary scientist - that Roberts's hypothesis is at least partially right. It makes a lot of data snap into focus. The pieces are well-supported individually.

But I don't think that Roberts has the whole story. There's something missing - something that would explain why the Shangri-La Diet lets some people control their weight as easily as a thermostat setting, and why others lose 30 pounds and then plateau well short of their goal, and why others simply find the Shangri-La diet ineffective. The Mystery of Shangri-La is not how the diet works when it does work; Roberts has made an excellent case for that. The question is why it sometimes doesn't work. There is a deeper law, I strongly suspect, that governs both the rule and the exception.

The problem is, though - and here's the really frustrating part - Roberts seems to think he does have the whole answer. If the diet doesn't work at first, his answer is to try more oil... which is a pretty scary answer if you're already gaining weight from the extra calorie intake! I decided not to go down this route because it didn't seem to work for the people on the forums who were reporting that the Shangri-La Diet didn't work for them. They just gained even more weight.

And what really makes this a catastrophe is that this theory has never been analyzed by controlled experiment, which drives me up the frickin' WALL. Roberts himself is a big advocate of "self-experimentation", which I suppose explains why he's not pushing harder for testing. (Though it's not like Roberts is a standard pseudoscientist, he's an academic in good standing.) But with reports of such drastic success from so many observers, some of them reliable, outside dietary scientists ought to be studying this. What the fsck, dietary scientists? Get off your butts and study this thing! NOW! Report these huge results in a peer-reviewed journal so that everyone gets excited and starts studying the exceptions to the rule!

It's awful; it seems like Roberts has gotten so close to burying the scourge Obesity, but the theory is still missing some final element, some completing piece that would explain the rule and the exception, and with that last piece it might be possible to make the diet work for everyone...

If we had a large-sized rationalist community going that had solved the group effort coordination problem, those of us who are metabolically disprivileged would be pooling resources and launching our own controlled study of this thing, and entering every conceivable variable we could report into the matrix, and hiring a professional biochemist to analyze our metabolisms before and afterward, and we would cryopreserve anyone who got in our way. You have no idea.

(Warning: Do not try the Shangri-La diet at home based on only the info here, there's a couple of caveats and I can't think offhand of a good complete description on the 'Net. Also you might want to reconsider the recommendation to use fructose in the sugar water route, because IIRC fructose has been shown to contribute to insulin resistance or something like that - sucrose may actually make more sense, despite the higher glycemic index.)

Continued in: Akrasia and Shangri-La

Referenced by: Akrasia and Shangri-La • Practical Advice Backed By Deep Theories • The Craft and the Community

Original with comments: The Unfinished Mystery of the Shangri-La Diet

Akrasia and Shangri-La

Eliezer Yudkowsky, 10 April 2009 08:53PM

Continuation of: The Unfinished Mystery of the Shangri-La Diet

My post about the Shangri-La Diet is there to make a point about akrasia. It's not just an excuse: people really are different and what works for one person sometimes doesn't work for another.

You can never be sure in the realm of the mind... but out in material foodland, I know that I was, in fact, drinking extra-light olive oil in the fashion prescribed. There is no reason within Roberts's theory why it shouldn't have worked.

Which just means Roberts's theory is incomplete. In the complicated mess that is the human metabolism there is something else that needs to be considered. (My guess would be "something to do with insulin".)

But if the actions needed to implement the Shangri-La Diet weren't so simple and verifiable... if some of them took place within the mind... if it took, not a metabolic trick, but willpower to get to that amazing state where dieting comes effortlessly and you can lose 30 pounds...

Then when the Shangri-La Diet didn't work, we unfortunate exceptions would get yelled at for doing it wrong and not having enough willpower. Roberts already seems to think that his diet ought to work for everyone; when someone says it's not working, Roberts tells them to drink more extra-light olive oil or try a slightly different variant of the diet, rather than saying, "This doesn't work for some people and I don't know why."

If the failure had occurred somewhere inside the dark recesses of my mind where it could be blamed on me, rather than within my metabolism...

If Roberts's hypothesis is correct, then I'm sure that plenty of people have made some dietary change, started losing weight due to the disrupted flavor-calorie association, and congratulated themselves on their wonderful willpower for eating less. When I moved out of my parents' home and started eating less and exercising and losing more than a pound a week, you can bet I was congratulating myself on my amazing willpower.

Hah. No, I just stumbled onto a metabolic pot of gold that let me lose a lot of weight using a sustainable expenditure of willpower. When that pot of gold was exhausted, willpower ceased to avail.

(The metabolically privileged don't believe in metabolic privilege, since they are able to lose weight by trying! harder! to diet and exercise, and the diet and exercise actually work the way they're supposed to... I remember the nine-month period in my life where that was true.)

When I look at the current state of the art in fighting akrasia, I see the same sort of mess.

People try all sorts of crazy things - and as in dieting, there's secretly a general reason why any crazy thing might seem to work: if you expect to win an internal conflict, you've already programmed yourself to do the right thing because you expect that to be your action; it takes less willpower to win an internal conflict you expect to win.

And people make up all sorts of fantastic stories to explain why their tricks worked for them.

But their tricks don't work for everyone - some others report success, some don't. The inventors do not know the deep generalizations that would tell them why and who, explain the rule and the exception. But the stories the inventors have created to explain their own successes, naturally praise their own willpower and other virtues, and contain no element of luck... and so they exhort others: Try harder! You're doing it wrong!

There is a place in the mind for willpower. Don't get me wrong, it's useful stuff. But people who assign their successes to willpower - who congratulate themselves on their stern characters - may be a tad reluctant to appreciate just how much you can be privileged or disprivileged by having a mental metabolism where expending willpower is effective, where you can achieve encouraging results, at an acceptable cost to yourself, and sustain the effort in the long run.

Sequence: The Craft and the Community

Beware of Other-Optimizing

Collective Apathy and the Internet

Referenced by: The Unfinished Mystery of the Shangri-La Diet • The Craft and the Community

Original with comments: Akrasia and Shangri-La

Twelve Virtues booklet printing?

Eliezer Yudkowsky, 11 April 2009 02:40PM

For a while now, I've been using a laser printer to print out a couple of hundred copies of the Twelve Virtues of Rationality (in its printable pamphlet version) and taking them with me to conferences, talks, and science fiction conventions. Cut, staple in the middle (using a large-sized, measuring stapler), and fold. This method is very cheap, probably something like ten cents a copy for ink and paper. But it produces crappy-looking pamphlets.

Does anyone know of a way of cheaply printing small 16-page pamphlets? Take a look at the pamphlet to see the current size. I would really like to see the pamphlets stack well so I can plump down 50 of them without the tower falling over, which is the main problem with the staple-and-fold method. But even more important is that they be cheap, considering the quantities in which I hand these out for free. Something conducive to a professional-looking cover (i.e. allowing for the top sheet to be glossy or a higher quality of paper) would also be nice, again cost permitting.

If I can find a good solution I'll also go ahead and get the pamphlet graphically redesigned before printing, of course, and include some more direct proselytizing material for Less Wrong on the back cover.

I've looked around online, but all the print shops I've seen have been way too expensive for giving away 200 copies per convention - even by myself, much less getting other people to do it on a routine basis. Does anyone know how to get this done cheaply? A minimum order of 10,000 for $1000 would be quite acceptable - I expect at that price I could ship some boxes to other LWers.

Original with comments: Twelve Virtues booklet printing?

Bystander Apathy

Eliezer Yudkowsky, 13 April 2009 01:26AM

The bystander effect, also known as bystander apathy, is that larger groups are less likely to act in emergencies - not just individually, but collectively. Put an experimental subject alone in a room and let smoke start coming up from under the door. 75% of the subjects will leave to report it. Now put three subjects in the room - real subjects, none of whom know what's going on. On only 38% of the occasions will anyone report the smoke. Put the subject with two confederates who ignore the smoke, and they'll only report it 10% on the time - even staying in the room until it becomes hazy. (Latane and Darley 1969.)

On the standard model, the two primary drivers of bystander apathy are:

- Diffusion of responsibility - everyone hopes that someone else will be first to step up and incur any costs of acting. When no one does act, being part of a crowd provides an excuse and reduces the chance of being held personally responsible for the results.

- Pluralistic ignorance - people try to appear calm while looking for cues, and see... that the others appear calm.

Cialdini (2001):

Very often an emergency is not obviously an emergency. Is the man lying in the alley a heart-attack victim or a drunk sleeping one off? ... In times of such uncertainty, the natural tendency is to look around at the actions of others for clues. We can learn from the way the other witnesses are reacting whether the event is or is not an emergency. What is easy to forget, though, is that everybody else observing the event is likely to be looking for social evidence, too. Because we all prefer to appear poised and unflustered among others, we are likely to search for that evidence placidly, with brief, camouflaged glances at those around us. Therefore everyone is likely to see everyone else looking unruffled and failing to act.

Cialdini suggests that if you're ever in emergency need of help, you point to one single bystander and ask them for help - making it very clear to whom you're referring. Remember that the total group, combined, may have less chance of helping than one individual.

I've mused a bit on the evolutionary psychology of the bystander effect. Suppose that in the ancestral environment, most people in your band were likely to be at least a little related to you - enough to be worth saving, if you were the only one who could do it. But if there are two others present, then the first person to act incurs a cost, while the other two both reap the genetic benefit of a partial relative being saved. Could there have been an arms race for who waited the longest?

As far as I've followed this line of speculation, it doesn't seem to be a good explanation - at the point where the whole group is failing to act, a gene that helps immediately ought to be able to invade, I would think. The experimental result is not a long wait before helping, but simply failure to help: if it's a genetic benefit to help when you're the only person who can do it (as does happen in the experiments) then the group equilibrium should not be no one helping (as happens in the experiments).

So I don't think an arms race of delay is a plausible evolutionary explanation. More likely, I think, is that we're looking at a nonancestral problem. If the experimental subjects actually know the apparent victim, the chances of helping go way up (i.e., we're not looking at the correlate of helping an actual fellow band member). If I recall correctly, if the experimental subjects know each other, the chances of action also go up.

Nervousness about public action may also play a role. If Robin Hanson is right about the evolutionary role of "choking", then being first to act in an emergency might also be taken as a dangerous bid for high status. (Come to think, I can't actually recall seeing shyness discussed in analyses of the bystander effect, but that's probably just my poor memory.)

Can the bystander effect be explained primarily by diffusion of moral responsibility? We could be cynical and suggest that people are mostly interested in not being blamed for not helping, rather than having any positive desire to help - that they mainly wish to escape antiheroism and possible retribution. Something like this may well be a contributor, but two observations that mitigate against it are (a) the experimental subjects did not report smoke coming in from under the door, even though it could well have represented a strictly selfish threat and (b) telling people about the bystander effect reduces the bystander effect, even though they're no more likely to be held publicly responsible thereby.

In fact, the bystander effect is one of the main cases I recall offhand where telling people about a bias actually seems able to strongly reduce it - maybe because the appropriate way to compensate is so obvious, and it's not easy to overcompensate (as when you're trying to e.g. adjust your calibration). So we should be careful not to be too cynical about the implications of the bystander effect and diffusion of responsibility, if we interpret individual action in terms of a cold, calculated attempt to avoid public censure. People seem at least to sometimes hold themselves responsible, once they realize they're the only ones who know enough about the bystander effect to be likely to act.

Though I wonder what happens if you know that you're part of a crowd where everyone has been told about the bystander effect...

Cialdini, R. (2001.) Influence: Science and Practice. Boston, MA: Allyn and Bacon.

Latane, B. and Darley, J. (1969.) Bystander "Apathy", American Scientist, 57: 244-268.

Referenced by: Collective Apathy and the Internet • The Craft and the Community • That Magical Click

Original with comments: Bystander Apathy

Collective Apathy and the Internet

Eliezer Yudkowsky, 14 April 2009 12:02AM

Previously in series: Beware of Other-Optimizing
Followup to: Bystander Apathy

Yesterday I convered the bystander effect, aka bystander apathy: given a fixed problem situation, a group of bystanders is actually less likely to act than a single bystander. The standard explanation for this result is in terms of pluralistic ignorance (if it's not clear whether the situation is an emergency, each person tries to look calm while darting their eyes at the other bystanders, and sees other people looking calm) and diffusion of responsibility (everyone hopes that someone else will be first to act; being part of a crowd diminishes the individual pressure to the point where no one acts).

Which may be a symptom of our hunter-gatherer coordination mechanisms being defeated by modern conditions. You didn't usually form task-forces with strangers back in the ancestral environment; it was mostly people you knew. And in fact, when all the subjects know each other, the bystander effect diminishes.

So I know this is an amazing and revolutionary observation, and I hope that I don't kill any readers outright from shock by saying this: but people seem to have a hard time reacting constructively to problems encountered over the Internet.

Perhaps because our innate coordination instincts are not tuned for:

- Being part of a group of strangers. (When all subjects know each other, the bystander effect diminishes.)

- Being part of a group of unknown size, of strangers of unknown identity.

- Not being in physical contact (or visual contact); not being able to exchange meaningful glances.

- Not communicating in real time.

- Not being much beholden to each other for other forms of help; not being codependent on the group you're in.

- Being shielded from reputational damage, or the fear of reputational damage, by your own apparent anonymity; no one is visibly looking at you, before whom your reputation might suffer from inaction.

- Being part of a large collective of other inactives; no one will single out you to blame.

- Not hearing a voiced plea for help.

Etcetera. I don't have a brilliant solution to this problem. But it's the sort of thing that I would wish for potential dot-com cofounders to ponder explicitly, rather than wondering how to throw sheep on Facebook. (Yes, I'm looking at you, Hacker News.) There are online activism web apps, but they tend to be along the lines of sign this petition! yay, you signed something! rather than How can we counteract the bystander effect, restore motivation, and work with native group-coordination instincts, over the Internet?

Some of the things that come to mind:

- Put a video of someone asking for help online.

- Put up names and photos or even brief videos if available of the first people who helped (or have some redditish priority algorithm that depends on a combination of amount-helped and recency).

- Give helpers a video thank-you from the founder of the cause that they can put up on their "people I've helped" page, which with enough standardization could be partially or wholly assembled automatically and easily embedded in their home webpage or Facebook account.

- Find a non-annoying idiom for "Tell a friend about cause X"; allow referrer link codes; then show people how many others they've evangelized (how many people who initially got here using referrer code X actually contributed or took some other action).

- (All of the above applies not just to donations, but to open-source projects to which people have contributed code. Or if people really do want nothing but signatures on a petition, then for signatures. There are ways to help besides money - even though money is usually the most effective. The main thing is that the form of help has to be verifiable online.)

- Make it easier for people to offer monetary bounties on subtasks whose performance is verifiable.

But mostly I just hand you an open, unsolved problem: make it possible / easier for groups of strangers to coalesce into an effective task force over the Internet, in defiance of the usual failure modes and the default reasons why this is a non-ancestral problem. Think of that old statistic about Wikipedia representing 1/2,000 of the time spent in the US alone on watching television. There's quite a lot of fuel out there, if there were only such a thing as an effective engine...

Sequence: The Craft and the Community

Akrasia and Shangri-La

Bayesians vs. Barbarians

Referenced by: Bayesians vs. Barbarians • The Craft and the Community

Original with comments: Collective Apathy and the Internet

Bayesians vs. Barbarians

Eliezer Yudkowsky, 14 April 2009 11:45PM

Previously in series: Collective Apathy and the Internet
Followup to: Helpless Individuals

Previously:

Let's say we have two groups of soldiers. In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy. In group 2, everyone at all levels knows all about tactics and strategy.

Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.

Suppose that a country of rationalists is attacked by a country of Evil Barbarians who know nothing of probability theory or decision theory.

Now there's a certain viewpoint on "rationality" or "rationalism" which would say something like this:

"Obviously, the rationalists will lose. The Barbarians believe in an afterlife where they'll be rewarded for courage; so they'll throw themselves into battle without hesitation or remorse. Thanks to their affective death spirals around their Cause and Great Leader Bob, their warriors will obey orders, and their citizens at home will produce enthusiastically and at full capacity for the war; anyone caught skimming or holding back will be burned at the stake in accordance with Barbarian tradition. They'll believe in each other's goodness and hate the enemy more strongly than any sane person would, binding themselves into a tight group. Meanwhile, the rationalists will realize that there's no conceivable reward to be had from dying in battle; they'll wish that others would fight, but not want to fight themselves. Even if they can find soldiers, their civilians won't be as cooperative: So long as any one sausage almost certainly doesn't lead to the collapse of the war effort, they'll want to keep that sausage for themselves, and so not contribute as much as they could. No matter how refined, elegant, civilized, productive, and nonviolent their culture was to start with, they won't be able to resist the Barbarian invasion; sane discussion is no match for a frothing lunatic armed with a gun. In the end, the Barbarians will win because they want to fight, they want to hurt the rationalists, they want to conquer and their whole society is united around conquest; they care about that more than any sane person would."

War is not fun. As many many people have found since the dawn of recorded history, as many many people have found out before the dawn of recorded history, as some community somewhere is finding out right now in some sad little country whose internal agonies don't even make the front pages any more.

War is not fun. Losing a war is even less fun. And it was said since the ancient times: "If thou would have peace, prepare for war." Your opponents don't have to believe that you'll win, that you'll conquer; but they have to believe you'll put up enough of a fight to make it not worth their while.

You perceive, then, that if it were genuinely the lot of "rationalists" to always lose in war, that I could not in good conscience advocate the widespread public adoption of "rationality".

This is probably the dirtiest topic I've discussed or plan to discuss on LW. War is not clean. Current high-tech militaries - by this I mean the US military - are unique in the overwhelmingly superior force they can bring to bear on opponents, which allows for a historically extraordinary degree of concern about enemy casualties and civilian casualties.

Winning in war has not always meant tossing aside all morality. Wars have been won without using torture. The unfunness of war does not imply, say, that questioning the President is unpatriotic. We're used to "war" being exploited as an excuse for bad behavior, because in recent US history that pretty much is exactly what it's been used for...

But reversed stupidity is not intelligence. And reversed evil is not intelligence either. It remains true that real wars cannot be won by refined politeness. If "rationalists" can't prepare themselves for that mental shock, the Barbarians really will win; and the "rationalists"... I don't want to say, "deserve to lose". But they will have failed that test of their society's existence.

Let me start by disposing of the idea that, in principle, ideal rational agents cannot fight a war, because each of them prefers being a civilian to being a soldier.

As has already been discussed at some length, I one-box on Newcomb's Problem.

Consistently, I do not believe that if an election is settled by 100,000 to 99,998 votes, that all of the voters were irrational in expending effort to go to the polling place because "my staying home would not have affected the outcome". (Nor do I believe that if the election came out 100,000 to 99,999, then 100,000 people were all, individually, solely responsible for the outcome.)

Consistently, I also hold that two rational AIs (that use my kind of decision theory), even if they had completely different utility functions and were designed by different creators, will cooperate on the true Prisoner's Dilemma if they have common knowledge of each other's source code. (Or even just common knowledge of each other's rationality in the appropriate sense.)

Consistently, I believe that rational agents are capable of coordinating on group projects whenever the (expected probabilistic) outcome is better than it would be without such coordination. A society of agents that use my kind of decision theory, and have common knowledge of this fact, will end up at Pareto optima instead of Nash equilibria. If all rational agents agree that they are better off fighting than surrendering, they will fight the Barbarians rather than surrender.

Imagine a community of self-modifying AIs who collectively prefer fighting to surrender, but individually prefer being a civilian to fighting. One solution is to run a lottery, unpredictable to any agent, to select warriors. Before the lottery is run, all the AIs change their code, in advance, so that if selected they will fight as a warrior in the most communally efficient possible way - even if it means calmly marching into their own death.

(A reflectively consistent decision theory works the same way, only without the self-modification.)

You reply: "But in the real, human world, agents are not perfectly rational, nor do they have common knowledge of each other's source code. Cooperation in the Prisoner's Dilemma requires certain conditions according to your decision theory (which these margins are too small to contain) and these conditions are not met in real life."

I reply: The pure, true Prisoner's Dilemma is incredibly rare in real life. In real life you usually have knock-on effects - what you do affects your reputation. In real life most people care to some degree about what happens to other people. And in real life you have an opportunity to set up incentive mechanisms.

And in real life, I do think that a community of human rationalists could manage to produce soldiers willing to die to defend the community. So long as children aren't told in school that ideal rationalists are supposed to defect against each other in the Prisoner's Dilemma. Let it be widely believed - and I do believe it, for exactly the same reason I one-box on Newcomb's Problem - that if people decided as individuals not to be soldiers or if soldiers decided to run away, then that is the same as deciding for the Barbarians to win. By that same theory whereby, if a lottery is won by 100,000 votes to 99,998 votes, it does not make sense for every voter to say "my vote made no difference". Let it be said (for it is true) that utility functions don't need to be solipsistic, and that a rational agent can fight to the death if they care enough about what they're protecting. Let them not be told that rationalists should expect to lose reasonably.

If this is the culture and the mores of the rationalist society, then, I think, ordinary human beings in that society would volunteer to be soldiers. That also seems to be built into human beings, after all. You only need to ensure that the cultural training does not get in the way.

And if I'm wrong, and that doesn't get you enough volunteers?

Then so long as people still prefer, on the whole, fighting to surrender; they have an opportunity to set up incentive mechanisms, and avert the True Prisoner's Dilemma.

You can have lotteries for who gets elected as a warrior. Sort of like the example above with AIs changing their own code. Except that if "be reflectively consistent; do that which you would precommit to do" is not sufficient motivation for humans to obey the lottery, then...

...well, in advance of the lottery actually running, we can perhaps all agree that it is a good idea to give the selectees drugs that will induce extra courage, and shoot them if they run away. Even considering that we ourselves might be selected in the lottery. Because in advance of the lottery, this is the general policy that gives us the highest expectation of survival.

...like I said: Real wars = not fun, losing wars = less fun.

Let's be clear, by the way, that I'm not endorsing the draft as practiced nowadays. Those drafts are not collective attempts by a populace to move from a Nash equilibrium to a Pareto optimum. Drafts are a tool of kings playing games in need of toy soldiers. The Vietnam draftees who fled to Canada, I hold to have been in the right. But a society that considers itself too smart for kings, does not have to be too smart to survive. Even if the Barbarian hordes are invading, and the Barbarians do practice the draft.

Will rational soldiers obey orders? What if the commanding officer makes a mistake?

Soldiers march. Everyone's feet hitting the ground in the same rhythm. Even, perhaps, against their own inclinations, since people left to themselves would walk all at separate paces. Lasers made out of people. That's marching.

If it's possible to invent some method of group decisionmaking that is superior to the captain handing down orders, then a company of rational soldiers might implement that procedure. If there is no proven method better than a captain, then a company of rational soldiers commit to obey the captain, even against their own separate inclinations. And if human beings aren't that rational... then in advance of the lottery, the general policy that gives you the highest personal expectation of survival is to shoot soldiers who disobey orders. This is not to say that those who fragged their own officers in Vietnam were in the wrong; for they could have consistently held that they preferred no one to participate in the draft lottery.

But an uncoordinated mob gets slaughtered, and so the soldiers need some way of all doing the same thing at the same time in the pursuit of the same goal, even though, left to their own devices, they might march off in all directions. The orders may not come from a captain like a superior tribal chief, but unified orders have to come from somewhere. A society whose soldiers are too clever to obey orders, is a society which is too clever to survive. Just like a society whose people are too clever to be soldiers. That is why I say "clever", which I often use as a term of opprobrium, rather than "rational".

(Though I do think it's an important question as to whether you can come up with a small-group coordination method that really genuinely in practice works better than having a leader. The more people can trust the group decision method - the more they can believe that it really is superior to people going their own way - the more coherently they can behave even in the absence of enforceable penalties for disobedience.)

I say all this, even though I certainly don't expect rationalists to take over a country any time soon, because I think that what we believe about a society of "people like us" has some reflection on what we think of ourselves. If you believe that a society of people like you would be too reasonable to survive in the long run... that's one sort of self-image. And it's a different sort of self-image if you think that a society of people all like you could fight the vicious Evil Barbarians and win - not just by dint of superior technology, but because your people care about each other and about their collective society - and because they can face the realities of war without losing themselves - and because they would calculate the group-rational thing to do and make sure it got done - and because there's nothing in the rules of probability theory or decision theory that says you can't sacrifice yourself for a cause - and because if you really are smarter than the Enemy and not just flattering yourself about that, then you should be able to exploit the blind spots that the Enemy does not allow itself to think about - and because no matter how heavily the Enemy hypes itself up before battle, you think that just maybe a coherent mind, undivided within itself, and perhaps practicing something akin to meditation or self-hypnosis, can fight as hard in practice as someone who theoretically believes they've got seventy-two virgins waiting for them.

Then you'll expect more of yourself and people like you operating in groups; and then you can see yourself as something more than a cultural dead end.

So look at it this way: Jeffreyssai probably wouldn't give up against the Evil Barbarians if he were fighting alone. A whole army of beisutsukai masters ought to be a force that no one would mess with. That's the motivating vision. The question is how, exactly, that works.

Sequence: The Craft and the Community

Collective Apathy and the Internet

Of Gender and Rationality

Referenced by: My Way • The Craft and the Community

Original with comments: Bayesians vs. Barbarians

Of Gender and Rationality

Eliezer Yudkowsky, 16 April 2009 12:56AM

Among all self-identified "rationalist" communities that I know of, and Less Wrong in particular, there is an obvious gender imbalance - a male/female ratio tilted strongly toward males.

Yet surely epistemic and instrumental rationality have no gender signature. There is no such thing as masculine probability theory or feminine decision theory.

There could be some entirely innocuous explanation for this imbalance. Perhaps, by sheer historical contingency, aspiring rationalists are recruited primarily from the atheist/libertarian/technophile cluster, which has a gender imbalance for its own reasons - having nothing to do with rationality or rationalists; and this is the entire explanation.

Uh huh. Sure.

And then there are the less innocuous explanations - those that point an accusing finger at the rationalist community, or at womankind.

If possible, let's try not to make things worse in the course of having this discussion. Remember that to name two parts of a community is to split that community - see the Robbers Cave experiment: Two labels → two groups. Let us try not to make some of our fellow rationalists feel singled-out as objects of scrutiny, here. But in the long run especially, it is not a good thing if half the potential audience is being actively filtered out; whatever the cause, the effect is noticeable, and we can't afford to ignore the question.

These are the major possibilities that I see:

(1) While the pure math of the right Way has no gender signatures on it, we can imagine that men and women are annoyed to different degrees by different mistakes. Suppose that Less Wrong is too disagreeable - that relative to the ideal, just-right, perfectly-rational amount of disagreement, we have a little more disagreement than that. You can imagine that to the men, this seems normal, forgivable, takeable in-stride - wrong, perhaps, but not really all that annoying. And you can imagine that conversely, the female-dominated mirror-image of Less Wrong would involve too much agreement relative to the ideal - lots of comments agreeing with each other - and that while this would seem normal, forgivable, takeable-in-stride to the female majority, it would drive the men up the wall, and some of them would leave, and the rest would be gritting their teeth. (This example plays to gender stereotypes, but that's because I'm speculating blindly; my brain only knows half the story and has to guess at the other half. Less obvious hypotheses are also welcome.) In a case like this, you begin by checking with trusted female rationalists to see if they think you're doing anything characteristically male, irrational, and annoying.

(2) The above points a finger at the rationalist community, and in particular its men, as making a mistake that drives away rational women. The complementary explanation would say: "No, we have exactly the rational amount of argument as it stands, or even too little. Male newcomers are fine with this, but female newcomers feel that there's too much conflict and disagreement and they leave." The true Way has no gender signature, but you can have a mistake that is characteristic of one sex but not the other, or a mistake that has been culturally inculcated in one gender but not the other. In this case we try to survey female newcomers to see what aspects seem like turn-offs (whether normatively rational or not), and then fix it (if not normatively rational) or try to soften the impact somehow (if normatively rational). (Ultimately, though, rationality is tough for everyone - there are parts that are hard for anyone to swallow, and you just have to make it as easy as you can.)

(3) It could be some indefinable difference of style - "indefinable" meaning that we can't pin it down tightly enough to duplicate - whereby male writers tend to attract male recruits and female writers attract female recruits. On this hypothesis, male writers end up with mostly male readers for much the same reason that Japanese writers end up with mostly Japanese readers. In this case I would suggest to potential female authors that they should write more, including new introductions and similar recruiting material. We could try for a mix of authorial genders in the material first encountered on-site. (By the same logic that if we wanted more Japanese rationalists we might encourage potential writers who happened to be Japanese.)

(4) We could be looking at a direct gender difference - where I parenthetically note that (by convention in such discussions) "gender" refers to a culture's concept of what it means to be a man or woman, while "sex" refers to actual distinctions of XX versus XY chromosomes. For example, consider this inspirational poster from a 1970s childrens' book. "Boys are pilots... girls are stewardesses... boys are doctors... girls are nurses." "Modern" cultures may still have a strong dose of "boys are rational, girls are un-self-controlled creatures of pure feeling who find logic and indeed all verbal argument to be vaguely unfeminine". I suppose the main remedy would be (a) to try and correct this the same way you would correct any other sort of childhood damage to sanity and (b) present strong female rationalist role models.

(5) The complementary hypothesis is a direct sex difference - i.e., the average female human actually is less interested in and compelled by deliberative reasoning compared to the average male human. If you were motivated to correct the sex balance regardless, you would consider e.g. where to find a prefiltered audience of people compellable by deliberative reasoning, a group that already happened to have good gender balance, and go recruiting there.

(6) We could be looking an indirect gender difference. Say, boys are raised to find a concept like "tsuyoku naritai" ("I want to become stronger") appealing, while girls are told to shut up and keep their heads down. If the masculine gender concept has a stronger endorsement of aspiring to self-improvement, it will, as a side effect, make a stronger endorsement of improving one's rationality. Again, the solutions would be female authors to tailor introductions to feminine audiences, and strong female role models. (If you're a woman and you're a talented writer and speaker, consider reading up on antitheism and trying to become a Fifth Horsewoman alongside Dawkins, Dennett, Harris and Hitchens...?)

(7) We could be looking at an indirect sex difference. The obvious evolutionary psychology hypothesis behind the imbalanced gender ratio in the iconoclastic community - the atheist/libertarian/technophile cluster - is the idea that males are inherently more attracted to gambles that seem high-risk and high-reward; they are more driven to try out strange ideas that come with big promises, because the genetic payoff for an unusually successful male has a much higher upper bound than the genetic payoff for an unusually successful female. It seems to me that male teenagers especially have something like a higher cognitive temperature, an ability to wander into strange places both good and bad. To some extent, this can be viewed as a problem of authorial style as well as innate dispositions - there's no law that says you have to emphasize the strangeness. You could start right out with pictures of a happy gender-balanced rationalist unchurch somewhere, and banner the page "A Return To Sanity". But a difference as basic as "more male teenagers have a high cognitive temperature" could prove very hard to address completely.

(8) Then there's the hypothesis made infamous by Larry Summers: Male variance in IQ (not the mean) is higher, so the right tail is dominated by males as you get further out. I know that just mentioning this sort of thing can cause a webpage to burst into flames, and so I would like to once again point out that individual IQ differences, whether derived from genes or eating lead-based paint as a kid, are already as awful as it gets - nothing is made any worse by talking about groups, since groups are just made out of individuals. The universe is already dreadful along this dimension, so we shouldn't care more whether groups are involved - though of course, thanks to our political instincts, we do care. The remedies in this not-actually-any-more-awful case are (a) continue the quest to systematize rationality training so that it is less exclusively the preserve of high-g individuals, and (b) recruit among prefiltered audiences that have good gender balance.

(9) Perhaps women are less underrepresented on Less Wrong than may at first appear, and men are more likely to comment for some reason. Or perhaps women are less likely to choose visibly feminine usernames. The gender ratio at physical meetups, while still unbalanced, seems noticeably better than the visible gender ratio among active commenters on the Internet. Not very plausible as a complete explanation; but we should consider hypotheses that involve unbalanced participation/visibility rather than unbalanced attraction/retention.

Sequence: The Craft and the Community

Bayesians vs. Barbarians

My Way

Referenced by: My Way • The Craft and the Community

Original with comments: Of Gender and Rationality

My Way

Eliezer Yudkowsky, 17 April 2009 01:25AM

Previously in series: Bayesians vs. Barbarians
Followup to: Of Gender and Rationality, Beware of Other-Optimizing

There is no such thing as masculine probability theory or feminine decision theory. In their pure form, the maths probably aren't even human. But the human practice of rationality - the arts associated with, for example, motivating yourself, or compensating factors applied to overcome your own biases - these things can in principle differ from gender to gender, or from person to person.

My attention was first drawn to this possibility of individual differences in optimization (in general) by thinking about rationality and gender (in particular). I've written rather more fiction than I've ever finished and published, including a story in which the main character, who happens to be the most rational person around, happens to be female. I experienced no particular difficulty in writing a female character who happened to be a rationalist. But she was not an obtrusive, explicit rationalist. She was not Jeffreyssai.

And it occurred to me that I could not imagine how to write Jeffreyssai as a woman; his way of teaching is paternal, not maternal. Even more, it occurred to me that in my writing there are women who are highly rational (on their way to other goals) but not women who are rationalists (as their primary, explicit role in the story).

It was at this point that I realized how much of my own take on rationality was specifically male, which hinted in turn that even more of it might be specifically Eliezer Yudkowsky.

A parenthetical, at this point, upon my own gender politics (lest anyone misinterpret me here). Of much of what passes for gender politics in present times, I have very little patience, as you might guess. But as recently as the 1970s this still passed for educational material, which makes me a bit more sympathetic.

So this about my gender politics: Unlike the case with, say, race, I don't think that an optimal outcome consists of gender distinctions being obliterated. If the day comes when no one notices or cares whether someone is black or white, any more than they notice eye color, I would only applaud. But obliterating the difference between male and female does not seem to me desirable, and I am glad that it is impossible using present-day technology; the fact that humanity has (at least) two sexes is part of what keeps life interesting.

But it seems to me that, as an inheritance from the dark ages, the concept of "normal" is tilted more toward male than female. Men are not constantly made aware that they are men in the same way that women are made constantly aware that they are women. (Though there are contexts where explicit masculinity is suddenly a focus.) It's not fun for women if female is defined as abnormal, as special. And so some feminists direct their efforts into trying to collapse gender distinctions, the way you would try to collapse racial distinctions. Just have everyone be normal, part of the same group. But I don't think that's realistic for our species - sex is real, it's not just gender - and in any case I prefer to live in a culture with (at least) two genders.

So - rather than obliterate the difference between genders into a common normality - I think that men should become more aware of themselves as men, so that being female isn't any more special or unusual or abnormal or worthy-of-remark than being male. Until a man sees his own argumentativeness as a distinctively male trait, he'll see women as abnormally passive (departures from the norm) rather than thinking "I am a male and therefore argumentative" (in the same way that women now identify various parts of themselves as feminine).

And yes, this does involve all sorts of dangers. Other cultures already have stronger male gender identities, and that's not always a good thing for the women in those cultures, if that culture already has an imbalance of power. But I'm not sure that the safe-seeming path of trying to obliterate as many distinctions as possible, is really available; men and women are different. Moreover, I like being a man free to express those forms of masculinity that I think are worthwhile, and I want to live in a world in which women are free to express whatever forms of feminity they think are worthwhile.

I'm saying all this, because I look over my accumulated essays and see that I am a distinctively male rationalist. Meanwhile, in another thread, a number of my fellow rationalists did go to some length to disidentify themselves as "female rationalists". I am sympathetic; from having been a child prodigy, I know how annoying it is to be celebrated as "having done so much while so young" rather than just "having done neat stuff in its own right regardless of age". I doubt that being singled out as an "amazing female rationalist" is any less annoying. But still: I built my art out of myself, and it became tied into every part of myself, and it happens to be a fact that I'm male. And if a woman were to pursue her art far enough, and tie it into every part of herself, she would, I think, find that her art came to resemble herself more and more, tied into her own motives and preferences; so that her art was, among other things, female.

It's hard to pin down this sort of thing exactly, because my own brain knows only half the story. My understanding of what it means to be female is too much shallower than my understanding of what it means to be male, it doesn't ring as true. I will try, though, to give an example of what I mean, if you will excuse me another excursion...

The single author I know who strikes me as most feminine is Jacqueline Carey. When I read her book Kushiel's Avatar, it gave me a feeling of being overwhelmingly outmatched as an author. I want to write characters with that kind of incredible depth and I can't. She is too far above me as an author. I write stories with female characters, and I wish I could write female characters who were as female as Carey's female characters, and so long as I'm dreaming, I also want to sprout wings and fly.

Let me give you an example, drawn from Kushiel's Avatar. This book - as have so many other books - involves, among its other plot points, saving the world. A shallow understanding of sex and gender, built mostly around abstract evolutionary psychology - such as I myself possess - would suggest that "taking great risks to save your tribe" is likely to be a more male sort of motivation - the status payoff from success would represent a greater fitness benefit to a man, and in the ancestral environment, it is the men who defend their tribe, etcetera. But in fact, reading SF and fantasy books by female authors, I have not noticed any particularly lower incidence of world-saving behavior by female protagonists.

If you told me to write a strongly feminine character, then I, with my shallow understanding, might try to have her risk everything to save her husband. The protagonist of Kushiel's Avatar, Phèdre nó Delaunay, does realize that the world is in danger and it needs to be saved. But she is also, in the same process, trying to rescue a kidnapped young boy. Her own child? That's how I would have written the story, but no; she is trying to rescue someone else's child. The child of her own archenemy, in fact, but no less innocent for all that. When I look at it after the fact, I can see how this reveals a deeper feminity, not the stereotype but a step beyond and behind the stereotype, something that rings true. Phèdre loves her husband - and this is shown not by how she puts aside saving the world to save him, but by how much it hurts her to put him in harm's way to save the world. Her feminity is shown, not by how protective she is toward her own child, but toward someone else's child.

It is this depth of writing that makes me aware of how my own brain is only regurgitating stereotypes by comparison.

I do dare say that I have developed my art of rationality as thoroughly as Carey has developed her thesis on love. And so my art taps into parts of me that are male. I cultivate the desire to become stronger; I accept and acknowledge within myself the desire to outdo others; I have learned to take pride in my identity as someone who faces down impossible challenges. While my own brain only knows half the story, it does seem to me that this is noticeably more a theme of shĹnen anime than shĹjo anime. Watch Hikaru no Go for an idea of what I mean.

And this is the reason why I can't write Jeffreyssai as a woman - I would not be able to really understand her motivations; I don't understand what taps female drives on that deep a level. I can regurgitate stereotypes, but reading Jacqueline Carey has made me aware that my grasp is shallow; it would not ring true.

What would the corresponding female rationalist be like? I don't know. I can't say. Some woman has to pursue her art as far as I've pursued mine, far enough that the art she learned from others fails her, so that she must remake her shattered art in her own image and in the image of her own task. And then tell the rest of us about it.

I sometimes think of myself as being like the protagonist in a classic SF labyrinth story, wandering further and further into some alien artifact, trying to call into a radio my description of the bizarre things I'm seeing, so that I can be followed. But what I'm finding is not just the Way, the thing that lies at the center of the labyrinth; it is also my Way, the path that I would take to come closer to the center, from whatever place I started out.

(Perhaps a woman would phrase the above, not as "Bayes's Theorem is the high pure abstract thing that is not male or female", but rather, "Bayes's Theorem is something we can all agree on". Or maybe that's only my own brain regurgitating stereotypes.)

Someone's bound to suggest, "Take the male parts out, then! Don't describe rationality as 'the martial art of mind'." Well... I may put in some work to gender-purify my planned book on rationality. It would be too much effort to make my blog posts less like myself, in that dimension. But I also want to point out that I enjoyed reading Kushiel's Avatar - I was not blocked from appreciating it on account of the book being visibly female.

I say all this because I want to convey this important idea, that there is the Way and my Way, the pure (or perhaps shared) thing at the center, and the many paths we take there from wherever we started out. To say that the path is individualized, is not to say that we are shielded from criticism by a screen of privacy (a common idiom of modern Dark Side Epistemology). There is still a common thing we are all trying to find. We should be aware that others' shortest paths may not be the same as our own, but this is not the same as giving up the ability to judge or to share.

Even so, you should be aware that I have radioed back my description of the single central shape and the path I took to get closer. If there are parts that are visibly male, then there are probably other parts - perhaps harder to identify - that are tightly bound to growing up with Orthodox Jewish parents, or (cough) certain other unusual features of my life.

I think there will not be a proper Art until many people have progressed to the point of remaking the Art in their own image, and then radioed back to describe their paths.

Sequence: The Craft and the Community

Of Gender and Rationality

The Sin of Underconfidence

Referenced by: Well-Kept Gardens Die By Pacifism • Go Forth and Create the Art! • The Craft and the Community

Original with comments: My Way

Rationality Quotes - April 2009

Eliezer Yudkowsky, 18 April 2009 01:26AM

A monthly thread for posting any interesting rationality-related quotes you've seen recently on the Internet, or had stored in your quotesfile for ages.

- Please post all quotes separately (so that they can be voted up (or down) separately) unless they are strongly related/ordered.

- Do not quote yourself.

- Do not quote comments/posts on LW/OB - if we do this, there should be a separate thread for it.

- No more than 5 quotes per person per monthly thread, please.

Original with comments: Rationality Quotes - April 2009

Great Books of Failure

Eliezer Yudkowsky, 19 April 2009 12:59AM

Followup to: Unteachable Excellence

As previously observed, extraordinary successes tend to be considered extraordinary precisely because it is hard to teach (relative to the then-current level of understanding and systematization). On the other hand, famous failures are much more likely to contain lessons on what to avoid next time.

Books about epic screwups have constituted some of my more enlightening reading. Do you have any such books to recommend?

Please break up multiple recommendations into multiple comments, one book per comment, so they can be voted on and discussed separately. And please say at least a little about the book's subject and what sort of lesson you learned from it.

Original with comments: Great Books of Failure

The Sin of Underconfidence

Eliezer Yudkowsky, 20 April 2009 06:30AM

There are three great besetting sins of rationalists in particular, and the third of these is underconfidence. Michael Vassar regularly accuses me of this sin, which makes him unique among the entire population of the Earth.

But he's actually quite right to worry, and I worry too, and any adept rationalist will probably spend a fair amount of time worying about it. When subjects know about a bias or are warned about a bias, overcorrection is not unheard of as an experimental result. That's what makes a lot of cognitive subtasks so troublesome - you know you're biased but you're not sure how much, and you don't know if you're correcting enough - and so perhaps you ought to correct a little more, and then a little more, but is that enough? Or have you, perhaps, far overshot? Are you now perhaps worse off than if you hadn't tried any correction?

You contemplate the matter, feeling more and more lost, and the very task of estimation begins to feel increasingly futile...

And when it comes to the particular questions of confidence, overconfidence, and underconfidence - being interpreted now in the broader sense, not just calibrated confidence intervals - then there is a natural tendency to cast overconfidence as the sin of pride, out of that other list which never warned against the improper use of humility or the abuse of doubt. To place yourself too high - to overreach your proper place - to think too much of yourself - to put yourself forward - to put down your fellows by implicit comparison - and the consequences of humiliation and being cast down, perhaps publicly - are these not loathesome and fearsome things?

To be too modest - seems lighter by comparison; it wouldn't be so humiliating to be called on it publicly, indeed, finding out that you're better than you imagined might come as a warm surprise; and to put yourself down, and others implicitly above, has a positive tinge of niceness about it, it's the sort of thing that Gandalf would do.

So if you have learned a thousand ways that humans fall into error and read a hundred experimental results in which anonymous subjects are humiliated of their overconfidence - heck, even if you've just read a couple of dozen - and you don't know exactly how overconfident you are - then yes, you might genuinely be in danger of nudging yourself a step too far down.

I have no perfect formula to give you that will counteract this. But I have an item or two of advice.

What is the danger of underconfidence?

Passing up opportunities. Not doing things you could have done, but didn't try (hard enough).

So here's a first item of advice: If there's a way to find out how good you are, the thing to do is test it. A hypothesis affords testing; hypotheses about your own abilities likewise. Once upon a time it seemed to me that I ought to be able to win at the AI-Box Experiment; and it seemed like a very doubtful and hubristic thought; so I tested it. Then later it seemed to me that I might be able to win even with large sums of money at stake, and I tested that, but I only won 1 time out of 3. So that was the limit of my ability at that time, and it was not necessary to argue myself upward or downward, because I could just test it.

One of the chief ways that smart people end up stupid, is by getting so used to winning that they stick to places where they know they can win - meaning that they never stretch their abilities, they never try anything difficult.

It is said that this is linked to defining yourself in terms of your "intelligence" rather than "effort", because then winning easily is a sign of your "intelligence", where failing on a hard problem could have been interpreted in terms of a good effort.

Now, I am not quite sure this is how an adept rationalist should think about these things: rationality is systematized winning and trying to try seems like a path to failure. I would put it this way: A hypothesis affords testing! If you don't know whether you'll win on a hard problem - then challenge your rationality to discover your current level. I don't usually hold with congratulating yourself on having tried - it seems like a bad mental habit to me - but surely not trying is even worse. If you have cultivated a general habit of confronting challenges, and won on at least some of them, then you may, perhaps, think to yourself "I did keep up my habit of confronting challenges, and will do so next time as well". You may also think to yourself "I have gained valuable information about my current level and where I need improvement", so long as you properly complete the thought, "I shall try not to gain this same valuable information again next time".

If you win every time, it means you aren't stretching yourself enough. But you should seriously try to win every time. And if you console yourself too much for failure, you lose your winning spirit and become a scrub.

When I try to imagine what a fictional master of the Competitive Conspiracy would say about this, it comes out something like: "It's not okay to lose. But the hurt of losing is not something so scary that you should flee the challenge for fear of it. It's not so scary that you have to carefully avoid feeling it, or refuse to admit that you lost and lost hard. Losing is supposed to hurt. If it didn't hurt you wouldn't be a Competitor. And there's no Competitor who never knows the pain of losing. Now get out there and win."

Cultivate a habit of confronting challenges - not the ones that can kill you outright, perhaps, but perhaps ones that can potentially humiliate you. I recently read of a certain theist that he had defeated Christopher Hitchens in a debate (severely so; this was said by atheists). And so I wrote at once to the Bloggingheads folks and asked if they could arrange a debate. This seemed like someone I wanted to test myself against. Also, it was said by them that Christopher Hitchens should have watched the theist's earlier debates and been prepared, so I decided not to do that, because I think I should be able to handle damn near anything on the fly, and I desire to learn whether this thought is correct; and I am willing to risk public humiliation to find out. Note that this is not self-handicapping in the classic sense - if the debate is indeed arranged (I haven't yet heard back), and I do not prepare, and I fail, then I do lose those stakes of myself that I have put up; I gain information about my limits; I have not given myself anything I consider an excuse for losing.

Of course this is only a way to think when you really are confronting a challenge just to test yourself, and not because you have to win at any cost. In that case you make everything as easy for yourself as possible. To do otherwise would be spectacular overconfidence, even if you're playing tic-tac-toe against a three-year-old.

A subtler form of underconfidence is losing your forward momentum - amid all the things you realize that humans are doing wrong, that you used to be doing wrong, of which you are probably still doing some wrong. You become timid; you question yourself but don't answer the self-questions and move on; when you hypothesize your own inability you do not put that hypothesis to the test.

Perhaps without there ever being a watershed moment when you deliberately, self-visibly decide not to try at some particular test... you just.... slow..... down......

It doesn't seem worthwhile any more, to go on trying to fix one thing when there are a dozen other things that will still be wrong...

There's not enough hope of triumph to inspire you to try hard...

When you consider doing any new thing, a dozen questions about your ability at once leap into your mind, and it does not occur to you that you could answer the questions by testing yourself...

And having read so much wisdom of human flaws, it seems that the course of wisdom is ever doubting (never resolving doubts), ever the humility of refusal (never the humility of preparation), and just generally, that it is wise to say worse and worse things about human abilities, to pass into feel-good feel-bad cynicism.

And so my last piece of advice is another perspective from which to view the problem - by which to judge any potential habit of thought you might adopt - and that is to ask:

Does this way of thinking make me stronger, or weaker? Really truly?

I have previously spoken of the danger of reasonableness - the reasonable-sounding argument that we should two-box on Newcomb's problem, the reasonable-sounding argument that we can't know anything due to the problem of induction, the reasonable-sounding argument that we will be better off on average if we always adopt the majority belief, and other such impediments to the Way. "Does it win?" is one question you could ask to get an alternate perspective. Another, slightly different perspective is to ask, "Does this way of thinking make me stronger, or weaker?" Does constantly reminding yourself to doubt everything make you stronger, or weaker? Does never resolving or decreasing those doubts make you stronger, or weaker? Does undergoing a deliberate crisis of faith in the face of uncertainty make you stronger, or weaker? Does answering every objection with a humble confession of you fallibility make you stronger, or weaker?

Are your current attempts to compensate for possible overconfidence making you stronger, or weaker? Hint: If you are taking more precautions, more scrupulously trying to test yourself, asking friends for advice, working your way up to big things incrementally, or still failing sometimes but less often then you used to, you are probably getting stronger. If you are never failing, avoiding challenges, and feeling generally hopeless and dispirited, you are probably getting weaker.

I learned the first form of this rule at a very early age, when I was practicing for a certain math test, and found that my score was going down with each practice test I took, and noticed going over the answer sheet that I had been pencilling in the correct answers and erasing them. So I said to myself, "All right, this time I'm going to use the Force and act on instinct", and my score shot up to above what it had been in the beginning, and on the real test it was higher still. So that was how I learned that doubting yourself does not always make you stronger - especially if it interferes with your ability to be moved by good information, such as your math intuitions. (But I did need the test to tell me this!)

Underconfidence is not a unique sin of rationalists alone. But it is a particular danger into which the attempt to be rational can lead you. And it is a stopping mistake - an error which prevents you from gaining that further experience which would correct the error.

Because underconfidence actually does seem quite common among aspiring rationalists who I meet - though rather less common among rationalists who have become famous role models) - I would indeed name it third among the three besetting sins of rationalists.

Sequence: The Craft and the Community

My Way

Well-Kept Gardens Die By Pacifism

Referenced by: Well-Kept Gardens Die By Pacifism • The Craft and the Community • That Magical Click

Original with comments: The Sin of Underconfidence

Well-Kept Gardens Die By Pacifism

Eliezer Yudkowsky, 21 April 2009 02:44AM

Previously in series: My Way
Followup to: The Sin of Underconfidence

Good online communities die primarily by refusing to defend themselves.

Somewhere in the vastness of the Internet, it is happening even now. It was once a well-kept garden of intelligent discussion, where knowledgeable and interested folk came, attracted by the high quality of speech they saw ongoing. But into this garden comes a fool, and the level of discussion drops a little - or more than a little, if the fool is very prolific in their posting. (It is worse if the fool is just articulate enough that the former inhabitants of the garden feel obliged to respond, and correct misapprehensions - for then the fool dominates conversations.)

So the garden is tainted now, and it is less fun to play in; the old inhabitants, already invested there, will stay, but they are that much less likely to attract new blood. Or if there are new members, their quality also has gone down.

Then another fool joins, and the two fools begin talking to each other, and at that point some of the old members, those with the highest standards and the best opportunities elsewhere, leave...

I am old enough to remember the USENET that is forgotten, though I was very young. Unlike the first Internet that died so long ago in the Eternal September, in these days there is always some way to delete unwanted content. We can thank spam for that - so egregious that no one defends it, so prolific that no one can just ignore it, there must be a banhammer somewhere.

But when the fools begin their invasion, some communities think themselves too good to use their banhammer for - gasp! - censorship.

After all - anyone acculturated by academia knows that censorship is a very grave sin... in their walled gardens where it costs thousands and thousands of dollars to enter, and students fear their professors' grading, and heaven forbid the janitors should speak up in the middle of a colloquium.

It is easy to be naive about the evils of censorship when you already live in a carefully kept garden. Just like it is easy to be naive about the universal virtue of unconditional nonviolent pacifism, when your country already has armed soldiers on the borders, and your city already has police. It costs you nothing to be righteous, so long as the police stay on their jobs.

The thing about online communities, though, is that you can't rely on the police ignoring you and staying on the job; the community actually pays the price of its virtuousness.

In the beginning, while the community is still thriving, censorship seems like a terrible and unnecessary imposition. Things are still going fine. It's just one fool, and if we can't tolerate just one fool, well, we must not be very tolerant. Perhaps the fool will give up and go away, without any need of censorship. And if the whole community has become just that much less fun to be a part of... mere fun doesn't seem like a good justification for (gasp!) censorship, any more than disliking someone's looks seems like a good reason to punch them in the nose.

(But joining a community is a strictly voluntary process, and if prospective new members don't like your looks, they won't join in the first place.)

And after all - who will be the censor? Who can possibly be trusted with such power?

Quite a lot of people, probably, in any well-kept garden. But if the garden is even a little divided within itself - if there are factions - if there are people who hang out in the community despite not much trusting the moderator or whoever could potentially wield the banhammer -

(for such internal politics often seem like a matter of far greater import than mere invading barbarians)

- then trying to defend the community is typically depicted as a coup attempt. Who is this one who dares appoint themselves as judge and executioner? Do they think their ownership of the server means they own the people? Own our community? Do they think that control over the source code makes them a god?

I confess, for a while I didn't even understand why communities had such trouble defending themselves - I thought it was pure naivete. It didn't occur to me that it was an egalitarian instinct to prevent chieftains from getting too much power. "None of us are bigger than one another, all of us are men and can fight; I am going to get my arrows", was the saying in one hunter-gatherer tribe whose name I forget. (Because among humans, unlike chimpanzees, weapons are an equalizer - the tribal chieftain seems to be an invention of agriculture, when people can't just walk away any more.)

Maybe it's because I grew up on the Internet in places where there was always a sysop, and so I take for granted that whoever runs the server has certain responsibilities. Maybe I understand on a gut level that the opposite of censorship is not academia but 4chan (which probably still has mechanisms to prevent spam). Maybe because I grew up in that wide open space where the freedom that mattered was the freedom to choose a well-kept garden that you liked and that liked you, as if you actually could find a country with good laws. Maybe because I take it for granted that if you don't like the archwizard, the thing to do is walk away (this did happen to me once, and I did indeed just walk away).

And maybe because I, myself, have often been the one running the server. But I am consistent, usually being first in line to support moderators - even when they're on the other side from me of the internal politics. I know what happens when an online community starts questioning its moderators. Any political enemy I have on a mailing list who's popular enough to be dangerous is probably not someone who would abuse that particular power of censorship, and when they put on their moderator's hat, I vocally support them - they need urging on, not restraining. People who've grown up in academia simply don't realize how strong are the walls of exclusion that keep the trolls out of their lovely garden of "free speech".

Any community that really needs to question its moderators, that really seriously has abusive moderators, is probably not worth saving. But this is more accused than realized, so far as I can see.

In any case the light didn't go on in my head about egalitarian instincts (instincts to prevent leaders from exercising power) killing online communities until just recently. While reading a comment at Less Wrong, in fact, though I don't recall which one.

But I have seen it happen - over and over, with myself urging the moderators on and supporting them whether they were people I liked or not, and the moderators still not doing enough to prevent the slow decay. Being too humble, doubting themselves an order of magnitude more than I would have doubted them. It was a rationalist hangout, and the third besetting sin of rationalists is underconfidence.

This about the Internet: Anyone can walk in. And anyone can walk out. And so an online community must stay fun to stay alive. Waiting until the last resort of absolute, blatent, undeniable egregiousness - waiting as long as a police officer would wait to open fire - indulging your conscience and the virtues you learned in walled fortresses, waiting until you can be certain you are in the right, and fear no questioning looks - is waiting far too late.

I have seen rationalist communities die because they trusted their moderators too little.

But that was not a karma system, actually.

Here - you must trust yourselves.

A certain quote seems appropriate here: "Don't believe in yourself! Believe that I believe in you!"

Because I really do honestly think that if you want to downvote a comment that seems low-quality... and yet you hesitate, wondering if maybe you're downvoting just because you disagree with the conclusion or dislike the author... feeling nervous that someone watching you might accuse you of groupthink or echo-chamber-ism or (gasp!) censorship... then nine times of ten, I bet, nine times out of ten at least, it is a comment that really is low-quality.

You have the downvote. Use it or USENET.

Sequence: The Craft and the Community

The Sin of Underconfidence

Practical Advice Backed By Deep Theories

Referenced by: Go Forth and Create the Art! • The Craft and the Community

Original with comments: Well-Kept Gardens Die By Pacifism

Proposal: Use the Wiki for Concepts

Eliezer Yudkowsky, 22 April 2009 05:21AM

So... the longer I think about this Wiki thing, the more it seems like a really good idea - a missing piece falling into place.

Here's my proposal, which I turn over to this, the larger community that suggested the Wiki in the first place:

The Wiki should consist mainly of short concept introductions plus links to longer posts, rather than original writing. Original writing goes in a post on Less Wrong, which may get voted up and down, or commented on; and this post should reference previous work by linking to the Wiki rather than other posts, to the extent that the concepts referred to can be given short summaries. The intent is to set up a resonance that bounces back and forth between the Wiki (short concept summaries that can be read standalone, and links to more info for in-depth exploration) and the posts (which make the actual arguments and do the actual analyses).

My role model here is TV Tropes, which manages to be, shall we say, really explorable, because of the resonance between the tropes, and the shows/events in which those tropes occur, and the other tropes that occur in those shows/events. And furthermore, you know that the trope explanation itself will be a short bite of joy, and that reading the further references is optional.

There would be exceptions to the "no original research" rule for projects that were multi-editor and not easily prosecuted through comments - for example, a project to make a list of all posts with one-sentence summaries in chronological order.

There are also obvious exceptions to the "link to the wiki" rule, such as for any case where it really was futile to reference anything except the complete argument; or where you wanted to talk about part of the argument, rather than the general concept argued; or when you wanted to talk about a conversational event that happened in a particular post.

I would suggest that the general format of a Wiki entry be a short summary / definition (that can maybe gloss or list some of the arguments if there's room, or even give the argument if it can be put really briefly), with the body of this being no more than a screenful as a general rule. Then links to posts, with descriptions (for each post) of why that post is relevant or what it has to say - probably one or two sentences, as a rule. Then references to outside posts on the same topic - although if the best reference is an outside discussion, that could come first.

Summaries of whole sequences could also go on the Wiki - since it seems more like static descriptive content, rather than debatable analysis and argument, which is how the wiki/blog dichotomy is starting to shape up in my mind.

Given unlimited development resources we'd want to integrate the two userbases, have a karma requirement to edit the Wiki, and such things, but we don't have much development resources (whines for Python volunteers again). But I would still like to see a list of recent edits and/or active pages in the Less Wrong blog sidebar, and a list of recent blog posts and recent comments in the Wiki sidebar. Of course the first priority is getting the Wiki set up on Less Wrong at all, rather than the current foreign host - I'm told this is in In Progress.

Once my old posts are imported from Overcoming Bias, it would be nice if someone went through them and changed the links to posts (that reference concepts per se, rather than conversational events or parts of arguments) to links to the Wiki, creating appropriate pages and concept summaries as necessary. That is, it would be nice if I didn't have to do this myself. Anyone interested in volunteering, leave a comment - it'd be nice if you had some comments to your name, by which to judge your writing skills, and perhaps some karma. This is a large job - though 10 posts a day would get it done in two months - so don't step up if you don't have the time. It does seem like the sort of thing that should definitely get done by someone who is not me.

I've already seen at least one person call Overcoming Bias "a bigger, and far more productive, time sink than Wikipedia or even TV Tropes". Done right, it really could be more addictive than TV Tropes, for the intellectually curious, because it would seem (and be!) more productive; when you browse TV Tropes it feels like you're wasting time.

I suppose I should feel slightly nervous about this, but it still seems like something that ought to be done if feasible, even though it sounds a bit scary - and I hope I'm not just saying that because I'm tempted to break out into mad scientist laughter.

Referenced by: The End (of Sequences) • Wiki.lesswrong.com Is Live

Original with comments: Proposal: Use the Wiki for Concepts

Go Forth and Create the Art!

Eliezer Yudkowsky, 23 April 2009 01:37AM

Previously in series: Well-Kept Gardens Die By Pacifism
Followup to: My Way

I have said a thing or two about rationality, these past months. I have said a thing or two about how to untangle questions that have become confused, and how to tell the difference between real reasoning and fake reasoning, and the will to become stronger that leads you to try before you flee; I have said something about doing the impossible.

And these are all techniques that I developed in the course of my own projects - which is why there is so much about cognitive reductionism, say - and it is possible that your mileage may vary in trying to apply it yourself. The one's mileage may vary. Still, those wandering about asking "But what good is it?" might consider rereading some of the earlier posts; knowing about e.g. the conjunction fallacy and how to spot it in an argument, hardly seems esoteric. Understanding why motivated skepticism is bad for you can constitute the whole difference, I suspect, between a smart person who ends up smart and a smart person who ends up stupid. Affective death spirals consume many among the unwary...

Yet there is, I think, more absent than present in this "art of rationality" - defeating akrasia and coordinating groups are two of the deficits I feel most keenly. I've concentrated more heavily on epistemic rationality than instrumental rationality, in general. And then there's training, teaching, verification, and becoming a proper experimental science based on that. And if you generalize a bit further, then building the Art could also be taken to include issues like developing better introductory literature, developing better slogans for public relations, establishing common cause with other Enlightenment subtasks, analyzing and addressing the gender imbalance problem...

But those small pieces of rationality that I've set out... I hope... just maybe...

I suspect - you could even call it a guess - that there is a barrier to getting started, in this matter of rationality. Where by default, in the beginning, you don't have enough to build on. Indeed so little that you don't have a clue that more exists, that there is an Art to be found. And if you do begin to sense that more is possible - then you may just instantaneously go wrong. As David Stove observes - I'm not going to link it, because it deserves its own post - most "great thinkers" in philosophy, e.g. Hegel, are properly objects of pity. That's what happens by default to anyone who sets out to develop the art of thinking; they develop fake answers.

When you try to develop part of the human art of thinking... then you are doing something not too dissimilar to what I was doing over in Artificial Intelligence. You will be tempted by fake explanations of the mind, fake accounts of causality, mysterious holy words, and the amazing idea that solves everything.

It's not that the particular, epistemic, fake-detecting methods that I use, are so good for every particular problem; but they seem like they might be helpful for discriminating good and bad systems of thinking.

I hope that someone who learns the part of the Art that I've set down here, will not instantaneously and automatically go wrong, if they start asking themselves, "How should people think, in order to solve new problem X that I'm working on?" They will not immediately run away; they will not just make stuff up at random; they may be moved to consult the literature in experimental psychology; they will not automatically go into an affective death spiral around their Brilliant Idea; they will have some idea of what distinguishes a fake explanation from a real one. They will get a saving throw.

It's this sort of barrier, perhaps, which prevents people from beginning to develop an art of rationality, if they are not already rational.

And so instead they... go off and invent Freudian psychoanalysis. Or a new religion. Or something. That's what happens by default, when people start thinking about thinking.

I hope that the part of the Art I have set down, as incomplete as it may be, can surpass that preliminary barrier - give people a base to build on; give them an idea that an Art exists, and somewhat of how it ought to be developed; and give them at least a saving throw before they instantaneously go astray.

That's my dream - that this highly specialized-seeming art of answering confused questions, may be some of what is needed, in the very beginning, to go and complete the rest.

A task which I am leaving to you. Probably, anyway. I make no promises as to where my attention may turn in the future. But y'know, there are certain other things I need to do. Even if I develop yet more Art by accident, it may be that I will not have the time to write any of it up.

Beyond all that I have said of fake answers and traps, there are two things I would like you to keep in mind.

The first - that I drew on multiple sources to create my Art. I read many different authors, many different experiments, used analogies from many different fields. You will need to draw on multiple sources to create your portion of the Art. You should not be getting all your rationality from one author - though there might be, perhaps, a certain centralized website, where you went to post the links and papers that struck you as really important. And a maturing Art will need to draw from multiple sources. To the best of my knowledge there is no true science that draws its strength from only one person. To the best of my knowledge that is strictly an idiom of cults. A true science may have its heroes, it may even have its lonely defiant heroes, but it will have more than one.

The second - that I created my Art in the course of trying to do some particular thing which animated all my efforts. Maybe I'm being too idealistic - maybe thinking too much of the way the world should work - but even so, I somewhat suspect that you couldn't develop the Art just by sitting around thinking to yourself, "Now how can I fight that akrasia thingy?" You'd develop the rest of the Art in the course of trying to do something. Maybe even - if I'm not overgeneralizing from my own history - some task difficult enough to strain and break your old understanding and force you to reinvent a few things. But maybe I'm wrong, and the next leg of the work will be done by direct, specific investigation of "rationality", without any need of a specific application considered more important.

My previous attempt to describe this principle in terms of respect bounded by a secret identity, was roundly rejected by my audience. Maybe "leave the house" would be more appropriate? It sounds to me like a really good, healthy idea. Still - perhaps I am deceived. We shall see where the next pieces of the Art do, in fact, come from.

I have striven for a long time now to convey, pass on, share a piece of the strange thing I touched, which seems to me so precious. And I'm not sure that I ever said the central rhythm into words. Maybe you can find it by listening to the notes. I can say these words but not the rule that generates them, or the rule behind the rule; one can only hope that by using the ideas, perhaps, similar machinery might be born inside you. Remember that all human efforts at learning arcana, slide by default into passwords, hymns, and floating assertions.

I have striven for a long time now to convey my Art. Mostly without success, before this present effort. Earlier I made efforts only in passing, and got, perhaps, as much success as I deserved. Like throwing pebbles in a pond, that generate a few ripples, and then fade away... This time I put some back into it, and heaved a large rock. Time will tell if it was large enough - if I really disturbed anyone deeply enough that the waves of the impact will continue under their own motion. Time will tell if I have created anything that moves under its own power.

(Not to mention that - I hope - the thing with the karma will stop the slide into virtual entropy that has destroyed every community I tried to build earlier as soon as I tried to pull back my attention a little.)

My last essay on having a secret identity was not well-received, so let me try again: I want people to go forth, but also to return. Or maybe even to go forth and stay simultaneously, because this is the Internet and we can get away with that sort of thing; I've learned some interesting things on Less Wrong, lately, and if continuing motivation over years is any sort of problem, talking to others (or even seeing that others are also trying) does often help.

But at any rate, if I have affected you at all, then I hope you will go forth and confront challenges, and achieve somewhere beyond your armchair, and create new Art; and then, remembering whence you came, radio back to tell others what you learned.

Sequence: The Craft and the Community

Less Meta

Referenced by: The Craft and the Community

Original with comments: Go Forth and Create the Art!

This Didn't Have To Happen

Eliezer Yudkowsky, 23 April 2009 07:07PM

My girlfriend/SO's grandfather died last night, running on a treadmill when his heart gave out.

He wasn't signed up for cryonics, of course. She tried to convince him, and I tried myself a little the one time I met her grandparents.

"This didn't have to happen. Fucking religion."

That's what my girlfriend said.

I asked her if I could share that with you, and she said yes.

Just so that we're clear that all the wonderful emotional benefits of self-delusion come with a price, and the price isn't just to you.

Original with comments: This Didn't Have To Happen

Practical Advice Backed By Deep Theories

Eliezer Yudkowsky, 25 April 2009 06:52PM

Once upon a time, Seth Roberts took a European vacation and found that he started losing weight while drinking unfamiliar-tasting caloric fruit juices.

Now suppose Roberts had not known, and never did know, anything about metabolic set points or flavor-calorie associations - all this high-falutin' scientific experimental research that had been done on rats and occasionally humans.

He would have posted to his blog, "Gosh, everyone! You should try these amazing fruit juices that are making me lose weight!" And that would have been the end of it. Some people would have tried it, it would have worked temporarily for some of them (until the flavor-calorie association kicked in) and there never would have been a Shangri-La Diet per se.

The existing Shangri-La Diet is visibly incomplete - for some people, like me, it doesn't seem to work, and there is no apparent reason for this or any logic permitting it. But the reason why as many people have benefited as they have - the reason why there was more than just one more blog post describing a trick that seemed to work for one person and didn't work for anyone else - is that Roberts knew the experimental science that let him interpret what he was seeing, in terms of deep factors that actually did exist.

One of the pieces of advice on OB/LW that was frequently cited as the most important thing learned, was the idea of "the bottom line" - that once a conclusion is written in your mind, it is already true or already false, already wise or already stupid, and no amount of later argument can change that except by changing the conclusion. And this ties directly into another oft-cited most important thing, which is the idea of "engines of cognition", minds as mapping engines that require evidence as fuel.

If I had merely written one more blog post that said, "You know, you really should be more open to changing your mind - it's pretty important - and oh yes, you should pay attention to the evidence too." And this would not have been as useful. Not just because it was less persuasive, but because the actual operations would have been much less clear without the explicit theory backing it up. What constitutes evidence, for example? Is it anything that seems like a forceful argument? Having an explicit probability theory and an explicit causal account of what makes reasoning effective, makes a large difference in the forcefulness and implementational details of the old advice to "Keep an open mind and pay attention to the evidence."

It is also important to realize that causal theories are much more likely to be true when they are picked up from a science textbook than when invented on the fly - it is very easy to invent cognitive structures that look like causal theories but are not even anticipation-controlling, let alone true.

This is the signature style I want to convey from all those posts that entangled cognitive science experiments and probability theory and epistemology with the practical advice - that practical advice actually becomes practically more powerful if you go out and read up on cognitive science experiments, or probability theory, or even materialist epistemology, and realize what you're seeing. This is the brand that can distinguish LW from ten thousand other blogs purporting to offer advice.

I could tell you, "You know, how much you're satisfied with your food probably depends more on the quality of the food than on how much of it you eat." And you would read it and forget about it, and the impulse to finish off a whole plate would still feel just as strong. But if I tell you about scope insensitivity, and duration neglect and the Peak/End rule, you are suddenly aware in a very concrete way, looking at your plate, that you will form almost exactly the same retrospective memory whether your portion size is large or small; you now possess a deep theory about the rules governing your memory, and you know that this is what the rules say. (You also know to save the dessert for last.)

I want to hear how I can overcome akrasia - how I can have more willpower, or get more done with less mental pain. But there are ten thousand people purporting to give advice on this, and for the most part, it is on the level of that alternate Seth Roberts who just tells people about the amazing effects of drinking fruit juice. Or actually, somewhat worse than that - it's people trying to describe internal mental levers that they pulled, for which there are no standard words, and which they do not actually know how to point to. See also the illusion of transparency, inferential distance, and double illusion of transparency. (Notice how "You overestimate how much you're explaining and your listeners overestimate how much they're hearing" becomes much more forceful as advice, after I back it up with a cognitive science experiment and some evolutionary psychology?)

I think that the advice I need is from someone who reads up on a whole lot of experimental psychology dealing with willpower, mental conflicts, ego depletion, preference reversals, hyperbolic discounting, the breakdown of the self, picoeconomics, etcetera, and who, in the process of overcoming their own akrasia, manages to understand what they did in truly general terms - thanks to experiments that give them a vocabulary of cognitive phenomena that actually exist, as opposed to phenomena they just made up. And moreover, someone who can explain what they did to someone else, thanks again to the experimental and theoretical vocabulary that lets them point to replicable experiments that ground the ideas in very concrete results, or mathematically clear ideas.

Note the grade of increasing difficulty in citing:

- Concrete experimental results (for which one need merely consult a paper, hopefully one that reported p < 0.01 because p < 0.05 may fail to replicate)

- Causal accounts that are actually true (which may be most reliably obtained by looking for the theories that are used by a majority within a given science)

- Math validly interpreted (on which I have trouble offering useful advice because so much of my own math talent is intuition that kicks in before I get a chance to deliberate)

If you don't know who to trust, or you don't trust yourself, you should concentrate on experimental results to start with, move on to thinking in terms of causal theories that are widely used within a science, and dip your toes into math and epistemology with extreme caution.

But practical advice really, really does become a lot more powerful when it's backed up by concrete experimental results, causal accounts that are actually true, and math validly interpreted.

Sequence: The Craft and the Community

Well-Kept Gardens Die By Pacifism

Less Meta

Referenced by: The Craft and the Community • Willpower Hax #487: Execute by Default

Original with comments: Practical Advice Backed By Deep Theories

Less Meta

Eliezer Yudkowsky, 26 April 2009 05:38AM

My recent sequence on the craft and the community is highly forward-looking - not an immediately whole recipe, but a list of action items and warnings for anyone setting out in the future. Having expended this much effort already, it seems worthwhile to try and leverage others' future efforts.

That sequence seemed like an appropriate finale, but putting it last had some side effects that I didn't expect. Thanks to the recency effect, people are now talking as if the entire œuvre had all been anticipation of future awesomeness, with no practical value in the present...

Okay, seriously, if you look over my posts on Overcoming Bias that are not just a couple of months old, you really should see quite a lot of practical day-to-day stuff. Yes, there's a long sequence on quantum mechanics, but there really is plenty of day-to-day stuff, right down to applying biases of evaluability to save money on holiday shopping. (Not to mention that I finally did derive real-world advice out of the QM detour!)

I suspect there may also be a problem here with present schools not teaching people the experience of creating new craft. What they present you with, is what you learn. So when I talk about my belief that we could be doing better, they look around and say: "But we aren't doing that well!" rather than "Hm, how can we make progress on this?"

And then your current accomplishments start to pale in the light of grander dreams, etcetera. One of the great lessons of Artificial Intelligence is that no matter how much progress you make, it can always be made to appear slow and unexciting just by having someone else quoted in the newspapers about much grander promises on which they fail to deliver.

Anyway. I think that discussion here is going a bit too meta, too much about the community, and that was not the final push I had planned to deliver. (I know, I know, obvious in retrospect, yes, but I still didn't see it coming.) Hence the quick add-on about practical advice backed by experimental results or deep theories - thankfully we are going back to those again in recent posts.

So if it's all right with you, dear readers, after the end of April, I will not promote more than one meta post unless at least four nonmeta posts have appeared before it - does that sound fair?

Sequence: The Craft and the Community

Practical Advice Backed By Deep Theories

Go Forth and Create the Art!

Referenced by: The Craft and the Community

Original with comments: Less Meta

The Craft and the Community

Eliezer Yudkowsky, 26 April 2009 05:52PM

This sequence ran from March to April of 2009 and dealt with the topic of building rationalist communities that could systematically improve on the art, craft, and science of human rationality. This is a highly forward-looking sequence - not so much an immediately complete recipe, as a list of action items and warnings for anyone setting out in the future to build a craft and a community.

- Raising the Sanity Waterline: Behind every particular failure of social rationality is a larger and more general failure of social rationality; even if all religious content were deleted tomorrow from all human minds, the larger failures that permit religion would still be present. Religion may serve the function of an asphyxiated canary in a coal mine - getting rid of the canary doesn't get rid of the gas. Even a complete social victory for atheism would only be the beginning of the real work of rationalists. What could you teach people without ever explicitly mentioning religion, that would raise their general epistemic waterline to the point that religion went underwater?

- A Sense That More Is Possible: The art of human rationality may have not been much developed because its practitioners lack a sense that vastly more is possible. The level of expertise that most rationalists strive to develop is not on a par with the skills of a professional mathematician - more like that of a strong casual amateur. Self-proclaimed "rationalists" don't seem to get huge amounts of personal mileage out of their craft, and no one sees a problem with this. Yet rationalists get less systematic training in a less systematic context than a first-dan black belt gets in hitting people.

- Epistemic Viciousness: An essay by Gillian Russell on "Epistemic Viciousness in the Martial Arts" generalizes amazingly to possible and actual problems with building a community around rationality. Most notably the extreme dangers associated with "data poverty" - the difficulty of testing the skills in the real world. But also such factors as the sacredness of the dojo, the investment in teachings long-practiced, the difficulty of book learning that leads into the need to trust a teacher, deference to historical masters, and above all, living in data poverty while continuing to act as if the luxury of trust is possible.

- Schools Proliferating Without Evidence: The branching schools of "psychotherapy", another domain in which experimental verification was weak (nonexistent, actually), show that an aspiring craft lives or dies by the degree to which it can be tested in the real world. In the absence of that testing, one becomes prestigious by inventing yet another school and having students, rather than excelling at any visible performance criterion. The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness - that there was a family of measures that by golly did validate well against each other. The act of creating a new measurement creates new science; if it's a good measurement, you get good science.

- 3 Levels of Rationality Verification: How far the craft of rationality can be taken, depends largely on what methods can be invented for verifying it. Tests seem usefully stratifiable into reputational, experimental, and organizational. A "reputational" test is some real-world problem that tests the ability of a teacher or a school (like running a hedge fund, say) - "keeping it real", but without being able to break down exactly what was responsible for success. An "experimental" test is one that can be run on each of a hundred students (such as a well-validated survey). An "organizational" test is one that can be used to preserve the integrity of organizations by validating individuals or small groups, even in the face of strong incentives to game the test. The strength of solution invented at each level will determine how far the craft of rationality can go in the real world.

- Why Our Kind Can't Cooperate: The atheist/libertarian/technophile/sf-fan/early-adopter/programmer/etc crowd, aka "the nonconformist cluster", seems to be stunningly bad at coordinating group projects. There are a number of reasons for this, but one of them is that people are as reluctant to speak agreement out loud, as they are eager to voice disagreements - the exact opposite of the situation that obtains in more cohesive and powerful communities. This is not rational either! It is dangerous to be half a rationalist (in general), and this also applies to teaching only disagreement but not agreement, or only lonely defiance but not coordination. The pseudo-rationalist taboo against expressing strong feelings probably doesn't help either.

- Tolerate Tolerance: One of the likely characteristics of someone who sets out to be a "rationalist" is a lower-than-usual tolerance for flawed thinking. This makes it very important to tolerate other people's tolerance - to avoid rejecting them because they tolerate people you wouldn't - since otherwise we must all have exactly the same standards of tolerance in order to work together, which is unlikely. Even if someone has a nice word to say about complete lunatics and crackpots - so long as they don't literally believe the same ideas themselves - try to be nice to them? Intolerance of tolerance corresponds to punishment of non-punishers, a very dangerous game-theoretic idiom that can lock completely arbitrary systems in place even when they benefit no one at all.

- You're Calling Who A Cult Leader?: Paul Graham gets exactly the same accusations about "cults" and "echo chambers" and "coteries" that I do, in exactly the same tone - e.g. comparing the long hours worked by Y Combinator startup founders to the sleep-deprivation tactic used in cults, or claiming that founders were asked to move to the Bay Area startup hub as a cult tactic of separation from friends and family. This is bizarre, considering our relative surface risk factors. It just seems to be a failure mode of the nonconformist community in general. By far the most cultish-looking behavior on Hacker News is people trying to show off how willing they are to disagree with Paul Graham, which, I can personally testify, feels really bizarre when you're the target. Admiring someone shouldn't be so scary - I don't hold back so much when praising e.g. Douglas Hofstadter; in this world there are people who have pulled off awesome feats and it is okay to admire them highly.

- On Things That Are Awesome: Seven followup thoughts: I can list more than one thing that is awesome; when I think of "Douglas Hofstadter" I am really thinking of his all-time greatest work; the greatest work is not the person; when we imagine other people we are imagining their output, so the real Douglas Hofstadter is the source of "Douglas Hofstadter"; I most strongly get the sensation of awesomeness when I see someone outdoing me overwhelmingly, at some task I've actually tried; we tend to admire unique detailed awesome things and overlook common nondetailed awesome things; religion and its bastard child "spirituality" tends to make us overlook human awesomeness.

- Your Price For Joining: The game-theoretical puzzle of the Ultimatum game has its reflection in a real-world dilemma: How much do you demand that an existing group adjust toward you, before you will adjust toward it? Our hunter-gatherer instincts will be tuned to groups of 40 with very minimal administrative demands and equal participation, meaning that we underestimate the inertia of larger and more specialized groups and demand too much before joining them. In other groups this resistance can be overcome by affective death spirals and conformity, but rationalists think themselves too good for this - with the result that people in the nonconformist cluster often set their joining prices way way way too high, like an 50-way split with each player demanding 20% of the money. Nonconformists need to move in the direction of joining groups more easily, even in the face of annoyances and apparent unresponsiveness. If an issue isn't worth personally fixing by however much effort it takes, it's not worth a refusal to contribute.

- Can Humanism Match Religion's Output?: Anyone with a simple and obvious charitable project - responding with food and shelter to a tidal wave in Thailand, say - would be better off by far pleading with the Pope to mobilize the Catholics, rather than with Richard Dawkins to mobilize the atheists. For so long as this is true, any increase in atheism at the expense of Catholicism will be something of a hollow victory, regardless of all other benefits. Can no rationalist match the motivation that comes from the irrational fear of Hell? Or does the real story have more to do with the motivating power of physically meeting others who share your cause, and group norms of participating?

- Church vs. Taskforce: Churches serve a role of providing community - but they aren't explicitly optimized for this, because their nominal role is different. If we desire community without church, can we go one better in the course of deleting religion? There's a great deal of work to be done in the world; rationalist communities might potentially organize themselves around good causes, while explicitly optimizing for community.

- Rationality: Common Interest of Many Causes: Many causes benefit particularly from the spread of rationality - because it takes a little more rationality than usual to see their case, as a supporter, or even just a supportive bystander. Not just the obvious causes like atheism, but things like marijuana legalization. In the case of my own work this effect was strong enough that after years of bogging down I threw up my hands and explicitly recursed on creating rationalists. If such causes can come to terms with not individually capturing all the rationalists they create, then they can mutually benefit from mutual effort on creating rationalists. This cooperation may require learning to shut up about disagreements between such causes, and not fight over priorities, except in specialized venues clearly marked.

- Helpless Individuals: When you consider that our grouping instincts are optimized for 50-person hunter-gatherer bands where everyone knows everyone else, it begins to seem miraculous that modern-day large institutions survive at all. And in fact, the vast majority of large modern-day institutions simply fail to exist in the first place. This is why funding of Science is largely through money thrown at Science rather than donations from individuals - research isn't a good emotional fit for the rare problems that individuals can manage to coordinate on. In fact very few things are, which is why e.g. 200 million adult Americans have such tremendous trouble supervising the 535 members of Congress. Modern humanity manages to put forth very little in the way of coordinated individual effort to serve our collective individual interests.

- Money: The Unit of Caring: Omohundro's resource balance principle implies that the inside of any approximately rational system has a common currency of expected utilons. In our world, this common currency is called "money" and it is the unit of how much society cares about something - a brutal yet obvious point. Many people, seeing a good cause, would prefer to help it by donating a few volunteer hours. But this avoids the tremendous gains of comparative advantage, professional specialization, and economies of scale - the reason we're not still in caves, the only way anything ever gets done in this world, the tools grownups use when anyone really cares. Donating hours worked within a professional specialty and paying-customer priority, whether directly, or by donating the money earned to hire other professional specialists, is far more effective than volunteering unskilled hours.

- Purchase Fuzzies and Utilons Separately: Wealthy philanthropists typically make the mistake of trying to purchase warm fuzzy feelings, status among friends, and actual utilitarian gains, simultaneously; this results in vague pushes along all three dimensions and a mediocre final result. It should be far more effective to spend some money/effort on buying altruistic fuzzies at maximum optimized efficiency (e.g. by helping people in person and seeing the results in person), buying status at maximum efficiency (e.g. by donating to something sexy that you can brag about, regardless of effectiveness), and spending most of your money on expected utilons (chosen through sheer cold-blooded shut-up-and-multiply calculation, without worrying about status or fuzzies).

- Selecting Rationalist Groups: Trying to breed e.g. egg-laying chickens by individual selection can produce odd side effects on the farm level, since a more dominant hen can produce more egg mass at the expense of other hens. Group selection is nearly impossible in Nature, but easy to impose in the laboratory, and group-selecting hens produced substantial increases in efficiency. Though most of my essays are about individual rationality - and indeed, Traditional Rationality also praises the lone heretic more than evil Authority - the real effectiveness of "rationalists" may end up determined by their performance in groups.

- Incremental Progress and the Valley: The optimality theorems for probability theory and decision theory, are for perfect probability theory and decision theory. There is no theorem that incremental changes toward the ideal, starting from a flawed initial form, must yield incremental progress at each step along the way. Since perfection is unattainable, why dare to try for improvement? But my limited experience with specialized applications suggests that given enough progress, one can achieve huge improvements over baseline - it just takes a lot of progress to get there.

- Whining-Based Communities: Many communities feed emotional needs by offering their members someone or something to blame for failure - say, those looters who don't approve of your excellence. You can easily imagine some group of "rationalists" congratulating themselves on how reasonable they were, while blaming the surrounding unreasonable society for keeping them down. But this is not how real rationality works - there's no assumption that other agents are rational. We all face unfair tests (and yes, they are unfair to different degrees for different people); and how well you do with your unfair tests, is the test of your existence. Rationality is there to help you win anyway, not to provide a self-handicapping excuse for losing. There are no first-person extenuating circumstances. There is absolutely no point in going down the road of mutual bitterness and consolation, about anything, ever.

- Mandatory Secret Identities: This post was not well-received, but the point was to suggest that a student must at some point leave the dojo and test their skills in the real world. The aspiration of an excellent student should not consist primarily of founding their own dojo and having their own students.

- Beware of Other-Optimizing: Aspiring rationalists often vastly overestimate their own ability to optimize other people's lives. They read nineteen webpages offering productivity advice that doesn't work for them... and then encounter the twentieth page, or invent a new method themselves, and wow, it really works - they've discovered the true method. Actually, they've just discovered the one method in twenty that works for them, and their confident advice is no better than randomly selecting one of the twenty blog posts. Other-Optimizing is exceptionally dangerous when you have power over the other person - for then you'll just believe that they aren't trying hard enough.

- Akrasia and Shangri-La: The Shangri-La diet works amazingly well for some people, but completely fails for others, for no known reason. Since the diet has a metabolic rationale and is not supposed to require willpower, its failure in my and other cases is unambigiously mysterious. If it required a component of willpower, then I and others might be tempted to blame myself for not having willpower. The art of combating akrasia (willpower failure) has the same sort of mysteries and is in the same primitive state; we don't know the deeper rule that explains why a trick works for one person but not another.

- Collective Apathy and the Internet: The causes of bystander apathy are even worse on the Internet. There may be an opportunity here for a startup to deliberately try to avert bystander apathy in online group coordination.

- Bayesians vs. Barbarians: Suppose that a country of rationalists is attacked by a country of Evil Barbarians who know nothing of probability theory or decision theory. There's a certain concept of "rationality" which says that the rationalists inevitably lose, because the Barbarians believe in a heavenly afterlife if they die in battle, while the rationalists would all individually prefer to stay out of harm's way. So the rationalist civilization is doomed; it is too elegant and civilized to fight the savage Barbarians... And then there's the idea that rationalists should be able to (a) solve group coordination problems, (b) care a lot about other people and (c) win...

- Of Gender and Rationality: Analysis of the gender imbalance that appears in "rationalist" communities, suggesting nine possible causes of the effect, and possible corresponding solutions.

- My Way: I sometimes think of myself as being like the protagonist in a classic SF labyrinth story, wandering further and further into some alien artifact, trying to radio back a description of what I'm seeing, so that I can be followed. But what I'm finding is not just the Way, the thing that lies at the center of the labyrinth; it is also my Way, the path that I would take to come closer to the center, from whatever place I started out. And yet there is still a common thing we are all trying to find. We should be aware that others' shortest paths may not be the same as our own, but this is not the same as giving up the ability to judge or to share.

- The Sin of Underconfidence: When subjects know about a bias or are warned about a bias, overcorrection is not unheard of as an experimental result. That's what makes a lot of cognitive subtasks so troublesome - you know you're biased but you're not sure how much, and if you keep tweaking you may overcorrect. The danger of underconfidence (overcorrecting for overconfidence) is that you pass up opportunities on which you could have been successful; not challenging difficult enough problems; losing forward momentum and adopting defensive postures; refusing to put the hypothesis of your inability to the test; losing enough hope of triumph to try hard enough to win. You should ask yourself "Does this way of thinking make me stronger, or weaker?"

- Well-Kept Gardens Die By Pacifism: Good online communities die primarily by refusing to defend themselves, and so it has been since the days of Eternal September. Anyone acculturated by academia knows that censorship is a very grave sin... in their walled gardens where it costs thousands and thousands of dollars to enter. A community with internal politics will treat any attempt to impose moderation as a coup attempt (since internal politics seem of far greater import than invading barbarians). In rationalist communities this is probably an instance of underconfidence - mildly competent moderators are probably quite trustworthy to wield the banhammer. On Less Wrong, the community is the moderator (via karma) and you will need to trust yourselves enough to wield the power and keep the garden clear.

- Practical Advice Backed By Deep Theories: Practical advice is genuinely much, much more useful when it's backed up by concrete experimental results, causal models that are actually true, or valid math that is validly interpreted. (Listed in increasing order of difficulty.) Stripping out the theories and giving the mere advice alone wouldn't have nearly the same impact or even the same message; and oddly enough, translating experiments and math into practical advice seems to be a rare niche activity relative to academia. If there's a distinctive LW style, this is it.

- Less Meta: The fact that this final series was on the craft and the community seems to have delivered a push in something of the wrong direction, (a) steering toward conversation about conversation and (b) making present accomplishment pale in the light of grander dreams. Time to go back to practical advice and deep theories, then.

- Go Forth and Create the Art!: I've developed primarily the art of epistemic rationality, in particular, the arts required for advanced cognitive reductionism... arts like distinguishing fake explanations from real ones and avoiding affective death spirals. There is much else that needs developing to create a craft of rationality - fighting akrasia; coordinating groups; teaching, training, verification, and becoming a proper experimental science; developing better introductory literature... And yet it seems to me that there is a beginning barrier to surpass before you can start creating high-quality craft of rationality, having to do with virtually everyone who tries to think lofty thoughts going instantly astray, or indeed even realizing that a craft of rationality exists and that you ought to be studying cognitive science literature to create it. It's my hope that my writings, as partial as they are, will serve to surpass this initial barrier. The rest I leave to you.

Referenced by: The End (of Sequences)

Original with comments: The Craft and the Community

The End (of Sequences)

Eliezer Yudkowsky, 27 April 2009 09:07PM

This concludes the final sequence on Overcoming Bias / Less Wrong. I have not said everything I wanted to say, but I hope I have said (almost) everything I needed to say. (Such that I actually could say it in these twenty-one months of daily posting, August 2007 through April 2009.)

The project to which Less Wrong is devoted - the art and science and craft of human rationality - is, indeed, important. But the calculus of choosing among altruistic efforts is, in some ways, a calculus of who can take your place. I am more easily replaced here, than elsewhere. And so it has come time for me to begin pulling my focus away from Less Wrong, and turning toward other matters, where I am less easily replaced.

But I do need replacing - or rather, the work that I was doing needs replacing, whether by one person or by many people or by e.g. a karma system.

And so my final sequence was my letter that describes the work that I can already see remaining to be done, gives some advice on how to configure the effort, and warns direly against standard failure modes.

Any idea that can produce great enthusiasm is a dangerous idea. It may be a necessary idea, but that does not make it any less dangerous. I do fear, to a certain extent, that I will turn my focus away, and then find out that someone has picked up the ideas and run with them and gotten it all wrong...

But you can only devote your whole life to one thing at a time. In those ways I have thought to anticipate, at least, I have placed a blocking Go stone or two, and you have been warned.

I am not going to turn my attention away entirely and all at once. My initial plan is to cut back my posting to no more than one post per week.

At some future point, though, there must come a time when I turn my attention entirely away from building rationalism, and focus only on that other task.

So, yes, just to belabor the point - if there's going to be a lasting community, and not just a body of online writing that people occasionally stumble across, it needs to set itself up to run without me.

The last explicit dependency left on me is promoting posts, and I've been mostly doing that based on user voting (though not entirely; my activation threshold is lower for posts I perceive as higher-quality). I plan to start trying to delegate that power to co-editors shortly.

For myself... I've been feeling rather burned out on writing, so I'm thinking of taking a short vacation and then studying math and working over the summer, before I go back to producing a book.

I'm no longer certain about the time-investment wisdom of trying to convert the OB/LW sequences into minibooks. The Wiki might accomplish a lot of the same purpose of making the info more accessible and organizing it. We'll see how the Wiki develops (especially once my old posts are imported which should happen Any Time Now).

Now, while I'm on semi-vacation, is a good time to have me speak at your hedge fund. If you wait until after I write a book and it comes out, then, if things go remotely well, it will cost you a lot more money to have me speak (because the marginal utility to me of additional money will have gone way down, and demand gone up). Right now, though, additional money does have substantial marginal utility to me. So those of you who work at the hedge funds who were rational enough to survive: please request me for your speaker programs. I'm told that I give a wonderful, amazing, etcetera talk for audiences interested in rationality, considerably more than twice as useful as talks from big-name speakers who cost considerably more than twice as much.

If the rationality book is written, sold, and takes off, I may well vanish entirely off the face of the Earth, all purposes of publicity having already been served. That is the optimal and desirable outcome - it means I am allowed to specialize narrowly.

I would like to take this moment to confess, once again, that I have had ulterior motives in all this - as is right and proper to a rationalist. It is not the pure service of rationality that diverted my focus here... though I have served that overt cause as a thing in its own right, with its own integrity.

And I will not insist on recruiting all the rationalists that I create...

But still...

In the end, these 21 months were wagered upon your active response, not your passive possession of truth. Even those of you who are not moved to care about the thing that I protect, may still be moved to align their activities with Seasteading, or the Methuselah Foundation, or Givewell, or cryonics. And, perhaps, tell them that we sent you, so that they know that this "rationality" business is working to their favor; so that maybe they will say a thing or two about rationality. And someone who encounters the craft there, will read on it further, and eventually donate professionally specialized hours (in one form or another) to the Singularity Institute. What goes around comes around, I hope.

To the extent that you don't feel moved to do anything in particular - even include a link in your signature, or link to apposite posts now and then in your blog - then I have lost my gamble of these 21 months. Or I have lost that part of the stakes which was about you and decision processes similar to you. (No, that doesn't mean you should seize this opportunity to post about how I lost my gamble with you. You should know better by now, if you want any rationalist cause to get anything done ever, whether or not you are a part of it.)

And this advice: If there is some rationalist cause you have decided to help eventually, I advise you very strongly to help that cause now - even if it's just a tiny amount. One of the regularities I have discovered, working in the nonprofit industry, is that people who donated last year donate the next year, and people who are planning to donate next year will, next year, still be planning to donate "next year". The gap between little helpers and big helpers is a lot more permeable than the membrane that separates helpers and procrastinators. This holds whether you would help my own cause, or any of the other causes that have rationality as their common interest.

As for why Earth needs rational activists in particular - I hope that by now this has become clear. In this fragile Earth there are many tasks which are underserved by irrational altruists. Scope insensitivity and the purchase of moral satisfaction leads people to donate to puppy pounds as easily as existential risk prevention; circular altruism prevents them from going so far as to multiply utilons by probabilities; unsocialized in basic economics, they see money as a dirty thing inferior to volunteering unspecialized labor; they try to purchase warm fuzzies and status and utilons all at the same time; they feel nervous outside of conventional groups and follow the first thought that associates to "charity"...

And these are all very normal and human mistakes, to be sure - forgiveable in others, if not in yourself. Nonetheless, I will advise you that a rationalist's efforts should not be wasted on causes that are already popular far outside of rationalist circles. There is nothing remotely approaching an efficient market in utilons.

Is all this inclusiveness a pretense? Did I, in the end, gamble only upon the portion of the activism that would flow to my own cause? Yes, of course I did; that is how the calculation comes out when I shut up and multiply.

But I have faithfully served the integrity of that pretense, because that inclusiveness matters to my own cause as well.

So I say to you now, on behalf of all our causes: Do, whatever you may find worth doing.

Original with comments: The End (of Sequences)

Final Words

Eliezer Yudkowsky, 27 April 2009 09:12PM

Sunlight enriched air already alive with curiosity, as dawn rose on Brennan and his fellow students in the place to which Jeffreyssai had summoned them.

They sat there and waited, the five, at the top of the great glassy crag that was sometimes called Mount Mirror, sometimes Mount Monastery, and more often simply left unnamed. The high top and peak of the mountain, from which you could see all the lands below and seas beyond.

(Well, not all the lands below, nor seas beyond. So far as anyone knew, there was no place in the world from which all the world was visible; nor, equivalently, any kind of vision that would see through all obstacle-horizons. In the end it was the top only of one particular mountain: there were other peaks, and from their tops you would see other lands below; even though, in the end, it was all a single world.)

"What do you think comes next?" said Hiriwa. Her eyes were bright, and she gazed to the far horizons like a lord.

Taji shrugged, though his own eyes were alive with anticipation. "Jeffreyssai's last lesson doesn't have any obvious sequel that I can think of. In fact, I think we've learned just about everything that I knew the beisutsukai masters knew. What's left, then -"

"Are the real secrets," Yin completed the thought.

Hiriwa and Taji and Yin shared a grin, among themselves.

Styrlyn wasn't smiling. Brennan suspected rather strongly that Styrlyn was older than he had admitted.

Brennan wasn't smiling either. He might be young, but he kept high company, and had witnesssed some of what went on behind the curtains of the world. Secrets had their price, always, that was the barrier that made them secrets; and Brennan thought he had a good idea of what this price might be.

There was a cough from behind them, at a moment when they had all happened to be looking in any other direction but that one.

As one, their heads turned.

Jeffreyssai stood there, in a casual robe that looked more like very glassy glass than any proper sort of mirrorweave.

Jeffreyssai stood there and looked at them, a strange abiding sorrow in those inscrutable ancient eyes.

"Sen...sei," Taji started, faltering as that bright anticipation stumbled over Jeffreyssai's return look. "What's next?"

"Nothing," Jeffreyssai said abruptly. "You're finished. It's done."

Hiriwa, Taji, and Yin all blinked, a perfect synchronized gesture of shock. Then, before their expressions could turn to outrage and objections -

"Don't," Jeffreyssai said. There was real pain in it. "Believe me, it hurts me more than it hurts you." He might have been looking at them; or at something far away, or long ago. "I don't know exactly what roads may lie before you - but yes, I know you're not ready. I know I'm sending you out unprepared. I know that everything I taught you is incomplete. That what I said is not what you heard. I know that I left out the one most important thing. That the rhythm at the center of everything is missing and astray. I know that you will harm yourself in the course of trying to use what I taught; so that I, personally, will have shaped, in some fashion unknown to me, the very knife that will cut you..."

"...that's the hell of being a teacher, you see," Jeffreyssai said. Something grim flickered in his expression. "Nonetheless, you're done. Finished, for now. What lies between you and mastery is not another classroom. We are fortunate, or perhaps not fortunate, that the road to power does not wend only through lecture halls. Or the quest would be boring to the bitter end. Still, I cannot teach you; and so it is a moot point whether I would. There is no master here whose art is all inherited. Even the beisutsukai have never discovered how to teach certain things; it is possible that such an event has been prohibited. And so you can only arrive at mastery by using to the fullest the techniques you have already learned, facing challenges and apprehending them, mastering the tools you have been taught until they shatter in your hands -"

Jeffreyssai's eyes were hard, as though steeled in acceptance of unwelcome news.

"- and you are left in the midst of wreckage absolute. That is where I, your teacher, am sending you. You are not beisutsukai masters. I cannot create masters. I cannot even come close. Go, then, and fail."

"But -" said Yin, and stopped herself.

"Speak," said Jeffreyssai.

"But then why," she said helplessly, "why teach us anything in the first place?"

Brennan's eyelids flickered just the tiniest amount.

It was enough for Jeffreyssai. "Answer her, Brennan, if you think you know."

"Because," Brennan said, "if we were not taught, there would be no chance at all of our becoming masters."

"Even so," said Jeffreyssai. "If you were not taught - then when you failed, you might simply think you had reached the limits of Reason itself. You would be discouraged and bitter amid the wreckage. You might not even realize when you had failed. No; you have been shaped into something that may emerge from the wreckage of your past self, determined to remake your art. And then you will remember much that will help you. If you had not been taught, your chances would be - less." His gaze passed over the group. "It should be obvious, but understand that the moment of your crisis cannot be provoked artificially. To teach you something, the catastrophe must come as a surprise."

Brennan made the gesture with his hand that indicated a question; and Jeffreyssai nodded in reply.

"Is this the only way in which Bayesian masters come to be, sensei?"

"I do not know," said Jeffreyssai, from which the overall state of the evidence was obvious enough. "But I doubt there would ever be a road that leads only through the monastery. We are the heirs in this world of mystics as well as scientists, just as the Competitive Conspiracy inherits from chessplayers alongside cagefighters. We have turned our impulses to more constructive uses - but we must still stay on our guard against old failure modes."

Jeffreyssai took a breath. "Three flaws above all are common among the beisutsukai. The first flaw is to look just the slightest bit harder for flaws in arguments whose conclusions you would rather not accept. If you cannot contain this aspect of yourself then every flaw you know how to detect will make you that much stupider. This is the challenge which determines whether you possess the art or its opposite: Intelligence, to be useful, must be used for something other than defeating itself."

"The second flaw is cleverness. To invent great complicated plans and great complicated theories and great complicated arguments - or even, perhaps, plans and theories and arguments which are commended too much by their elegance and too little by their realism. There is a widespread saying which runs: 'The vulnerability of the beisutsukai is well-known; they are prone to be too clever.' Your enemies will know this saying, if they know you for a beisutsukai, so you had best remember it also. And you may think to yourself: 'But if I could never try anything clever or elegant, would my life even be worth living?' This is why cleverness is still our chief vulnerability even after its being well-known, like offering a Competitor a challenge that seems fair, or tempting a Bard with drama."

"The third flaw is underconfidence, modesty, humility. You have learned so much of flaws, some of them impossible to fix, that you may think that the rule of wisdom is to confess your own inability. You may question yourself so much, without resolution or testing, that you lose your will to carry on in the Art. You may refuse to decide, pending further evidence, when a decision is necessary; you may take advice you should not take. Jaded cynicism and sage despair are less fashionable than once they were, but you may still be tempted by them. Or you may simply - lose momentum."

Jeffreyssai fell silent then.

He looked from each of them, one to the other, with quiet intensity.

And said at last, "Those are my final words to you. If and when we meet next, you and I - if and when you return to this place, Brennan, or Hiriwa, or Taji, or Yin, or Styrlyn - I will no longer be your teacher."

And Jeffreyssai turned and walked swiftly away, heading back toward the glassy tunnel that had emitted him.

Even Brennan was shocked. For a moment they were all speechless.

Then -

"Wait!" cried Hiriwa. "What about our final words to you? I never said -"

"I will tell you what my sensei told me," Jeffreyssai's voice came back as he disappeared. "You can thank me after you return, if you return. One of you at least seems likely to come back."

"No, wait, I -" Hiriwa fell silent. In the mirrored tunnel, the fractured reflections of Jeffreyssai were already fading. She shook her head. "Never... mind, then."

There was a brief, uncomfortable silence, as the five of them looked at each other.

"Good heavens," Taji said finally. "Even the Bardic Conspiracy wouldn't try for that much drama."

Yin suddenly laughed. "Oh, this was nothing. You should have seen my send-off when I left Diamond Sea University." She smiled. "I'll tell you about it sometime - if you're interested."

Taji coughed. "I suppose I should go back and... pack my things..."

"I'm already packed," Brennan said. He smiled, ever so slightly, when the other three turned to look at him.

"Really?" Taji asked. "What was the clue?"

Brennan shrugged with careful carelessness. "Beyond a certain point, it is futile to inquire how a beisutsukai master knows a thing -"

"Come off it!" Yin said. "You're not a beisutsukai master yet."

"Neither is Styrlyn," Brennan said. "But he has already packed as well." He made it a statement rather than a question, betting double or nothing on his image of inscrutable foreknowledge.

Styrlyn cleared his throat. "As you say. Other commitments call me, and I have already tarried longer than I planned. Though, Brennan, I do feel that you and I have certain mutual interests, which I would be happy to discuss with you -"

"Styrlyn, my most excellent friend, I shall be happy to speak with you on any topic you desire," Brennan said politely and noncommitally, "if we should meet again." As in, not now. He certainly wasn't selling out his Mistress this early in their relationship.

There was an exchange of goodbyes, and of hints and offers.

And then Brennan was walking down the road that led toward or away from Mount Monastery (for every road is a two-edged sword), the smoothed glass pebbles clicking under his feet.

He strode out along the path with purpose, vigor, and determination, just in case someone was watching.

Some time later he stopped, stepped off the path, and wandered just far enough away to prevent anyone from finding him unless they were deliberately following.

Then he sagged wearily back against a tree-trunk. It was a sparse clearing, with only a few trees poking out of the ground; not much present in the way of distracting scenery, unless you counted the red-tinted stream flowing out of a dark cave-mouth. And Brennan deliberately faced away from that, leaving only the far grey of the horizons, and the blue sky and bright sun.

Now what?

He had thought that the Bayesian Conspiracy, of all the possible trainings that existed in this world, would have cleared up his uncertainty about what to do with the rest of his life.

Power, he'd sought at first. Strength to prevent a repetition of the past. "If you don't know what you need, take power" - so went the proverb. He had gone first to the Competitive Conspiracy, then to the beisutsukai.

And now...

Now he felt more lost than ever.

He could think of things that made him happy. But nothing that he really wanted.

The passionate intensity that he'd come to associate with his Mistress, or with Jeffreyssai, or the other figures of power that he'd met... a life of pursuing small pleasures seemed to pale in comparison, next to that.

In a city not far from the center of the world, his Mistress waited for him (in all probability, assuming she hadn't gotten bored with her life and run away). But to merely return, and then drift aimlessly, waiting to fall into someone else's web of intrigue... no. That didn't seem like... enough.

Brennan plucked a blade of grass from the ground and stared at it, half-unconsciously looking for anything interesting about it; an old, old game that his very first teacher had taught him, what now seemed like ages ago.

Why did I believe that going to Mount Mirror would tell me what I wanted?

Well, decision theory did require that your utility function be consistent, but...

If the beisutsukai knew what I wanted, would they even tell me?

At the Monastery they taught doubt. So now he was falling prey to the third besetting sin of which Jeffreyssai had spoken: lost momentum, indeed. For he had learned to question the image that he held of himself in his mind.

Are you seeking power because that is your true desire, Brennan?

Or because you have a picture in your mind, of the role that you play as an ambitious young man, and you think it is what someone playing your role would do?

Almost everything he'd done up until now, even going to Mount Mirror, had probably been the latter.

And when he blanked out the old thoughts and tried to see the problem as though for the first time...

...nothing much came to mind.

What do I want?

Maybe it wasn't reasonable to expect the beisutsukai to tell him outright. But was there anything they had taught him by which he might answer?

Brennan closed his eyes and thought.

First, suppose there is something I would passionately desire. Why would I not know what it is?

Because I have not yet encountered it, or ever imagined it?

Or because there is some reason I would not admit it to myself?

Brennan laughed out loud, then, and opened his eyes.

So simple, once you thought of it that way. So obvious in retrospect. That was what they called a silver-shoes moment, and yet, if he hadn't gone to Mount Mirror, it wouldn't ever have occurred to him.

Of course there was something he wanted. He knew exactly what he wanted. Wanted so desperately he could taste it like an sharp tinge on his tongue.

It just hadn't come to mind earlier, because... if he acknowledged his desire explicitly... then he also had to see that it was difficult. High, high, above him. Far out of his reach. "Impossible" was the word that came to mind, though it was not, of course, impossible.

But once he asked himself if he preferred to wander aimlessly through his life - once it was put that way, the answer became obvious. Pursuing the unattainable would make for a hard life, but not a sad one. He could think of things that made him happy, either way. And in the end - it was what he wanted.

Brennan stood up, and took his first steps, in the exact direction of Shir L'or, the city that lies in the center of the world. He had a plot to hatch, and he did not know who would be part of it.

And then Brennan stumbled, when he realized that Jeffreyssai had already known.

One of you at least seems likely to come back...

Brennan had thought he was talking about Taji. Taji had probably thought he was talking about Taji. It was what Taji said he wanted. But how reliable of an indicator was that, really?

There was a proverb, though, about that very road he had just left: Whoever sets out from Mount Mirror seeking the impossible, will surely return.

When you considered Jeffreyssai's last warning - and that the proverb said nothing of succeeding at the impossible task itself - it was a less optimistic saying than it sounded.

Brennan shook his head wonderingly. How could Jeffreyssai possibly have known before Brennan knew himself?

Well, beyond a certain point, it is futile to inquire how a beisutsukai master knows a thing -

Brennan halted in mid-thought.

No.

No, if he was going to become a beisutsukai master himself someday, then he ought to figure it out.

It was, Brennan realized, a stupid proverb.

So he walked, and this time, he thought about it carefully.

As the sun was setting, red-golden, shading his footsteps in light.

Original with comments: Final Words

Special Status Needs Special Support

Eliezer Yudkowsky, 04 May 2009 10:59PM

I just recorded another BHTV with Adam Frank, though it's not out yet, and I had a thought that seems worth recording. At a certain point in the dialogue, Adam Frank was praising the wisdom and poetry in religion. I retorted, "Tolkien's got great poetry, and some parts that are wise and some that are unwise; but you don't see people wearing little rings around their neck in memory of Frodo."

(I don't remember whether this observation is original to me, so if anyone knows a prior source for this exact wording, please comment it!)

The general structure of this critique is that Frank wants to assign a special status to the Book of Job, but he gives a reason that would be equally applicable to The Lord of the Rings (good poetry and some wise parts). So if those are his real reasons, he should feel just the same way about God and Gandalf. Or if not that exact particular book, then some other work of poetic fiction that was always understood to be poetic fiction.

Later on I did demand of Adam Frank to say whether he thought the Book of Job ought to be assigned any different status from The Merchant of Venice, and Frank did reply "No". I'm not sure that he lives up to this reply, frankly. I strongly suspect he grants the two works a different emotional status. One is widely revered as Sacred Religious Truth while the other is merely a Great Work of Literature. Frank, while not a religious believer himself, does have different modes of thought for Sacred Truth and Great Literature and he knows that Job is supposed to be Sacred Truth.

When I challenged the sacredness of the Book of Job, Frank reacted by trying to praise Job's "great poetry", which positive affect then seems to justify the positive-affect sacred status via the affect heuristic / halo effect. But "great poetry" would apply to Tolkien as well; and yet if you talked about Tolkien the way that Frank talked about Job, most people would write you down as a hopeless fanboy/fangirl...

So the general form of the bias that I'm critiquing is to try and justify a special positive (negative) status by pointing to positive (negative) attributes, saying, "Therefore I can assign it this very positive status!", but the same attributes belong to many other works that you don't grant the special positive status.

Other places to watch out for this would be if, say, you thought that Morton Smerdley was the greatest genius ever, and someone called on you to justify this, and you replied "Morton Smerdley became a Math Professor at just the age of 27" - but there are other people who became math professors at 27, or even 26, and yet you don't feel the special reverence toward them that you attach to Smerdley.

Original with comments: Special Status Needs Special Support

Wiki.lesswrong.com Is Live

Eliezer Yudkowsky, 06 May 2009 05:17AM

http://wiki.lesswrong.com/ is now live, for all our Wiki needs. The previous Wikia wiki has been imported. Knock yourself out on linking there from comments or blog posts (yes, you will have to link manually, there is no CamelCase convention yet on the blog and comments).

See here for proposed wiki usage guidelines - note that these do not yet seem to appear in the Wiki itself, hint hint.

Original with comments: Wiki.lesswrong.com Is Live

Willpower Hax #487: Execute by Default

Eliezer Yudkowsky, 12 May 2009 06:46AM

This is a trick that I use for getting out of bed in the morning - quite literally: I count down from 10 and get out of bed after the "1".

It works because instead of deciding to get out of bed, I just have to decide to implement the plan to count down from 10 and then get out of bed. Once the plan is in motion, the final action no longer requires an effortful decision - that's the theory, anyway. And to start the plan doesn't require as much effort because I just have to think "10, 9..."

As usual with such things, there's no way to tell whether it works because it's based on any sort of realistic insight or if it works because I believe it works; and in fact this is one of those cases that blurs the boundary between the two.

The technique was originally inspired by reading some neurologist suggesting that what we have is not "free will" so much as "free won't": that is, frontal reflection is mainly good for suppressing the default mode of action, more than originating new actions.

Pondering that for a bit inspired the idea that - if the brain carries out certain plans by default - it might conserve willpower to first visualize a sequence of actions and try to 'mark' it as the default plan, and then lift the attention-of-decision that agonizes whether or not to do it, thus allowing that default to happen.

For the record, I can remember a time some years ago when I would have been all enthusiastic about this sort of thing, believing that I had discovered this incredible new technique that might revolutionize my whole daily life. Today, while I know that there are discoverables with that kind of power, I also know that it usually requires beginning from firmer foundations - reports of controlled experiments, a standard theory in the field, and maybe even math. On the scale of depth I now use, this sort of trick ranks as pretty shallow - and in fact I really do use it just for getting out of bed.

I offer this trick as an example of practical advice not backed by deep theories, of the sort that you can find on a hundred other productivity blogs. At best it may work for some of you some of the time. Consider yourself warned about the enthusiasm thing.

Original with comments: Willpower Hax #487: Execute by Default

"Open-Mindedness" - the video

Eliezer Yudkowsky, 14 May 2009 06:17AM

An interesting little Flash-like video on "openmindedness" by someone named QualiaSoup (hopefully ironically).

Does anyone know how much effort is required to produce this sort of video, perhaps from a script? We need at least another thousand of these.

Original with comments: "Open-Mindedness" - the video

"What Is Wrong With Our Thoughts"

Eliezer Yudkowsky, 17 May 2009 07:24AM

"But let us never forget, either, as all conventional history of philosophy conspires to make us forget, what the 'great thinkers' really are: proper objects, indeed, of pity, but even more, of horror."

David Stove's "What Is Wrong With Our Thoughts" is a critique of philosophy that I can only call epic.

The astute reader will of course find themselves objecting to Stove's notion that we should be catologuing every possible way to do philosophy wrong. It's not like there's some originally pure mode of thought, being tainted by only a small library of poisons. It's just that there are exponentially more possible crazy thoughts than sane thoughts, c.f. entropy.

But Stove's list of 39 different classic crazinesses applied to the number three is absolute pure epic gold. (Scroll down about halfway through if you want to jump there directly.)

I especially like #8: "There is an integer between two and four, but it is not three, and its true name and nature are not to be revealed."

Original with comments: "What Is Wrong With Our Thoughts"

This Failing Earth

Eliezer Yudkowsky, 24 May 2009 04:09PM

Suppose I told you about a certain country, somewhere in the world, in which some of the cities have degenerated into gang rule. Some such cities are ruled by a single gang leader, others have degenerated into almost complete lawlessness. You would probably conclude that the cities I was talking about were located inside what we call a "failed state".

So what does the existence of North Korea say about this Earth?

No, it's not a perfect analogy. But the thought does sometimes occur to me, to wonder if the camel has two humps. If there are failed Earths and successful Earths, in the great macroscopic superposition popularly known as "many worlds" - and we're not one of the successful. I think of this as the "failed Earth" hypothesis.

Of course the camel could also have three or more humps, and it's quite easy to imagine Earths that are failing much worse than this, epic failed Earths ruled by the high-tech heirs of Genghis Khan or the Catholic Church. Oh yes, it could definitely be worse...

...and the "failed state" analogy is hardly perfect; "failed state" usually refers to failure to integrate into the global economy, but a failed Earth is not failing to integrate into anything larger...

...but the question does sometimes haunt me, as to whether in the alternative Everett branches of Earth, we could identify a distinct cluster of "successful" Earths, and we're not in it. It may not matter much in the end; the ultimate test of a planet's existence probably comes down to Friendly AI, and Friendly AI may come down to nine people in a basement doing math. I keep my hopes up, and think of this as a "failing Earth" rather than a "failed Earth".

But it's a thought that comes to mind, now and then. Reading about the ongoing Market Complexity Collapse and wondering if this Earth failed to solve one of the basic functions of global economics, in the same way that Rome, in its later days, failed to solve the problem of orderly transition of power between Caesars.

Of course it's easy to wax moralistic about people who aren't solving their coordination problems the way you like. I don't mean this to degenerate into a standard diatribe about the sinfulness of this Earth, the sort of clueless plea embodied perfectly by Simon and Garfunkel:

I dreamed I saw a mighty room
The room was filled with men
And the paper they were signing said
They'd never fight again

It's a cheap pleasure to wax moralistic about failures of global coordination.

But visualizing the alternative Everett branches of Earth, spread out and clustered - for me, at least, that seems to help trigger my mind into a non-Simon-and-Garfunkel mode of thinking. If the successful Earths lack a North Korea, how did they get there? Surely not just by signing a piece of paper saying they'd never fight again.

Indeed, our Earth's Westphalian concept of sovereign states is the main thing propping up Somalia and North Korea. There was a time when any state that failed that badly would be casually conquered by a more successful neighbor. So maybe the successful Earths don't have a Westphalian concept of sovereignty; maybe our Earth's concept of inviolable borders represents a failure to solve one of the key functions of a planetary civilization.

Maybe the successful Earths are the ones where the ancient Greeks, or equivalent thereof, had the "Aha!" of Darwinian evolution... and at least one country started a eugenics program that successfully selected for intelligence, well in advance of nuclear weapons being developed. If that makes you uncomfortable, it's meant to - the successful Earths may not have gotten there through Simon and Garfunkel. And yes, of course the ancient Greeks attempting such a policy could and probably would have gotten it terribly wrong; maybe the epic failed Earths are the ones where some group had the Darwinian insight and then successfully selected for prowess as warriors. I'm not saying "Go eugenics!" would have been a systematically good idea for ancient Greeks to try as policy...

But maybe the top cluster of successful Earths, among human Everett branches, stumbled into that cluster because some group stumbled over eugenic selection for intelligence, and then, being a bit smarter, realized what it was they were doing right, so that the average IQ got up to 140 well before anyone developed nuclear weapons. (And then conquered the world, rather than respecting the integrity of borders.)

What would a successful Earth look like? How high is their standard sanity waterline? Are there large organized religions in successful Earths - is their presence here a symptom of our failure to solve the problems of a planetary civilization? You can ring endless changes on this theme, and anyone with an accustomed political hobbyhorse is undoubtedly imagining their pet Utopia already. For my own part, I'll go ahead and wonder, if there's an identifiable "successful" cluster among the human Earths, what percentage of them have worldwide cryonic preservation programs in place.

One point that takes some of the sting out of our ongoing muddle - at least from my perspective - is my suspicion that the Earths in the successful cluster, even those with an average IQ of 140 as they develop computers, may not be in much of a better position to really succeed, to solve the Friendly AI problem. A rising tide lifts all boats, and Friendly AI is a race between cautiously developed AI and insufficiently-cautiously-developed AI. "Successful" Earths might even be worse off, if they solve their global coordination problems well enough to put the whole world's eyes on the problem and turn the development over to prestigious bureaucrats. It's not a simple issue like cryonics that we're talking about. If, in the end, "successful Earths" of the human epoch aren't in a much better position for the catastrophically high-level pass-fail test of the posthuman transition, than our own "failing Earth"... then this Earth isn't all that much more doomed just because we screwed up our financial system, international relations, and basic rationality training.

Is such speculation at all useful? "Live in your own world", as the saying goes...

...Well, it might not be a saying here, but it's probably a saying in those successful Earths where the scientific community is long since trained in formal Bayesianism and they readily accepted the obvious truth of many-worlds... as opposed to our own world and its constantly struggling academia where senior scientists spend most of their time writing grant proposals...

(Michael Vassar has an extended thesis on how the scientific community in our Earth has been slowly dying since 1910 or so, but I'll let him decide whether it's worth his time to write up that post.)

It's usually not my intent to depress people. I have an accustomed saying that if you want to depress yourself, look at the future, and if you want to cheer yourself up, look at the past. By analogy - well, for all we know, we might be in the second-highest major cluster, or in the top 10% of all Earths even if not one of the top 1%. It might be that most Earths have global orders descended from the conquering armies of the local Church. I recently had occasion to visit the National Museum of Australia in Canberra, and it's shocking to think of how easily a human culture can spend thirty thousand years without inventing the bow and arrow. Really, we did do quite well for ourselves in a lot of ways... I think?

A sense of beleaguredness, a sense that everything is decaying and dying into sinfulness - these memes are more useful for gluing together cults than for inspiring people to solve their coordination problems.

But even so - it's a thought that I have, when I see some aspect of the world going epically awry, to wonder if we're in the cluster of Earths that fail. It's the sort of thought that inspires me, at least, to go down into that basement and solve the math problem and make everything come out all right anyway. Because if there's one thing that the intelligence explosion really messes up, it's the dramatic unity of human progress - if this were a world with a supervised course of history we'd be worrying about making it to Akon's world through a continuous developmental schema, not making a sudden left turn to solve a math problem.

It may be that in the fractiles of the human Everett branches, we live in a failing Earth - but it's not failed until someone messes up the first AI. I find that a highly motivating thought. Your mileage may vary.

Original with comments: This Failing Earth

Anime Explains the Epimenides Paradox

Eliezer Yudkowsky, 27 May 2009 09:12PM

The Epimenides Paradox or Liar Paradox is "This sentence is false." Type hierarchies are supposed to resolve the Epimenides paradox... Using an indefinitely extensible, indescribably infinite, ordinal hierarchy of meta-languages. No meta-language can contain its own truth predicate - no meta-language can talk about the "truth" or "falsity" of its own sentences - and so for every meta-language we need a meta-meta-language.

I didn't create this video and I don't know who did - but it does a pretty good job of depicting how I feel about infinite type hierarchies: namely, pretty much the same way I feel about the original Epimenides Paradox.

Bonus problem: In what language did I write the description of this video?

Original with comments: Anime Explains the Epimenides Paradox

Do Fandoms Need Awfulness?

Eliezer Yudkowsky, 28 May 2009 06:03AM

Stephen Bond, "Objects of Fandom":

...my theory is that for something to attract fans, it must have an aspect of truly monumental badness about it.

Raiders of the Lost Ark is a robust potboiler, tongue-in-cheek, very competently done. I think it's enjoyable, but even among those who don't, it's hard to see the film attracting actual derision. Boredom or irritation, probably, but nothing more. Star Wars, on the other hand.... From one perspective, it's an entertaining space opera, but from a slightly different perspective, an imperceptible twist of the glass, it's laughably awful. Utterly ridiculously bad. And it's this very badness that makes so many people take up arms in its defence.

...It's impossible to imagine a fan of Animal Farm, the Well-Tempered Clavier, or the theory of gravity. Such works can defend themselves. But badness, especially badness of an obvious, monumental variety, inspires devotion. The quality of the work, in the face of such glaring shortcomings, becomes a matter of faith -- and faith is a much stronger bond than mere appreciation. It drives fans together, gives them strength against those who sneer... And so the fan groups of Tolkien, Star Trek, Spider-man, Japanese kiddie-cartoons etc. develop an almost cult-like character.

"Uh oh," I said to myself on first reading this, "Is this why my fans are more intense than Robin Hanson's fans? And if I write a rationality book, should I actually give in to temptation and self-indulgence and write in Twelve Virtues style, just so that it has something attackable for fans to defend?"

But the second time I turned my thoughts toward this question, I performed that oft-neglected operation, asking: "I read it on the Internet, but is it actually true?" Just because it's unpleasant doesn't mean it's true. And just because it provides a bit of cynicism that would give me rationality-credit to acknowledge, doesn't mean it becomes true just so I can earn the rationality-credit.

The first counterexample that came to mind was Jack Vance. Jack Vance is a science-fiction writer who, to the best of my knowledge, I've never heard accused of any great sin (or any lesser sin, actually). He is - was - the supremely competent craftsman of SF: his words flow, his plots race, and his human cultures are odder than other authors' aliens, to say nothing of his aliens. Vance didn't have his characters give controversial political speeches like Heinlein. Vance just wrote consistently excellent science fiction.

And some of Vance's fans got together and produced the Vance Integral Edition, a complete collection of Vance in leather-bound hardcover books with high-quality long-lasting paper. They contracted to get the books printed, and when the books arrived, enough Vance fans showed up to ship them all. (They referred to themselves as "packing scum".)

That's serious fandom. Aimed at work that - like Animal Farm or the Well-Tempered Clavier - is merely excellent, without an aspect of monumental badness to defend.

Godel, Escher, Bach - maybe I'm prejudiced here, and I've heard a word or two said against it, but really, I don't think the fandom that it has stems from it being frequently attacked. On the other hand, there aren't annual conventions for fans of self-referential sentences, so maybe it's not as much of a data point as I might like.

Star Wars really did have something going for it that Raiders of the Lost Ark didn't, namely, it introduced a lot of impressionable minds to science fiction. Or space opera, if you like. The point is that the romance of space is not the romance of archeology.

On due reflection, I'm not sure that utter ridiculous monumental badness is all it's cracked up to be.

But there are annual Star Trek conventions. And there are not annual Jack Vance conventions. Douglas Hofstadter might be far more widely beloved - but Ayn Rand has more fanatic fans.

If Jack Vance had been so clever as to keep all the poetic phrasing and alien societies, but now and then have his characters make crazy political speeches - if he had deliberately introduced an aspect of monumental badness - would he now be worshiped, instead of just loved?

Can anyone think of a true, pure counterexample of a reasonably fanatic fandom (to the level of annual conventions, though not necessarily suicide bombers) of something that is just sheer good professional craftwork, and not commonly criticized? And of course the acid test is not whether you think it is just sheer good craftsmanship, but whether this is widely believed within the broad context of the relevant social community - can you have fanatic fans when their object of worship really is that good and the mainstream believes it too?

I do think that Stephen Bond's Objects of Fandom is pointing to a real effect, if not the only effect. So in the same vein that we should try to be attracted to basic science textbooks and not just poorly written press releases about "breaking news", let us try to be fans of those merely excellent works that lack an aspect of monumental awfulness to defend.

Original with comments: Do Fandoms Need Awfulness?

With whom shall I diavlog?

Eliezer Yudkowsky, 03 June 2009 03:20AM

Bloggingheads.tv can't exactly call up, say, the President of France and get him to do a diavlog, but they have some street cred with mid-rank celebrities and academics. With that in mind, how would you fill in this blank?

"I would really love to see a diavlog between Yudkowsky and ____________."

Original with comments: With whom shall I diavlog?

Honesty: Beyond Internal Truth

Eliezer Yudkowsky, 06 June 2009 02:59AM

When I expect to meet new people who have no idea who I am, I often wear a button on my shirt that says:

SPEAK THE TRUTH,
EVEN IF YOUR VOICE TREMBLES

Honesty toward others, it seems to me, obviously bears some relation to rationality. In practice, the people I know who seem to make unusual efforts at rationality, are unusually honest, or, failing that, at least have unusually bad social skills.

And yet it must be admitted and fully acknowledged, that such morals are encoded nowhere in probability theory. There is no theorem which proves a rationalist must be honest - must speak aloud their probability estimates. I have said little of honesty myself, these past two years; the art which I've presented has been more along the lines of:

SPEAK THE TRUTH INTERNALLY,
EVEN IF YOUR BRAIN TREMBLES

I do think I've conducted my life in such fashion, that I can wear the original button without shame. But I do not always say aloud all my thoughts. And in fact there are times when my tongue emits a lie. What I write is true to the best of my knowledge, because I can look it over and check before publishing. What I say aloud sometimes comes out false because my tongue moves faster than my deliberative intelligence can look it over and spot the distortion. Oh, we're not talking about grotesque major falsehoods - but the first words off my tongue sometimes shade reality, twist events just a little toward the way they should have happened...

From the inside, it feels a lot like the experience of un-consciously-chosen, perceptual-speed, internal rationalization. I would even say that so far as I can tell, it's the same brain hardware running in both cases - that it's just a circuit for lying in general, both for lying to others and lying to ourselves, activated whenever reality begins to feel inconvenient.

There was a time - if I recall correctly - when I didn't notice these little twists. And in fact it still feels embarrassing to confess them, because I worry that people will think: "Oh, no! Eliezer lies without even thinking! He's a pathological liar!" For they have not yet noticed the phenomenon, and actually believe their own little improvements on reality - their own brain being twisted around the same way, remembering reality the way it should be (for the sake of the conversational convenience at hand). I am pretty damned sure that I lie no more pathologically than average; my pathology - my departure from evolutionarily adapted brain functioning - is that I've noticed the lies.

The fact that I'm going ahead and telling you about this mortifying realization - that despite my own values, I literally cannot make my tongue speak only truth - is one reason why I am not embarrassed to wear yon button. I do think I meet the spirit well enough.

It's the same "liar circuitry" that you're fighting, or indulging, in the internal or external case - that would be my second guess for why rational people tend to be honest people. (My first guess would be the obvious: respect for the truth.) Sometimes the Eli who speaks aloud in real-time conversation, strikes me as almost a different person than the Eliezer Yudkowsky who types and edits. The latter, I think, is the better rationalist, just as he is more honest. (And if you asked me out loud, my tongue would say the same thing. I'm not that internally divided. I think.)

But this notion - that external lies and internal lies are correlated by their underlying brainware - is not the only view that could be put forth, of the interaction between rationality and honesty.

An alternative view - which I do not myself endorse, but which has been put forth forcefully to me - is that the nerd way is not the true way; and that a born nerd, who seeks to become even more rational, should allow themselves to lie, and give themselves safe occasions to practice lying, so that they are not tempted to twist around the truth internally - the theory being that if you give yourself permission to lie outright, you will no longer feel the need to distort internal belief. In this view the choice is between lying consciously and lying unconsciously, and a rationalist should choose the former.

I wondered at this suggestion, and then I suddenly had a strange idea. And I asked the one, "Have you been hurt in the past by telling the truth?" "Yes", he said, or "Of course", or something like that -

(- and my brain just flashed up a small sign noting how convenient it would be if he'd said "Of course" - how much more smoothly that sentence would flow - but in fact I don't remember exactly what he said; and if I'd been speaking out loud, I might have just said, "'Of course', he said" which flows well. This is the sort of thing I'm talking about, and if you don't think it's dangerous, you don't understand at all how hard it is to find truth on real problems, where a single tiny shading can derail a human train of thought entirely -)

- and at this I suddenly realized, that what worked for me, might not work for everyone. I haven't suffered all that much from my project of speaking truth - though of course I don't know exactly how my life would have been otherwise, except that it would be utterly different. But I'm good with words. I'm a frickin' writer. If I need to soften a blow, I can do with careful phrasing what would otherwise take a lie. Not everyone scores an 800 on their verbal SAT, and I can see how that would make it a lot harder to speak truth. So when it comes to white lies, in particular, I claim no right to judge - and also it is not my primary goal to make the people around me happier.

Another counterargument that I can see to the path I've chosen - let me quote Roger Zelazny:

"If you had a choice between the ability to detect falsehood and the ability to discover truth, which one would you take? There was a time when I thought they were different ways of saying the same thing, but I no longer believe that. Most of my relatives, for example, are almost as good at seeing through subterfuge as they are at perpetrating it. I’m not at all sure, though, that they care much about truth. On the other hand, I’d always felt there was something noble, special, and honorable about seeking truth... Had this made me a sucker for truth's opposite?"

If detecting falsehood and discovering truth are not the same skill in practice, then practicing honesty probably makes you better at discovering truth and worse at detecting falsehood. If I thought I was going to have to detect falsehoods - if that, not discovering a certain truth, were my one purpose in life - then I'd probably apprentice myself out to a con man.

What, in your view, and in your experience, is the nature of the interaction between honesty and rationality? Between external truthtelling and internal truthseeking?

Original with comments: Honesty: Beyond Internal Truth

ESR's comments on some EY:OB/LW posts

Eliezer Yudkowsky, 20 June 2009 12:16AM

Eric S. Raymond's comments on some of my Overcoming Bias posts.

In his reply to my To Lead, You Must Stand Up, he writes:

"I think your exhortations here are nearly useless. Experience I’ve collected over the last ten years suggests to me that the kind of immunity to stage fright you and I have is a function of basic personality type at the neurotransmitter-balance level, and not really learnable by most people."

This is a particularly interesting observation if combined with Hanson's hypothesis that people choke to submit.

"I disagree with The Futility of Emergence," says ESR. Yea, many have said this to me. And they go on to say: Emergence has the useful meaning that... And it's a different meaning every time. In ESR's case it's:

"The word 'emergent' is a signal that we believe a very specific thing about the relationship between 'neurons firing' and 'intelligence', which is that there is no possible account of intelligence in which the only explanatory units are neurons or subsystems of neurons."

Let me guess, you think the word "emergence" means something useful but that's not exactly it, although ESR's definition does aim in the rough general direction of what you think is the right definition...

So-called "words" like this should not be actually spoken from one human to another. It is tempting fate. It would be like trying to have a serious discussion between two theologians if both of them were allowed to say the word "God" directly, instead of always having to say whatever they meant by the word.

Original with comments: ESR's comments on some EY:OB/LW posts

Nonparametric Ethics

Eliezer Yudkowsky, 20 June 2009 11:31AM

(Inspired by a recent conversation with Robin Hanson.)

Robin Hanson, in his essay on "Minimal Morality", suggests that the unreliability of our moral reasoning should lead us to seek simple moral principles:

"In the ordinary practice of fitting a curve to a set of data points, the more noise one expects in the data, the simpler a curve one fits to that data. Similarly, when fitting moral principles to the data of our moral intuitions, the more noise we expect in those intuitions, the simpler a set of principles we should use to fit those intuitions. (This paper elaborates.)"

In "the limit of expecting very large errors of our moral intuitions", says Robin, we should follow an extremely simple principle - the simplest principle we can find that seems to compress as much morality as possible. And that principle, says Robin, is that it is usually good for people to get what they want, if no one else objects.

Now I myself carry on something of a crusade against trying to compress morality down to One Great Moral Principle. I have developed at some length the thesis that human values are, in actual fact, complex, but that numerous biases lead us to underestimate and overlook this complexity. From a Friendly AI perspective, the word "want" in the English sentence above is a magical category.

But Robin wasn't making an argument in Friendly AI, but in human ethics: he's proposing that, in the presence of probable errors in moral reasoning, we should look for principles that seem simple to us, to carry out at the end of the day. The more we distrust ourselves, the simpler the principles.

This argument from fitting noisy data, is a kind of logic that can apply even when you have prior reason to believe the underlying generator is in fact complicated. You'll still get better predictions from the simpler model, because it's less sensitive to noise.

Even so, my belief that human values are in fact complicated, leads me to two objections and an alternative proposal:

The first objection is that we do, in fact, have enough data to support moral models that are more complicated than a small set of short English sentences. If you have a thousand data points, even noisy data points, it may be a waste of evidence to try to fit them to a straight line, especially if you have prior reason to believe the true generator is not linear.

And my second fear is that people underestimate the complexity and error-proneness of the reasoning they do to apply their Simple Moral Principles. If you try to reduce morality to the Four Commandments, then people are going to end up doing elaborate, error-prone rationalizations in the course of shoehorning their real values into the Four Commandments.

But in the ordinary practice of machine learning, there's a different way to deal with noisy data points besides trying to fit simple models. You can use nonparametric methods. The classic example is k-nearest-neighbors: To predict the value at a new point, use the average of the 10 nearest points previously observed.

A line has two parameters - slope and intercept; to fit a line, we try to pick values for the slope and intercept that well-match the data. (Minimizing squared error corresponds to maximizing the likelihood of the data given Gaussian noise, for example.) Or we could fit a cubic polynomial, and pick four parameters that best-fit the data.

But the nearest-neighbors estimator doesn't assume a particular shape of underlying curve - not even that the curve is a polynomial. Technically, it doesn't even assume continuity. It just says that we think that the true values at nearby positions are likely to be similar. (If we furthermore believe that the underlying curve is likely to have continuous first and second derivatives, but don't want to assume anything else about the shape of that curve, then we can use cubic splines to fit an arbitrary curve with a smoothly changing first and second derivative.)

And in terms of machine learning, it works. It is done rather less often in science papers - for various reasons, some good, some bad; e.g. academics may prefer models with simple extractable parameters that they can hold up as the triumphant fruits of their investigation: Behold, this is the slope! But if you're trying to win the Netflix Prize, and you find an algorithm that seems to do well by fitting a line to a thousand data points, then yes, one of the next things you try is substituting some nonparametric estimators of the same data; and yes, this often greatly improves the estimates in practice. (Added: And conversely there are plenty of occasions where ridiculously simple-seeming parametric fits to the same data turn out to yield surprisingly good predictions. And lots of occasions where added complexity for tighter fits buys you very little, or even makes predictions worse. In machine learning this is usually something you find out by playing around, AFAICT.)

It seems to me that concepts like equality before the law, or even the notion of writing down stable laws in the first place, reflect a nonparametric approach to the ethics of error-prone moral reasoning.

We don't suppose that society can be governed by only four laws. In fact, we don't even need to suppose that the 'ideal' morality (obtained as the limit of perfect knowledge and reflection, etc.) would in fact subject different people and different occasions to the same laws. We need only suppose that we believe, a priori, that similar moral dilemmas are likely ceteris paribus to have similar resolutions, and that moral reasoning about adjustment to specific people is highly error-prone - that, given unlimited flexibility to 'perfectly fit' the solution to the person, we're likely to favor our friends and relatives too much. (And not in an explicit, internally and externally visible way, that we could correct just by having a new rule not to favor friends and relatives.)

So instead of trying to recreate, each time, the judgment that is the perfect fit to the situation and the people, we try to use the ethical equivalent of a cubic spline - have underlying laws that are allowed to be complicated, but have to be written down for stability, and are supposed to treat neighboring points similarly.

Nonparametric ethics says: "Let's reason about which moral situations are at least rough neighbors so that an acceptable solution to one should be at least mostly-acceptable to another; and let's reason about where people are likely to be highly biased in their attempt to adjust to specifics; and then, to reduce moral error, let's enforce similar resolutions across neighboring cases." If you think that good moral codes will treat different people similarly, and/or that people are highly biased in how they adjust their judgments to different people, then you will come up with the ethical solution of equality before the law.

Now of course you can still have laws that are too complicated, and that try to sneak in too much adaptation to particular situations. This would correspond to a nonparametric estimator that doesn't smooth enough, like using 1-nearest-neighbor instead of 10-nearest-neighbors, or like a cubic spline that tried to exactly fit every point without trying to minimize the absolute value of third derivatives.

And of course our society may not succeed at similarly treating different people in similar situations - people who can afford lawyers experience a different legal system.

But if nothing else, coming to grips with the concept of nonparametric ethics helps us see the way in which our society is failing to deal with the error-proneness of its own moral reasoning.

You can interpret a fair amount of my coming-of-age as my switch from parametric ethics to nonparametric ethics - from the pre-2000 search for simple underlying morals and my attempts to therefore reject values that seemed complicated; to my later acceptance that my values were actually going to be complicated, and that both I and my AI designs needed to come to terms with that. Friendly AI can be viewed as the problem of coming up with - not the Three Simple Laws of Robotics that are all a robot needs - but rather a regular and stable method for learning, predicting, and renormalizing human values that are and should be complicated.

Original with comments: Nonparametric Ethics

Atheism = Untheism + Antitheism

Eliezer Yudkowsky, 01 July 2009 02:19AM

One occasionally sees such remarks as, "What good does it do to go around being angry about the nonexistence of God?" (on the one hand) or "Babies are natural atheists" (on the other). It seems to me that such remarks, and the rather silly discussions that get started around them, show that the concept "Atheism" is really made up of two distinct components, which one might call "untheism" and "antitheism".

A pure "untheist" would be someone who grew up in a society where the concept of God had simply never been invented - where writing was invented before agriculture, say, and the first plants and animals were domesticated by early scientists. In this world, superstition never got past the hunter-gatherer stage - a world seemingly haunted by mostly amoral spirits - before coming into conflict with Science and getting slapped down.

Hunter-gatherer superstition isn't much like what we think of as "religion". Early Westerners often derided it as not really being religion at all, and they were right, in my opinion. In the hunter-gatherer stage the supernatural agents aren't particularly moral, or charged with enforcing any rules; they may be placated with ceremonies, but not worshipped. But above all - they haven't yet split their epistemology. Hunter-gatherer cultures don't have special rules for reasoning about "supernatural" entities, or indeed an explicit distinction between supernatural entities and natural ones; the thunder spirits are just out there in the world, as evidenced by lightning, and the rain dance is supposed to manipulate them - it may not be perfect but it's the best rain dance developed so far, there was that famous time when it worked...

If you could show hunter-gatherers a raindance that called on a different spirit and worked with perfect reliability, or, equivalently, a desalination plant, they'd probably chuck the old spirit right out the window. Because there are no special rules for reasoning about it - nothing that denies the validity of the Elijah Test that the previous rain-dance just failed. Faith is a post-agricultural concept. Before you have chiefdoms where the priests are a branch of government, the gods aren't good, they don't enforce the chiefdom's rules, and there's no penalty for questioning them.

And so the Untheist culture, when it invents science, simply concludes in a very ordinary way that rain turns out to be caused by condensation in clouds rather than rain spirits; and at once they say "Oops" and chuck the old superstitions out the window; because they only got as far as superstitions, and not as far as anti-epistemology.

The Untheists don't know they're "atheists" because no one has ever told them what they're supposed to not believe in - nobody has invented a "high god" to be chief of the pantheon, let alone monolatry or monotheism.

However, the Untheists do know that they don't believe in tree spirits. And we shall even suppose that the Untheists don't believe in tree spirits, because they have a sophisticated and good epistemology - they understand why it is in general a bad idea to postulate ontologically basic mental entities.

So if you come up to the Untheists and say:

"The universe was created by God -"

"By what?"

"By a, ah, um, God is the Creator - the Mind that chose to make the universe -"

"So the universe was created by an intelligent agent. Well, that's the standard Simulation Hypothesis, but do you have actual evidence confirming this? You sounded very certain -"

"No, not like the Matrix! God isn't in another universe simulating this one, God just... is. He's indescribable. He's the First Cause, the Creator of everything -"

"Okay, that sounds like you just postulated an ontologically basic mental entity. And you offered a mysterious answer to a mysterious question. Besides, where are you getting all this stuff? Could you maybe start by telling us about your evidence - the new observation you're trying to interpret?"

"I don't need any evidence! I have faith!"

"You have what?"

And at this very moment the Untheists have become, for the first time, Atheists. And what they just acquired, between the two points, was Antitheism - explicit arguments against explicit theism. You can be an Untheist without ever having heard of God, but you can't be an Antitheist.

Of course the Untheists are not inventing new rules to refute God, just applying their standard epistemological guidelines that their civilization developed in the course of rejecting, say, vitalism. But then that's just what we rationalist folk claim antitheism is supposed to be, in our own world: a strictly standard analysis of religion which turns out to issue a strong rejection - both epistemically and morally, and not after too much time. Every antitheist argument is supposed to be a special case of general rules of epistemology and morality which ought to have applications beyond religion - visible in the encounters of science with vitalism, say.

With this distinction in hand, you can make a bit more sense of some modern debates - for example, "Why care so much about God not existing?" could become "What is the public benefit from publicizing antitheism?" Or "What good is it to just be against something? Where is the positive agenda?" becomes "Less antitheism and more untheism in our atheism, please!" And "Babies are born atheists", which sounds a bit odd, is now understood to sound odd because babies have no grasp of antitheism.

And as for the claim that religion is compatible with Reason - well, is there a single religious claim that a well-developed, sophisticated Untheist culture would not reject? When they have no reason to suspend judgment, and no anti-epistemology of separate magisteria, and no established religions in their society to avoid upsetting?

There's nothing inherently fulfilling about arguing against Goddism - in a society of Untheists, no one would ever give the issue a second thought. But in this world, at least, insanity is not a good thing, and sanity is worth defending, and explicit antitheism by the likes of Richard Dawkins would surely be a public service conditioning on the proposition of it actually working. (Which it may in fact be doing; the next generation is growing up increasingly atheist.) Yet in the long run, the goal is an Untheistic society, not an Atheistic one - one in which the question "What's left, when God is gone?" is greeted by a puzzled look and "What exactly is missing?"

Original with comments: Atheism = Untheism + Antitheism

Jul 12 Bay Area meetup - Hanson, Vassar, Yudkowsky

Eliezer Yudkowsky, 11 July 2009 09:13PM

Just a reminder that the July 12th Overcoming Bias / Less Wrong meetup in Santa Clara, CA @7pm will feature Robin Hanson, Michael Vassar, and the Singularity Institute summer interns. This meetup will also take place in a larger house - there should be plenty of room to mingle.

Original with comments: Jul 12 Bay Area meetup - Hanson, Vassar, Yudkowsky

The Strangest Thing An AI Could Tell You

Eliezer Yudkowsky, 15 July 2009 02:27AM

Human beings are all crazy. And if you tap on our brains just a little, we get so crazy that even other humans notice. Anosognosics are one of my favorite examples of this; people with right-hemisphere damage whose left arms become paralyzed, and who deny that their left arms are paralyzed, coming up with excuses whenever they're asked why they can't move their arms.

A truly wonderful form of brain damage - it disables your ability to notice or accept the brain damage. If you're told outright that your arm is paralyzed, you'll deny it. All the marvelous excuse-generating rationalization faculties of the brain will be mobilized to mask the damage from your own sight. As Yvain summarized:

After a right-hemisphere stroke, she lost movement in her left arm but continuously denied it. When the doctor asked her to move her arm, and she observed it not moving, she claimed that it wasn't actually her arm, it was her daughter's. Why was her daughter's arm attached to her shoulder? The patient claimed her daughter had been there in the bed with her all week. Why was her wedding ring on her daughter's hand? The patient said her daughter had borrowed it. Where was the patient's arm? The patient "turned her head and searched in a bemused way over her left shoulder".

I find it disturbing that the brain has such a simple macro for absolute denial that it can be invoked as a side effect of paralysis. That a single whack on the brain can both disable a left-side motor function, and disable our ability to recognize or accept the disability. Other forms of brain damage also seem to both cause insanity and disallow recognition of that insanity - for example, when people insist that their friends have been replaced by exact duplicates after damage to face-recognizing areas.

And it really makes you wonder...

...what if we all have some form of brain damage in common, so that none of us notice some simple and obvious fact? As blatant, perhaps, as our left arms being paralyzed? Every time this fact intrudes into our universe, we come up with some ridiculous excuse to dismiss it - as ridiculous as "It's my daughter's arm" - only there's no sane doctor watching to pursue the argument any further. (Would we all come up with the same excuse?)

If the "absolute denial macro" is that simple, and invoked that easily...

Now, suppose you built an AI. You wrote the source code yourself, and so far as you can tell by inspecting the AI's thought processes, it has no equivalent of the "absolute denial macro" - there's no point damage that could inflict on it the equivalent of anosognosia. It has redundant differently-architected systems, defending in depth against cognitive errors. If one system makes a mistake, two others will catch it. The AI has no functionality at all for deliberate rationalization, let alone the doublethink and denial-of-denial that characterizes anosognosics or humans thinking about politics. Inspecting the AI's thought processes seems to show that, in accordance with your design, the AI has no intention to deceive you, and an explicit goal of telling you the truth. And in your experience so far, the AI has been, inhumanly, well-calibrated; the AI has assigned 99% certainty on a couple of hundred occasions, and been wrong exactly twice that you know of.

Arguably, you now have far better reason to trust what the AI says to you, than to trust your own thoughts.

And now the AI tells you that it's 99.9% sure - having seen it with its own cameras, and confirmed from a hundred other sources - even though (it thinks) the human brain is built to invoke the absolute denial macro on it - that...

...what?

What's the craziest thing the AI could tell you, such that you would be willing to believe that the AI was the sane one?

(Some of my own answers appear in the comments.)

Referenced by: Are You Anosognosic? • Dreams with Damaged Priors

Original with comments: The Strangest Thing An AI Could Tell You

Are You Anosognosic?

Eliezer Yudkowsky, 19 July 2009 04:35AM

Followup to: The Strangest Thing An AI Could Tell You

Brain damage patients with anosognosia are incapable of considering, noticing, admitting, or realizing even after being argued with, that their left arm, left leg, or left side of the body, is paralyzed. Again I'll quote Yvain's summary:

After a right-hemisphere stroke, she lost movement in her left arm but continuously denied it. When the doctor asked her to move her arm, and she observed it not moving, she claimed that it wasn't actually her arm, it was her daughter's. Why was her daughter's arm attached to her shoulder? The patient claimed her daughter had been there in the bed with her all week. Why was her wedding ring on her daughter's hand? The patient said her daughter had borrowed it. Where was the patient's arm? The patient "turned her head and searched in a bemused way over her left shoulder".

A brief search didn't turn up a base-rate frequency in the population for left-arm paralysis with anosognosia, but let's say the base rate is 1 in 10,000,000 individuals (so around 670 individuals worldwide).

Supposing this to be the prior, what is your estimated probability that your left arm is currently paralyzed?

Added: This interests me because it seems to be a special case of the same general issue discussed in The Modesty Argument and Robin's reply Sleepy Fools - when pathological minds roughly similar to yours update based on fabricated evidence to conclude they are not pathological, under what circumstances can you update on different-seeming evidence to conclude that you are not pathological?

Referenced by: Article upvoting

Original with comments: Are You Anosognosic?

Article upvoting

Eliezer Yudkowsky, 19 July 2009 05:03PM

Except in rare cases (like Wei Dai's Fair Division of Black-Hole Negentropy) I'm still using article upvotes to partially determine whether to promote articles to the front page - some informal mixture of "number of upvotes" + "editor's judgment". I mention this because while comment voting is still healthy, the amount of article voting seems to be dropping off. As of now I'm still drawing the inference that no one thinks "Are You Anosognosic?" worthy of promotion, or wants to see similar articles from me in the future - since other articles have at least gotten more votes than 0. But as the amount of article voting diminishes, it becomes harder to trust such inferences. Maybe people liked that article (or others I haven't promoted) and just didn't bother to upvote.

I'm posting this observation just in case people figure that upvoting articles doesn't make a difference. It does. It also encourages authors to write similar posts in the future, or alternatively not.

Original with comments: Article upvoting

Timeless Decision Theory: Problems I Can't Solve

Eliezer Yudkowsky, 20 July 2009 12:02AM

Suppose you're out in the desert, running out of water, and soon to die - when someone in a motor vehicle drives up next to you. Furthermore, the driver of the motor vehicle is a perfectly selfish ideal game-theoretic agent, and even further, so are you; and what's more, the driver is Paul Ekman, who's really, really good at reading facial microexpressions. The driver says, "Well, I'll convey you to town if it's in my interest to do so - so will you give me $100 from an ATM when we reach town?"

Now of course you wish you could answer "Yes", but as an ideal game theorist yourself, you realize that, once you actually reach town, you'll have no further motive to pay off the driver. "Yes," you say. "You're lying," says the driver, and drives off leaving you to die.

If only you weren't so rational!

This is the dilemma of Parfit's Hitchhiker, and the above is the standard resolution according to mainstream philosophy's causal decision theory, which also two-boxes on Newcomb's Problem and defects in the Prisoner's Dilemma. Of course, any self-modifying agent who expects to face such problems - in general, or in particular - will soon self-modify into an agent that doesn't regret its "rationality" so much. So from the perspective of a self-modifying-AI-theorist, classical causal decision theory is a wash. And indeed I've worked out what seems like an elegant theory, tentatively labeled "timeless decision theory", which covers these three Newcomblike problems and delivers a first-order answer that is already reflectively consistent, without need to explicitly consider such notions as "precommitment". Unfortunately this "timeless decision theory" would require a long sequence to write up, and it's not my current highest writing priority unless someone offers to let me do a PhD thesis on it.

However, there are some other timeless decision problems for which I do not possess a general theory.

For example, there's a problem introduced to me by Gary Drescher's marvelous Good and Real (OOPS: The below formulation was independently invented by Vladimir Nesov; Drescher's book actually contains a related dilemma in which box B is transparent, and only contains $1M if Omega predicts you will one-box whether B appears full or empty, and Omega has a 1% error rate) which runs as follows:

Suppose Omega (the same superagent from Newcomb's Problem, who is known to be honest about how it poses these sorts of dilemmas) comes to you and says:

"I just flipped a fair coin. I decided, before I flipped the coin, that if it came up heads, I would ask you for $1000. And if it came up tails, I would give you $1,000,000 if and only if I predicted that you would give me $1000 if the coin had come up heads. The coin came up heads - can I have $1000?"

Obviously, the only reflectively consistent answer in this case is "Yes - here's the $1000", because if you're an agent who expects to encounter many problems like this in the future, you will self-modify to be the sort of agent who answers "Yes" to this sort of question - just like with Newcomb's Problem or Parfit's Hitchhiker.

But I don't have a general theory which replies "Yes". At the point where Omega asks me this question, I already know that the coin came up heads, so I already know I'm not going to get the million. It seems like I want to decide "as if" I don't know whether the coin came up heads or tails, and then implement that decision even if I know the coin came up heads. But I don't have a good formal way of talking about how my decision in one state of knowledge has to be determined by the decision I would make if I occupied a different epistemic state, conditioning using the probability previously possessed by events I have since learned the outcome of... Again, it's easy to talk informally about why you have to reply "Yes" in this case, but that's not the same as being able to exhibit a general algorithm.

Another stumper was presented to me by Robin Hanson at an OBLW meetup. Suppose you have ten ideal game-theoretic selfish agents and a pie to be divided by majority vote. Let's say that six of them form a coalition and decide to vote to divide the pie among themselves, one-sixth each. But then two of them think, "Hey, this leaves four agents out in the cold. We'll get together with those four agents and offer them to divide half the pie among the four of them, leaving one quarter apiece for the two of us. We get a larger share than one-sixth that way, and they get a larger share than zero, so it's an improvement from the perspectives of all six of us - they should take the deal." And those six then form a new coalition and redivide the pie. Then another two of the agents think: "The two of us are getting one-eighth apiece, while four other agents are getting zero - we should form a coalition with them, and by majority vote, give each of us one-sixth."

And so it goes on: Every majority coalition and division of the pie, is dominated by another majority coalition in which each agent of the new majority gets more pie. There does not appear to be any such thing as a dominant majority vote.

(Robin Hanson actually used this to suggest that if you set up a Constitution which governs a society of humans and AIs, the AIs will be unable to conspire among themselves to change the constitution and leave the humans out in the cold, because then the new compact would be dominated by yet other compacts and there would be chaos, and therefore any constitution stays in place forever. Or something along those lines. Needless to say, I do not intend to rely on such, but it would be nice to have a formal theory in hand which shows how ideal reflectively consistent decision agents will act in such cases (so we can prove they'll shed the old "constitution" like used snakeskin.))

Here's yet another problem whose proper formulation I'm still not sure of, and it runs as follows. First, consider the Prisoner's Dilemma. Informally, two timeless decision agents with common knowledge of the other's timeless decision agency, but no way to communicate or make binding commitments, will both Cooperate because they know that the other agent is in a similar epistemic state, running a similar decision algorithm, and will end up doing the same thing that they themselves do. In general, on the True Prisoner's Dilemma, facing an opponent who can accurately predict your own decisions, you want to cooperate only if the other agent will cooperate if and only if they predict that you will cooperate. And the other agent is reasoning similarly: They want to cooperate only if you will cooperate if and only if you accurately predict that they will cooperate.

But there's actually an infinite regress here which is being glossed over - you won't cooperate just because you predict that they will cooperate, you will only cooperate if you predict they will cooperate if and only if you cooperate. So the other agent needs to cooperate if they predict that you will cooperate if you predict that they will cooperate... (...only if they predict that you will cooperate, etcetera).

On the Prisoner's Dilemma in particular, this infinite regress can be cut short by expecting that the other agent is doing symmetrical reasoning on a symmetrical problem and will come to a symmetrical conclusion, so that you can expect their action to be the symmetrical analogue of your own - in which case (C, C) is preferable to (D, D). But what if you're facing a more general decision problem, with many agents having asymmetrical choices, and everyone wants to have their decisions depend on how they predict that other agents' decisions depend on their own predicted decisions? Is there a general way of resolving the regress?

On Parfit's Hitchhiker and Newcomb's Problem, we're told how the other behaves as a direct function of our own predicted decision - Omega rewards you if you (are predicted to) one-box, the driver in Parfit's Hitchhiker saves you if you (are predicted to) pay $100 on reaching the city. My timeless decision theory only functions in cases where the other agents' decisions can be viewed as functions of one argument, that argument being your own choice in that particular case - either by specification (as in Newcomb's Problem) or by symmetry (as in the Prisoner's Dilemma). If their decision is allowed to depend on how your decision depends on their decision - like saying, "I'll cooperate, not 'if the other agent cooperates', but only if the other agent cooperates if and only if I cooperate - if I predict the other agent to cooperate unconditionally, then I'll just defect" - then in general I do not know how to resolve the resulting infinite regress of conditionality, except in the special case of predictable symmetry.

You perceive that there is a definite note of "timelessness" in all these problems.

Any offered solution may assume that a timeless decision theory for direct cases already exists - that is, if you can reduce the problem to one of "I can predict that if (the other agent predicts) I choose strategy X, then the other agent will implement strategy Y, and my expected payoff is Z", then I already have a reflectively consistent solution which this margin is unfortunately too small to contain.

(In case you're wondering, I'm writing this up because one of the SIAI Summer Project people asked if there was any Friendly AI problem that could be modularized and handed off and potentially written up afterward, and the answer to this is almost always "No", but this is actually the one exception that I can think of. (Anyone actually taking a shot at this should probably familiarize themselves with the existing literature on Newcomblike problems - the edited volume "Paradoxes of Rationality and Cooperation" should be a sufficient start (and I believe there's a copy at the SIAI Summer Project house.)))

Referenced by: Ingredients of Timeless Decision Theory • Timeless Decision Theory and Meta-Circular Decision Theory

Original with comments: Timeless Decision Theory: Problems I Can't Solve

Of Exclusionary Speech and Gender Politics

Eliezer Yudkowsky, 21 July 2009 07:22AM

I suspect that the ick reaction being labeled "objectification" actually has more to do with the sense that the speaker is addressing a closed group that doesn't include you.

Suppose I wrote a story about a man named Frank, whose twin brother (Frank has learned) is in the process of being framed for murder this very night. Frank is in the middle of a complicated plot to give his brother an alibi. He's already found the cabdriver and tricked him into waiting outside a certain apartment for an hour. Now all he needs is the last ingredient of his plan - a woman to go home with him (as he poses as his brother). Frank is, with increasing desperation, propositioning ladies at the bar - any girl will do for his plan, it doesn't matter who she is or what she's about...

I'd bet I could write that story without triggering the ick reaction, because Frank is an equal-opportunity manipulator - he manipulated the cabdriver, too. The story isn't about Frank regarding women as things on the way to implementing his plan, it's about Frank regarding various people, men and women alike, as means to the end of saving his brother.

If a woman reads that story, I think, she won't get a sense of being excluded from the intended audience.

I suspect that's what the ick factor being called "objectification" is really about - the sense that someone who says "...but you'll still find women alluring" is talking to an audience that doesn't include you, a woman. It doesn't matter if you happen to be a bi woman. You still get the sense that it never crossed the writer's mind that there might be any women in the audience, and so you are excluded.

In general, starting from a perceptual reaction, it is a difficult cognitive task to say in words exactly why that reaction occurred - to accurately state the necessary and sufficient conditions for its triggering. If the reaction is affective, a good or bad reaction, there is an additional danger: You'll be tempted to zoom in on any bad (good) aspect of the situation, and say, "Ah, that must be the reason it's bad (good)!" It's wrong to treat people as means rather than ends, right? People have their own feelings and inner life, and it's wrong to forget that? Clearly, that's a problem with saying, "And this is how you get girls..." But is that exactly what went wrong originally - what triggered the original ick reaction?

And this (I say again) is a tricky cognitive problem in general - the introspective jump from the perceptual to the abstract. It is tricky far beyond the realms of gender...

But I do suspect that the real problem is speech that makes a particular gender feel excluded. And if that's so, then for the purposes of Less Wrong, I think, it may make sense to zoom in on that speech property. Politics of all sorts have always been a dangerous bit of attractive flypaper, and I think we've had a sense, on Less Wrong, that we ought to steer clear of it - that politics is the mindkiller. And so I hope that no one will feel that their gender politics are being particularly targeted, if I suggest that, like some other political issues, we might want to steer sort of clear of that.

I've previously expressed that to build a rationalist community sustainable over time, the sort of gender imbalance that appears among e.g. computer programmers, is not a good thing to have. And so it may make sense, as rationalists qua rationalists, to target gender-exclusionary speech. To say, "Less Wrong does not want to make any particular gender feel unwelcome."

But I also think that you can just have a policy like that, without opening the floor to discussion of all gender politics qua gender politics. Without having a position on whether, say, "privilege" is a useful way to think about certain problems, or a harmful one.

And the coin does have two sides. It is possible to make men, and not just women, feel unwelcome as a gender. It is harder, because men have fewer painful memories of exclusion to trigger. A single comment by a woman saying "All men are idiots" won't do it. But if you've got a conversational thread going between many female posters all agreeing that men are privileged idiots, then a man can start to pick up a perceptual impression of "This is not a place where I'm welcome; this is a women's locker room." And LW shouldn't send that message, either.

So if we're going to do this, then let's have a policy which says that we don't want to make either gender feel unwelcome. And that aside from this, we're not saying anything official about gender politics qua gender politics. And indeed we might even want to discourage gender-political discussion, because it's probably not going to contribute to our understanding of systematic and general methods of epistemic and instrumental rationality, which is our actual alleged topic around here.

But even if we say we're just going to have a non-declarative procedural rule to avoid language or behavior that makes a gender feel excluded... it still takes us into thorny waters.

After all, jumping on every tiny hint - say, objecting to the Brennan stories because Brennan is male - will make men feel unwelcome; that this is a blog only for people who agree with feminist politics; that men have to tiptoe while women are allowed to tapdance...

Now with that said: the point is to avoid language that makes someone feel unwelcome. So if someone says that they felt excluded as a gender, pay attention. The issue is not how to prove they're "wrong". Just listen to the one who heard you, when they tell you what they heard. We want to avoid any or either gender, feeling excluded and leaving. So it is the impression that is the key thing. You can argue, perhaps, that the one's threshold for offense was set unforgivably low, that they were listening so hard that no one could whisper softly enough. But not argue that they misunderstood you. For that is still a fact about your speech and its consequences. We shall just try to avoid certain types of misunderstanding, not blame the misunderstander.

And what if someone decides she's offended by all discussion of evolutionary psychology because that's a patriarchal plot...?

Well... I think there's something to be said here, about her having impugned the honor of female rationalists everywhere. But let a female rationalist be the one to say it. And then we can all downvote the comment into oblivion.

And if someone decides that all discussion of the PUA (pickup artist) community, makes her feel excluded...?

Er... I have to say... I sort of get that one. I too can feel the locker-room ambiance rising off it. Now, yes, we have a lot of men here who are operating in gender-imbalanced communities, and we have men here who are nerds; and if you're the sort of person who reads Less Wrong, there is a certain conditional probability that you will be the sort of person who tries to find a detailed manual that solves your problems...

...while not being quite sane enough to actually notice you're driving away the very gender you're trying to seduce from our nascent rationalist community, and consequentially shut up about PUA...

...oh, never mind. Gender relations much resembles the rest of human existence, in that it largely consists of people walking around with shotguns shooting off their own feet. In the end, PUA is not something we need to be talking about here, and if it's giving one entire gender the wrong vibes on this website, I say the hell with it.

And if someone decides that it's not enough that a comment has been downvoted to -5; it needs to be banned, or the user needs to be banned, in order to signify that this website is sufficiently friendly...?

Sorry - downvoting to -5 should be enough to show that the community disapproves of this lone commenter.

If someone demands explicit agreement with their-favorite-gender-politics...?

Then they're probably making the other gender feel unwelcome - the coin does have two sides.

If someone argues against gay marriage...?

Respond not to trolls; downvote to oblivion without a word. That's not gender politics, it's kindergarten.

If you just can't seem to figure out what's wrong with your speech...?

Then just keep on accepting suggested edits. If you literally don't understand what you're doing wrong, then realize that you have a blind spot and need to steer around it. And if you do keep making the suggested edits, I think that's as much as someone could reasonably ask of you. We need a bit more empathy in all directions here, and that includes empathy for the hapless plight of people who just don't get it, and who aren't going to get it, but who are still doing what they can.

If you just can't get someone to agree with your stance on explicit gender politics...?

Take it elsewhere, both of you, please.

Is it clear from this what sort of general policy I'm driving at? What say you?

Original with comments: Of Exclusionary Speech and Gender Politics

The Hero With A Thousand Chances

Eliezer Yudkowsky, 31 July 2009 04:25AM

"Allow me to make sure I have this straight," the hero said. "I've been untimely ripped from my home world to fight unspeakable horrors, and you say I'm here because I'm lucky?"

Aerhien dipped her eyelashes in elegant acknowledgment; and quietly to herself, she thought: Thirty-seven. Thirty-seven heroes who'd said just that, more or less, on arrival.

Not a sign of the thought showed on her outward face, where the hero could see, or the other council members of the Eerionnath take note. Over the centuries since her accidental immortality she'd built a reputation for serenity, more or less because it seemed to be expected.

"There are kinds and kinds of luck," Aerhien said serenely. "Not every person desires their personal happiness above all else. Those who are lucky in aiding others, those whose luck is great in succor and in rescue, these ones are not always happy themselves. You are here, hero, because you have a hero's luck. The boy whose dusty heirloom sword proves to be magical. The peasant girl who finds herself the heir to a great kingdom. Those who discover, in time of sudden stress, an untrained wild magic within themselves. Success born not of learning, not of skill, not of determination, but unplanned coincidence and fortunes of birth: That is a hero's luck."

"Gosh," said the hero after a long, awkward pause, "thanks for the compliment."

"It is not a compliment," Aerhien said, "but this is: that you have taken good advantage of your luck. Our enemy does not speak, we do not know if there is any aliveness in it to think; but it learns, or seems to learn. We have never won against it using the same trick twice. It is rare now that a hero succeeds in conceiving a genuinely new trick, for we have fought this shadow long under our sun. For this reason we have taken to summoning heroes from distant dimensions with other modes of thought; sometimes one such knows a truly new technique, and at least they fight differently. But far more often, hero, the hero wins by luck."

"Huh," said the hero. He frowned; more in thought, it seemed, than in displeasure. "How... very odd. I wonder why that is. What kind of enemy can be defeated only by luck?"

"A nameless enemy and null," said Aerhien. "Structureless and empty, horrible and dark, the most terrifying thing imaginable: We call it Dust. That seems to be its only desire, to tear down every bit of structure in the world, grind it into specks of perfect chaos. Always the Dust is defeated, always it takes a new shape immune to its last defeat."

"I wonder," murmured the hero, "if it will run out of shapes, and then end; or if it will finally become invincible."

(One of the other Eerionnath shuddered.)

"I do not know," Aerhien said simply. "I do not know the nature of the Dust, nor the nature of the Counter-Force that opposes it. The Dust is terrible and our world should long since have ended. We are not fools enough to believe we could be lucky so many times by chance alone. But the Counter-Force has never acted openly; it never reveals itself except in - a hero's luck. And so we, the council Eerionnath to prevent the world from destruction, are at your disposal to command; and all the power and resource that this world holds, for your battle."

And she, Aerhien, and the council Eerionnath, bowed low.

Then they waited to see if the hero would demand dominions or slaves as payment, before condescending to rescue a people in distress.

If so they would dispose of him, and summon another.

This one, though, seemed to have at least some qualities of a true hero; his face showed no avarice, only an abstracted puzzlement. "A hidden Counter-Force..." he murmured. "Excuse me, but this is all very vague. Can you give me a specific example of a hero's luck?"

Aerhien opened her mouth, and then the breath caught in her throat; suddenly and involuntarily, her memory went back to that huge spell gone out of control which had blasted the then-form of the Dust, killed the hero her lover, ruined their home and country, and rendered her accidentally immortal, all those centuries ago -

Ghandhol, the second-oldest of the council, must have guessed her silent distress; for he spoke up to cover the gap: "There was a certain time," he said gravely, "when the hero of that age, sent off the entire army of the world in a diversionary attack against the strongest fortification of the enemy. While he, with but a single friend, walked directly into enemy territory, carrying undefended the single most valuable magic the Dust could possibly gain. Then the Dust captured and corrupted the hero's mind. And when all seemed absolutely lost, they only won because - in an event that was no part at all of their original plan - a hungry creature bit off the hero's finger and then accidentally fell into an open lava flow, which in turn caused -"

"That was an extreme case," said one of the younger councilors; that one looked a bit nervous, lest this hero get the wrong idea. "None since have tried to imitate the Volcano Suicide Hero -"

"Ah!" said the hero in a tone of sudden enlightenment.

Then the hero frowned. "Oh, dear..." he said under his breath.

The councilors looked at one another in mute puzzlement. The hairs pricked on Aerhien's neck; she had lived long enough to have seen almost everything at least once before. And her lover had frowned, just like that, an instant before his spell went wild.

The hero's brow was furrowed like a father whose child has just asked a question which has an answer, but whose answer no child can understand. "Do you..." he said at last. "Do you have knowledge... about the khanfhighur... that's not even translating, is it. Do you know about... the things that things are made of? And are the things constantly splitting all the time? Not singly, but in - in groups -"

The other councilors Eerionnath were staring at him in mute incomprehension. But Aerhien, who had been through it all before, gravely shook her head. "We do not possess that knowledge; nor do we know why our sun burns, or why the sky is red, or what makes a word a spell; nor has any summoned hero succeeded in raveling them." Aerhien held up her hand. "Hand, made of fingers; beneath the finger, skin and muscle and vein, beneath the muscle, sharrak and flom. That is the limit of our knowledge. Some worlds, it seems, are harder to ravel than others."

The hero waved it off. "No, it doesn't matter - well, it matters a great deal, but not for now. I only asked to see if I could get confirmation... it doesn't matter."

Aerhien waited patiently; they were rare, this sort of hero, but the more distant and alien sort did sometimes treat her world as a puzzle to be solved. She usually sought those similar enough in body and mind to feel empathy for her people's plight; but sometimes she thought of the great victory won by the Icky Blob Hero, and wondered if she should look further afield.

"What would happen if the Dust won?" asked the hero. "Would the whole world be destroyed in a single breath?"

Aerhien's brow quirked ever so slightly. "No," she said serenely. Then, because the question was strange enough to demand a longer answer: "The Dust expands slowly, using territory before destroying it; it enslaves people to its service, before slaying them. The Dust is patient in its will to destruction."

The hero flinched, then bowed his head. "I suppose that was too much to hope for; there wasn't really any reason to hope, except hope... it's not required by the logic of the situation, alas..."

Suddenly the hero looked up sharply; there was a piercing element, now, in his gaze. "There's a great deal you're neglecting to tell me about this heroing business. Were you planning to mention that the 'hero' which your council chooses and anoints, often turns out not to be the real hero at all? That the Counter-Force often ends up working through someone else entirely?"

The members of the council traded glances. "You didn't exactly ask about that," said Ghandhol mildly.

The hero nodded. "I suppose not. And the Volcano Suicide Hero - what exactly happened to him, that caused no hero to ever dare tempt fate so much again, in the history you remember?"

"His home country was ruined," Aerhien said softly, "while the army marched elsewhere on his diversion. It threw him into a misery from which he never recovered, until one day he set sail in a ship and did not return."

The hero nodded. "Poor payment, one would think, for saving the world." The hero's face grew grim, and his voice became solemn and formal, mimicking Aerhien's cadences. "But the Counter-Force is not the pure power of Good. It seems to care only and absolutely about stopping the Dust. It cares nothing for heroes, or countries, or innocent lives and victims. If it could save a thousand children from death, only by nudging the fall of a pebble, it would not bother; it has had such opportunities, and not acted."

Ghufhus, the youngest member of the council, grimaced, looking offended. "How is it our right to ask for more?" he demanded. "That we are saved from the Dust is miracle enough -"

Ghufhus stopped, noticing then that the other Eerionnath were sitting frozen. Even Aerhien's mask of dispassion had cracked.

"Ah..." Ghufhus said, puzzled. "How do you... know all this? Is there a Counter-Force in your own world?"

Fool, Aerhien thought to herself. The hero had seemed puzzled by the idea, at first, and had needed to ask for examples. She decided then and there that Ghufhus would meet with an accident before the next council meeting; their world had no room for stupid Eerionnath.

And the hero himself shook his head. "No," the hero said. "You have never summoned a hero who remembers a Counter-Force like yours."

This was also true.

"Nor will you ever," the hero added, "unless you try some way of seeking that specifically, in your summoning. It would never happen by accident."

Aerhien willed her stiff lips to move. It should have been wonderful news, but the hero himself seemed anything but happy. "You... have fathomed the nature of the Counter-Force?"

The hero nodded.

"And?" Aerhien said. "What is the rest of it? The part you are still considering whether to tell us?"

Ghandhol's eyebrows went up a tiny fraction, and his head tilted ever so slightly toward her, signaling his surprise and appreciation.

The hero hesitated. Then he sighed.

"The Counter-Force isn't going to help you this time. No hero's luck. Nothing but creativity and any scraps of real luck - and true random chance is as liable to hurt you as the Dust. Even if you do survive this time, the Counter-Force won't help you next time either. Or the time after that. What you remember happening before - will not happen for you ever again."

Aerhien felt the nausea; like a blow to the pit of her stomach it felt, the end of the world. The rest of the council Eerionnath seemed torn between fear and skepticism; but her own instincts, honed over long centuries, left little room for doubt. The distant heroes sometimes knew things... and sometimes guessed wrong. But after a hero had been right a few times, you learned to listen to that one, even if you couldn't understand the reasons or the logic...

"Why?" Ghufhus said, sounding skeptical. "Why would the Counter-Force work all this time, and then suddenly -"

Ghandhol interrupted with the far more urgent question. "How can we restore the Counter-Force?"

"You can't," said the hero.

There was a remote sadness in his eyes, the only sign that he knew exactly what he was saying.

"Then you have pronounced the absolute doom of this world," Ghandhol said heavily.

And then the hero smiled, and it was twisted and grim and defiant, all at the same time. "Oh... not quite absolute doom. In my own world, we have our own notions about heroes, which are not about heroic luck. One of us said: a hero is someone who can stand there at the moment when all hope is dead, and look upon the abyss without flinching. Another said: a superhero is someone who can save people who could not be saved by any ordinary means; whether it is few people or many people, a superhero is someone who can save people who cannot be saved. We shall try a little of my own world's style of heroism, then. Your world cannot be saved by any ordinary means; it is doomed. Like a child born with a fatal disease; it contained the seed of its own death from the beginning. Your annihilation is not an unlucky chance to be prevented, or an unpleasant possibility to avert. It is your destiny that has already been written from the beginning. You are the walking dead, and this is a dead world spinning, and many other worlds like this one are already destroyed."

"But this world is going to live anyway. I have decided it."

"That is my own world's heroism."

"How?" Aerhien said simply. "How can our world live, if what you say is true?"

The hero's eyes had gone unfocused, his face somewhat slack. "You will deliver to me the record of every single hero that your history remembers. You will bring historians here for my consultation. Your world cannot survive if it must fight this battle over and over again, with the Dust growing stronger each time. It is my thought that on this attempt, we must neutralize the Dust once and for all -"

"Do you think that hasn't been tried?" Ghufhus demanded incredulously.

The hero smiled that twisted smile again. "Ah, but if you had succeeded, you would not have needed to summon me, now would you? Though I am not quite sure that is valid logic, in a case like this... But it does seem that none of the other heroes fathomed your Counter-Force, which puts an upper limit on their perception." The hero nodded to himself. "All things have a pattern. Bring me the records, and I will see if I can fathom this Dust, and the limit of its learning ability - there must be a limit, or no amount of luck could ever save you. All things have a cause: If something like the Dust came into existence once, perhaps a true Counter-Force can be created to oppose it. Those are the ideas that occur to me in the first thirty seconds, at any rate. I must study. Bring me your keepers of knowledge. They will be my army."

Aerhien bowed, in truth this time, and very low, and the Eerionnath bowed with her. "Command and we shall obey, hero," she said simply.

The hero turned from her, and looked out the window at the red sky, and the small dots on the land that were the homes of the innocents to be protected.

"Don't call me that," he said, and it was a command. "You can call me that after we've won."

"But -"

It was Ghufhus who said it, and Aerhien promised herself that if it was a stupid question, his accident would be a painful one.

"But what is - what was the Counter-Force?"

Aerhien wavered, then decided against it.

It might not matter now, but she also wanted to know.

The hero sighed. "It's a long story," he said. "And to be frank, if you're to understand this properly, there's a lot of other things I have to explain first before I get to the ahntharhapik principle."

Original with comments: The Hero With A Thousand Chances

Unspeakable Morality

Eliezer Yudkowsky, 04 August 2009 05:57AM

It is a general and primary principle of rationality, that we should not believe that which there is insufficient reason to believe; likewise, a principle of social morality that we should not enforce upon our fellows a law which there is insufficient justification to enforce.

Nonetheless, I've always felt a bit nervous about demanding that people be able to explain things in words, because, while I happen to be pretty good at that, most people aren't.

"I remember this paper I wrote on existentialism. My teacher gave it back with an F. She’d underlined true and truth wherever it appeared in the essay, probably about twenty times, with a question mark beside each. She wanted to know what I meant by truth." —Danielle Egan (journalist)

This experience permanently traumatized Ms. Egan, by the way. Because years later, at a WTA conference, one of the speakers said that something was true, and Ms. Egan said "What do you mean, 'true'?", and the speaker gave some incorrect answer or other; and afterward I quickly walked over to Ms. Egan and explained the correspondence theory of truth: "The sentence 'snow is white' is true if and only if snow is white"; if you're using a bucket of pebbles to count sheep then an empty bucket is true if and only if the pastures are empty. I don't know if this cured her; I suspect that it didn't. But up until that point, at any rate, it seems Ms. Egan had been so traumatized by this childhood experience that she believed there was no such thing as truth - that because her teacher had demanded a definition in words, and she hadn't been able to give a good definition in words, that no good definition existed.

Of which I usually say: "There was a time when no one could define gravity in exquisitely rigorous detail, but if you walked off a cliff, you would fall."

On the other hand - it is a general and primary principle of rationality that when you have no justification, it is very important that there be some way of saying "Oops", losing hope, and just giving up already. (I really should post, at some point, on how the ability to just give up already is one of the primary distinguishing abilities of a rationalist.) So, really, if you find yourself totally unable to justify something in words, one possibility is that there is no justification. To ignore this and just casually stroll along, would not be a good thing.

And with moral questions, this problem is doubled and squared. For any given person, the meaning of "right" is a huge complicated function, not explicitly believed so much as implicitly embodied. And if we keep asking "Why?", at some point we end up replying "Because that is just what the term 'right', means; there is no pure essence of rightness that you can abstract away from the specific content of your values."

But if you were allowed to answer this in response to any demand for justification, and have the other bow and walk away - well, you would no longer be computing what we know as morality, where 'right' does mean some things and not others.

Not to mention that in questions of public policy, it ought to require some overlap in values to make a law. I do think that human values often overlap enough that different people can legitimately use the same word 'right' to refer to that-which-they-compute. But if someone wants a legal ban on pepperoni pizza because it's inherently wrong, then I may feel impelled to ask, "Why do you think this is part of the overlap in our values?"

Demands for moral justification have their Charybdis and their Scylla:

The traditionally given Charybdis is letting someone say that interracial marriage should be legally banned because it "feels icky" to them. We could call this "the unwisdom of repugnance" - if you can just say "That feels repugnant" and win a case for public intervention, then you lose all the cases of what we now regard as tremendous moral progress, which made someone feel vaguely icky at the time; women's suffrage, divorces, atheists not being burned at the stake. Moral progress - which I currently see as an iterative process of learning new facts, processing new arguments, and becoming more the sort of person you wished you were - demands that people go on thinking about morality, for which purpose it is very useful to have people go on arguing about morality. If saying the word "intuition" is a moral trump card, then people, who, by their natures, are lazy, will just say "intuition!" all the time, believing that no one is allowed to question that or argue with it; and that will be the end of their moral thinking.

And the Scylla, I think, was excellently presented by Silas Barta when... actually this whole comment is just worth quoting directly:

Let's say we're in an alternate world with strong, codified rules about social status and authority, but weak, vague, unspoken norms against harm that nevertheless keep harm at a low level.

Then let's say you present the people of this world with this "dilemma" to make Greene's point:

Say your country is at war with another country that is particularly aggressive and willing to totally demolish your social order and enslave your countrymen. In planning how to best fight off this threat, your President is under a lot of stress. To help him relieve his stress, he orders a citizen, Bob, to be brought before him and tortured and murdered, while the President laughs his head off at the violence.

He feels much more relieved and so is able to craft and motivate a war plan that leads to the unconditional surrender of the enemy. The President promises that this was just a one-time thing he had to do to handle the tremendous pressure he was under to win the war and protect his people. Bob's family, in turn, says that they are honored by the sacrifice Bob has made for his country. Everyone agrees that the President is the legitimate ruler of the country and the Constitution and tradition give him authority to do what he did to Bob.

Was it okay for the President to torture and kill Bob for his personal enjoyment?

Then, because of the deficiency in the vocabulary of "harms", you would get responses like:

"Look, I can't explain why, but obviously, it's wrong to torture and kill someone for enjoyment. No disrespect to the President, of course."

"What? I don't get it. Why would the President order a citizen killed? There would be outrage. He'd feel so much guilt that it wouldn't even relieve the stress you claim it does."

"Yeah, I agree the President has authority to do that, but God, it just burns me up to think about someone getting tortured like that for someone else's enjoyment, even if it is our great President."

Would you draw the same conclusion Greene does about these responses?

Unfortunately, it does happen to be a fact that most people are not good at explaining themselves in words, unless they've already heard the explanation from someone else. Even if you challenge a professional philosopher who holds a position, to justify it, and they can't... well, frankly, you can't conclude much even from that, in terms of inferring that no good explanation exists. Philosophers, I've observed, are not much good at this sort of job either. It's Bayesian evidence, by the law of conservation of evidence; if a good explanation would be a sign that justification exists, then the absence of such explanation must be evidence that justification does not exist. It's just not very strong evidence, because we don't strongly anticipate that even professional philosophers will be able to put a justification into words, correctly and convincingly, when justification does in fact exist.

Even conditioning on the proposition that there is overlap in what you and others mean by 'right' - the huge function that is what-we-try-to-do - and that the judgment in question is stable when taken to the limits of knowledge, thought, and reflective coherence - well, it's still not sure that you'd be able to put it into words. You might be able to. But you might not.

And we also have to allow a certain probability of convincing-sounding complicated verbal justification, in cases where no justification exists. But then if you use that as an excuse to flush all disliked arguments down the toilet, you shall be left rotting forever in a pit of convenient skepticism, saying, "All that intellekshual stuff could be wrong, after all."

So here are my proposed rules of conduct for arguing morality in words:

- "Intuition" is not a trump card. If you had to spell out what your intuition was, and where it came from (evolution? culture?), and whether it has consequences beyond itself, it's possible that we would find it unconvincing in the stark light of reflection; that we would wish to intuit some other intuition than this. We can't hold up the intuition for reflective judgment unless we know what it is. So spelling it out, is important; and if you can win arguments by saying "Intuition!" then no one will bother to spell things out any more. Please try to say what sort of intuition it is.

- "I can't put it into words" is believable to some extent, but constitutes weak evidence against the existence of valid justification. If this is a popular debate and no one on your side, politician or philosopher or interested scientist or eloquent blogger, is able to give a convincing justification in words, then that is stronger evidence that no good justification exists. The longer the failure continues, the stronger the evidence.

- Still, at the end of the day, we don't really expect people to be very good at verbalizing moral intuitions, especially since most of them have incoherent explicit metaethics. So if you can give a justification for your political policy that stutters off into incoherence only at the point of explaining why pain is a bad thing - if you can give reasonable arguments for everything else up until that point - that's probably about as much as we can demand of anyone short of a full-fledged master reductionist.

- But we also expect that people may pass judgments that they would revoke in the light of better information or new arguments; and, especially before passing to that limit, it may be that sociopaths do not overlap with the values shared by most in a society. So if A says that event B is inherently wrong and awful, and C disagrees on the grounds that it just doesn't seem all that awful to them, then the burden of argument needs to lie on A before any social, legal, public action is brought into play. We should bear in mind that people of the past would have a lot of icky feelings about things that we, today, think are not only permitted but virtuous or even mandatory - the challenging of these icky feelings for good and sufficient public justification, was a key element of their relinquishment, which we regard as moral progress.

Original with comments: Unspeakable Morality

Dreams with Damaged Priors

Eliezer Yudkowsky, 08 August 2009 10:31PM

Dreaming is the closest I've gotten to testing myself against the challenge of maintaining rationality under brain damage. So far, my trials have exhibited mixed results.

In one memorable dream a few years ago, I dreamed that the Wall Street Journal had published an article about "Eliezer Yudkowsky", but it wasn't me, it was a different "Eliezer Yudkowsky", and in the dream I wondered if I needed to write a letter to clarify this. Then I realized I was dreaming within the dream... and worried to myself, still dreaming: "But what if the Wall Street Journal really does have an article about an 'Eliezer Yudkowsky' who isn't me?"

But then I thought: "Well, the probability that I would dream about a WSJ article like that, given that a WSJ article like that actually exists in this morning's paper, is the same as the probability that I would have such a dream, given that no such article is in this morning's paper. So by Bayes's Theorem, the dream isn't evidence one way or the other. Thus there's no point in trying to guess the answer now - I'll find out in the morning whether there's an article like that." And, satisfied, my mind went back to ordinary sleep.

I find it fascinating that I was able to explicitly apply Bayes's Theorem in my sleep to correctly compute the 1:1 likelihood ratio, but my dreaming mind didn't notice the damaged prior - didn't notice that the prior probability was too low to justify raising the hypothesis to my attention.

At this point even I must concede that there is something to the complaint that, in real-world everyday life, Bayesians dispense too little advice about how to compute priors. With a damaged intuition for the weight of evidence, my dreaming mind was able to explicitly compute a likelihood ratio and correct itself. But with a damaged intuition for the prior probability, my mind didn't successfully check itself, or even notice a problem - didn't get as far as asking "But what is the prior probability?"

On July 20 I had an even more dramatic dream - sparking this essay - when I dreamed that I'd googled my own name and discovered that one of my OBLW articles had been translated into German and published, without permission but with attribution, in a special issue of the Journal of Applied Logic to commemorate the death of Richard Thaler (don't worry, he is in fact still alive)...

Then I half woke-up... and wondered if maybe one of my OBLW articles really had been "borrowed" this way. But I reasoned, in my half-awake state, that the dream couldn't be evidence because it didn't form part of a causal chain wherein the outside environment impressed itself onto my brain, and that only actual sensory impressions of Google results could form the base of a legitimate chain of inferences.

So - still half-asleep - I wanted to get out of bed and actually look at Google, to see if a result turned up for the Journal of Applied Logic issue.

And several times I fell back asleep and dreamed I'd looked at Google and seen the result; but each time on half-awaking I thought: "No, I still seem to be in bed; that was a dream, not a sense-impression, so it's not valid evidence - I still need to actually look at Google." And the cycle continued.

By the time I woke up entirely, my brain had fully switched on and I realized that the prior probability was tiny; and no, I did not bother to check the actual Google results. Though I did Google to check whether Richard Thaler was alive, since I was legitimately unsure of that when I started writing this post.

If my dreaming brain had been talking in front of an audience, that audience might have applauded the intelligent-sounding sophisticated reasoning about what constituted evidence - which was even correct, so far as it went. And yet my half-awake brain didn't notice that at the base of the whole issue was a big complicated specific hypothesis whose prior probability fell off a cliff and vanished. EliezerDreaming didn't try to measure the length of the message, tot up the weight of burdensome details, or even explicitly ask, "What is the prior probability?"

I'd mused before that the state of being religious seemed similar to the state of being half-asleep. But my recent dream made me wonder if the analogy really is a deep one. Intelligent theists can often be shepherded into admitting that their argument X is not valid evidence. Intelligent theists often confess explicitly that they have no supporting evidence - just like I explicitly realized that my dreams offered no evidence about the actual Wall Street Journal or the Journal of Applied Logic. But then they stay "half-awake" and go on wondering whether the dream happens to be true. They don't "wake up completely" and realize that, in the absence of evidence, the whole thing has a prior probability too low to deserve specific attention.

My dreaming brain can, in its sleep, reason explicitly about likelihood ratios, Bayes's Theorem, cognitive chains of causality, permissible inferences, strong arguments and non-arguments. And yet still maintain a dreaming inability to reasonably evaluate priors, to notice burdensome details and sheer ridiculousness. If my dreaming brain's behavior is a true product of dissociation - of brainware modules or software modes that can be independently switched on or off - then the analogy to religion may be more than surface similarity.

Conversely it could just be a matter of habits playing out in in my dreaming self; that I habitually pay more attention to arguments than priors, or habitually evaluate arguments deliberately but priors intuitively.

Original with comments: Dreams with Damaged Priors

Singularity Summit 2009 (quick post)

Eliezer Yudkowsky, 16 August 2009 11:29PM

Someone else may do a more formal announcement later, but since early registration expires on August 20th, I'm doing a quick heads-up to Less Wrong readers:

The Singularity Summit 2009 is in New York on Oct 3-4.

There are discounts for students, blog mentions, referrals, and registration before August 20th.

Speakers of note to rationalists will include Robin Hanson, Gary Drescher (author of Good and Real, one of the few master-level works of reductionism out there), and David Chalmers. Also speaking will be Marcus Hutter and Juergen Schmidhuber, as well as some of the usual suspects: Aubrey de Grey, Peter Thiel, Ben Goertzel, and Ray Kurzweil.

They're really trying to raise the intellectual level this year.

Singularity Summit 2009 home page, program, and registration.

Original with comments: Singularity Summit 2009 (quick post)

Ingredients of Timeless Decision Theory

Eliezer Yudkowsky, 19 August 2009 01:10AM

Followup to: Newcomb's Problem and Regret of Rationality, Towards a New Decision Theory

Wei Dai asked:

"Why didn't you mention earlier that your timeless decision theory mainly had to do with logical uncertainty? It would have saved people a lot of time trying to guess what you were talking about."

...

All right, fine, here's a fast summary of the most important ingredients that go into my "timeless decision theory". This isn't so much an explanation of TDT, as a list of starting ideas that you could use to recreate TDT given sufficient background knowledge. It seems to me that this sort of thing really takes a mini-book, but perhaps I shall be proven wrong.

The one-sentence version is: Choose as though controlling the logical output of the abstract computation you implement, including the output of all other instantiations and simulations of that computation.

The three-sentence version is: Factor your uncertainty over (impossible) possible worlds into a causal graph that includes nodes corresponding to the unknown outputs of known computations; condition on the known initial conditions of your decision computation to screen off factors influencing the decision-setup; compute the counterfactuals in your expected utility formula by surgery on the node representing the logical output of that computation.

To obtain the background knowledge if you don't already have it, the two main things you'd need to study are the classical debates over Newcomblike problems, and the Judea Pearl synthesis of causality. Canonical sources would be "Paradoxes of Rationality and Cooperation" for Newcomblike problems and "Causality" for causality.

For those of you who don't condescend to buy physical books, Marion Ledwig's thesis on Newcomb's Problem is a good summary of the existing attempts at decision theories, evidential decision theory and causal decision theory. You need to know that causal decision theories two-box on Newcomb's Problem (which loses) and that evidential decision theories refrain from smoking on the smoking lesion problem (which is even crazier). You need to know that the expected utility formula is actually over a counterfactual on our actions, rather than an ordinary probability update on our actions.

I'm not sure what you'd use for online reading on causality. Mainly you need to know:

- That a causal graph factorizes a correlated probability distribution into a deterministic mechanism of chained functions plus a set of uncorrelated unknowns as background factors.

- Standard ideas about "screening off" variables (D-separation).

- The standard way of computing counterfactuals (through surgery on causal graphs).

It will be helpful to have the standard Less Wrong background of defining rationality in terms of processes that systematically discover truths or achieve preferred outcomes, rather than processes that sound reasonable; understanding that you are embedded within physics; understanding that your philosophical intutions are how some particular cognitive algorithm feels from inside; and so on.

The first lemma is that a factorized probability distribution which includes logical uncertainty - uncertainty about the unknown output of known computations - appears to need cause-like nodes corresponding to this uncertainty.

Suppose I have a calculator on Mars and a calculator on Venus. Both calculators are set to compute 123 * 456. Since you know their exact initial conditions - perhaps even their exact initial physical state - a standard reading of the causal graph would insist that any uncertainties we have about the output of the two calculators, should be uncorrelated. (By standard D-separation; if you have observed all the ancestors of two nodes, but have not observed any common descendants, the two nodes should be independent.) However, if I tell you that the calculator at Mars flashes "56,088" on its LED display screen, you will conclude that the Venus calculator's display is also flashing "56,088". (And you will conclude this before any ray of light could communicate between the two events, too.)

If I was giving a long exposition I would go on about how if you have two envelopes originating on Earth and one goes to Mars and one goes to Venus, your conclusion about the one on Venus from observing the one on Mars does not of course indicate a faster-than-light physical event, but standard ideas about D-separation indicate that completely observing the initial state of the calculators ought to screen off any remaining uncertainty we have about their causal descendants so that the descendant nodes are uncorrelated, and the fact that they're still correlated indicates that there is a common unobserved factor, and this is our logical uncertainty about the result of the abstract computation. I would also talk for a bit about how if there's a small random factor in the transistors, and we saw three calculators, and two showed 56,088 and one showed 56,086, we would probably treat these as likelihood messages going up from nodes descending from the "Platonic" node standing for the ideal result of the computation - in short, it looks like our uncertainty about the unknown logical results of known computations, really does behave like a standard causal node from which the physical results descend as child nodes.

But this is a short exposition, so you can fill in that sort of thing yourself, if you like.

Having realized that our causal graphs contain nodes corresponding to logical uncertainties / the ideal result of Platonic computations, we next construe the counterfactuals of our expected utility formula to be counterfactuals over the logical result of the abstract computation corresponding to the expected utility calculation, rather than counterfactuals over any particular physical node.

You treat your choice as determining the result of the logical computation, and hence all instantiations of that computation, and all instantiations of other computations dependent on that logical computation.

Formally you'd use a Godelian diagonal to write:

Argmax[A in Actions] in Sum[O in Outcomes](Utility(O)*P(this computation yields A []-> O|rest of universe))

(where P( X=x []-> Y | Z ) means computing the counterfactual on the factored causal graph P, that surgically setting node X to x, leads to Y, given Z)

Setting this up correctly (in accordance with standard constraints on causal graphs, like noncircularity) will solve (yield reflectively consistent, epistemically intuitive, systematically winning answers to) 95% of the Newcomblike problems in the literature I've seen, including Newcomb's Problem and other problems causing CDT to lose, the Smoking Lesion and other problems causing EDT to fail, Parfit's Hitchhiker which causes both CDT and EDT to lose, etc.

Note that this does not solve the remaining open problems in TDT (though Nesov and Dai may have solved one such problem with their updateless decision theory). Also, although this theory goes into much more detail about how to compute its counterfactuals than classical CDT, there are still some visible incompletenesses when it comes to generating causal graphs that include the uncertain results of computations, computations dependent on other computations, computations uncertainly correlated to other computations, computations that reason abstractly about other computations without simulating them exactly, and so on. On the other hand, CDT just has the entire counterfactual distribution rain down on the theory as mana from heaven (e.g. James Joyce, Foundations of Causal Decision Theory), so TDT is at least an improvement; and standard classical logic and standard causal graphs offer quite a lot of pre-existing structure here. (In general, understanding the causal structure of reality is an AI-complete problem, and so in philosophical dilemmas the causal structure of the problem is implicitly given in the story description.)

Among the many other things I am skipping over:

- Some actual examples of where CDT loses and TDT wins, EDT loses and TDT wins, both lose and TDT wins, what I mean by "setting up the causal graph correctly" and some potential pitfalls to avoid, etc.

- A rather huge amount of reasoning which defines reflective consistency on a problem class; explains why reflective consistency is a rather strong desideratum for self-modifying AI; why the need to make "precommitments" is an expensive retreat to second-best and shows lack of reflective consistency; explains why it is desirable to win and get lots of money rather than just be "reasonable" (that is conform to pre-existing intuitions generated by a pre-existing algorithm); which notes that, considering the many pleas from people who want, but can't find any good intermediate stage between CDT and EDT, it's a fascinating little fact that if you were rewriting your own source code, you'd rewrite it to one-box on Newcomb's Problem and smoke on the smoking lesion problem...

- ...and so, having given many considerations of desirability in a decision theory, shows that the behavior of TDT corresponds to reflective consistency on a problem class in which your payoff is determined by the type of decision you make, but not sensitive to the exact algorithm you use apart from that - that TDT is the compact way of computing this desirable behavior we have previously defined in terms of reflectively consistent systematic winning.

- Showing that classical CDT, given self-modification ability, modifies into a crippled and inelegant form of TDT.

- Using TDT to fix the non-naturalistic behavior of Pearl's version of classical causality in which we're supposed to pretend that our actions are divorced from the rest of the universe - the counterfactual surgery, written out Pearl's way, will actually give poor predictions for some problems (like someone who two-boxes on Newcomb's Problem and believes that box B has a base-rate probability of containing a million dollars, because the counterfactual surgery says that box B's contents have to be independent of the action). TDT not only gives the correct prediction, but explains why the counterfactual surgery can have the form it does - if you condition on the initial state of the computation, this should screen off all the information you could get about outside things that affect your decision; then your actual output can be further determined only by the Godel-diagonal formula written out above, permitting the formula to contain a counterfactual surgery that assumes its own output, so that the formula does not need to infinitely recurse on calling itself.

- An account of some brief ad-hoc experiments I performed on IRC to show that a majority of respondents exhibited a decision pattern best explained by TDT rather than EDT or CDT.

- A rather huge amount of exposition of what TDT decision theory actually corresponds to in terms of philosophical intuitions, especially those about "free will". For example, this is the theory I was using as hidden background when I wrote in "Causality and Moral Responsibility" that factors like education and upbringing can be thought of as determining which person makes a decision - that you rather than someone else makes a decision - but that the decision made by that particular person is up to you. This corresponds to conditioning on the known initial state of the computation, and performing the counterfactual surgery over its output. I've actually done a lot of this exposition on OBLW without explicitly mentioning TDT, like Timeless Control and Thou Art Physics for reconciling determinism with choice (actually effective choice requires determinism, but this confuses humans for reasons given in Possibility and Could-ness). But if you read the other parts of the solution to "free will", and then furthermore explicitly formulate TDT, then this is what utterly, finally, completely, and without even a tiny trace of confusion or dissatisfaction or a sense of lingering questions, kills off entirely the question of "free will".

- Some concluding chiding of those philosophers who blithely decided that the "rational" course of action systematically loses; that rationalists defect on the Prisoner's Dilemma and hence we need a separate concept of "social rationality"; that the "reasonable" thing to do is determined by consulting pre-existing intuitions of reasonableness, rather than first looking at which agents walk away with huge heaps of money and then working out how to do it systematically; people who take their intuitions about free will at face value; assuming that counterfactuals are fixed givens raining down from the sky rather than non-observable constructs which we can construe in whatever way generates a winning decision theory; et cetera. And celebrating of the fact that rationalists can cooperate with each other, vote in elections, and do many other nice things that philosophers have claimed they can't. And suggesting that perhaps next time one should extend "rationality" a bit more credit before sighing and nodding wisely about its limitations.

- In conclusion, rational agents are not incapable of cooperation, rational agents are not constantly fighting their own source code, rational agents do not go around helplessly wishing they were less rational, and finally, rational agents win.

Those of you who've read the quantum mechanics sequence can extrapolate from past experience that I'm not bluffing. But it's not clear to me that writing this book would be my best possible expenditure of the required time.

Referenced by: Timeless Decision Theory and Meta-Circular Decision Theory

Original with comments: Ingredients of Timeless Decision Theory

Timeless Decision Theory and Meta-Circular Decision Theory

Eliezer Yudkowsky, 20 August 2009 10:07PM

(This started as a reply to Gary Drescher's comment here in which he proposes a Metacircular Decision Theory (MCDT); but it got way too long so I turned it into an article, which also contains some amplifications on TDT which may be of general interest.)

Part 1: How timeless decision theory does under the sort of problems that Metacircular Decision Theory talks about.

Say we have an agent embodied in the universe. The agent knows some facts about the universe (including itself), has an inference system of some sort for expanding on those facts, and has a preference scheme that assigns a value to the set of facts, and is wired to select an action--specifically, the/an action that implies (using its inference system) the/a most-preferred set of facts.

But without further constraint, this process often leads to a contradiction. Suppose the agent's repertoire of actions is A1, ...An, and the value of action Ai is simply i. Say the agent starts by considering the action A7, and dutifully evaluates it as 7. Next, it contemplates the action A6, and reasons as follows: "Suppose I choose A6. I know I'm a utility-maximizing agent, and I already know there's another choice that has value 7. Therefore, if follows from my (hypothetical) choice of A6 that A6 has a value of at least 7." But that inference, while sound, contradicts the fact that A6's value is 6.

This is why timeless decision theory is a causality-based decision theory. I don't recall if you've indicated that you've studied Pearl's synthesis of Bayesian networks and causal graphs(?) (though if not you should be able to come up to speed on them pretty quickly).

So in the (standard) formalism of causality - just causality, never mind decision theory as yet - causal graphs give us a way to formally compute counterfactuals: We set the value of a particular node surgically. This means we delete the structural equations that would ordinarily give us the value at the node N_i as a function of the parent values P_i and the background uncertainty U_i at that node (which U_i must be uncorrelated to all other U, or the causal graph has not been fully factored). We delete this structural equation for N_i and make N_i parentless, so we don't send any likelihood messages up to the former parents when we update our knowledge of the value at N_i. However, we do send prior-messages from N_i to all of its descendants, maintaining the structural equations for the children of which N_i is a parent, and their children, and so on.

That's the standard way of computing counterfactuals in the Pearl/Spirtes/Verma synthesis of causality, as found in "Causality: Models, Reasoning, and Inference" and "Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference".

Classical causal decision theory says that your expected utility formula is over the counterfactual expectation of your physical act. Now, although the CDTs I've read have not in fact talked about Pearl - perhaps because it's a relatively recent mathematical technology, or perhaps because I last looked into the literature a few years back - and have just taken the counterfactual distribution as intuitively obvious mana rained from heaven - nonetheless it's pretty clear that their intuitions are operating pretty much the Pearlian way, via counterfactual surgery on the physical act.

So in calculating the "expected utility" of an act - the computation that classical CDT uses to choose an action - CDT assumes the act to be severed from its physical causal parents. Let's say that there's a Smoking Lesion problem, where the same gene causes a taste for cigarettes and an increased probability of cancer. Seeing someone else smoke, we would infer that they have an increased probability of cancer - this sends a likelihood-message upward to the node which represents the probability of having the gene, and this node in turns sends a prior-message downward to the node which represents the probability of getting cancer. But the counterfactual surgery that CDT performs on its physical acts, means that it calculates the expected utility as though the physical act is severed from its parent nodes. So CDT calculates the expected utility as though it has the base-rate probability of having the cancer gene regardless of its act, and so chooses to smoke, since it likes cigarettes. This is the common-sense and reflectively consistent action, so CDT appears to "win" here in terms of giving the winning answer - but it's worth noting that the internal calculation performed is wrong; if you act to smoke cigarettes, your probability of getting cancer is not the base rate.

And on Newcomb's Problem this internal error comes out into the open; the inside of CDT's counterfactual expected utility calculation, expects box B to contain a million dollars at the base rate, since it surgically severs the act of taking both boxes from the parent variable of your source code, which correlates to your previous source code at the moment Omega observed it, which correlates to Omega's decision whether to leave box B empty.

Now turn to timeless decision theory, in which the (Godelian diagonal) expected utility formula is written as follows:

Argmax[A in Actions] in Sum[O in Outcomes](Utility(O)*P(this computation yields A []-> O|rest of universe))

The interior of this formula performs counterfactual surgery to sever the logical output of the expected utility formula, from the initial conditions of the expected utility formula. So we do not conclude, in the inside of the formula as it performs the counterfactual surgery, that if-counterfactually A_6 is chosen over A_7 then A_6 must have higher expected utility. If-evidentially A_6 is chosen over A_7, then A_6 has higher expected utility - but this is not what the interior of the formula computes. As we compute the formula, the logical output is divorced from all parents; we cannot infer anything about its immediately logical precedents. This counterfactual surgery may be necessary, in fact, to stop an infinite regress in the formula, as it tries to model its own output in order to decide its own output; and this, arguably, is exactly why the decision counterfactual has the form it does - it is why we have to talk about counterfactual surgery within decisions in the first place.

Descendants of the logical output, however, continue to update their values within the counterfactual, which is why TDT one-boxes on Newcomb's Problem - both your current self's physical act, and Omega's physical act in the past, are logical-causal descendants of the computation, and are recalculated accordingly inside the counterfactual.

If you desire to smoke cigarettes, this would be observed and screened off by conditioning on the fixed initial conditions of the computation - the fact that the utility function had a positive term for smoking cigarettes, would already tell you that you had the gene. (Eells's "tickle".) If you can't observe your own utility function then you are actually taking a step outside the timeless decision theory as formulated.

So from the perspective of Metacircular Decision Theory - what is done with various facts - timeless decision theory can state very definitely how it treats the various facts, within the interior of its expected utility calculation. It does not update any physical or logical parent of the logical output - rather, it conditions on the initial state of the computation, in order to screen off outside influences; then no further inferences about them are made. And if you already know anything about the consequences of your logical output - its descendants in the logical causal graph - you will recompute what they would have been if you'd had a different output.

This last codicil is important for cases like Parfit's Hitchhiker, in which Omega (or perhaps Paul Ekman), driving a car through the desert, comes across yourself dying of thirst, and will give you a ride to the city only if they expect you to pay them $100 after you arrive in the city. (With the whole scenario being trued by strict selfishness, no knock-on effects, and so on.) There is, of course, no way of forcing the agreement - so will you compute, in the city, that it is better for you to give $100 to Omega, after having already been saved? Both evidential decision theory and causal decision theory will give the losing (dying in the desert, hence reflectively inconsistent) answer here; but TDT answers, "If I had decided not to pay, then Omega would have left me in the desert." So the expected utility of not paying $100 remains lower, even after you arrive in the city, given the way TDT computes its counterfactuals inside the formula - which is the dynamically and reflectively consistent and winning answer.. And note that this answer is arrived at in one natural step, without needing explicit reflection, let alone precommitment - you will answer this way even if the car-driver Omega made its prediction without you being aware of it, so long as Omega can credibly establish that it was predicting you with reasonable accuracy rather than making a pure uncorrelated guess. (And since it's not a very complicated calculation, Omega knowing that you are a timeless decision theorist is credible enough.)

I wonder if it might be open to the criticism that you're effectively postulating the favored answer to Newcomb's Problem (and other such scenarios) by postulating that when you surgically alter one of the nodes, you correspondingly alter the nodes for the other instances of the computation.

This is where one would refer to the omitted extended argument about a calculator on Mars and a calculator on Venus, where both calculators were manufactured at the same factory on Earth and observed before being transported to Mars and Venus. If we manufactured two envelopes on Earth, containing the same letter, and transported them to Mars and Venus without observing them, then indeed the contents of the two envelopes would be correlated in our probability distribution, even though the Mars-envelope is not a cause of the Venus-envelope, nor the Venus-envelope a cause of the Mars-envelope, because they have a common cause in the background. But if we observe the common cause - look at the message as it is written, before being Xeroxed and placed into the two envelopes - then the standard theory of causality requires that our remaining uncertainty about the two envelopes be uncorrelated; we have observed the common cause and screened it off. If N_i is not a cause of N_j or vice versa, and you know the state of all the common ancestors A_ij of N_i and N_j, and you do not know the state of any mutual descendants D_ij of N_i and N_j, then the standard rules of causal graphs (D-separation) show that your probabilities at N_i and N_j must be independent.

However, if you manufacture on Earth two calculators both set to calculate 123 * 456, and you have not yet performed this calculation in your head, then you can observe completely the physical state of the two calculators before they leave Earth, and yet still have correlated uncertainty about what result will flash on the screen on Mars and the screen on Venus. So this situation is simply not compatible with the mathematical axioms on causal graphs if you draw a causal graph in which the only common ancestor of the two calculators is the physical factory that made them and produced their correlated initial state. If you are to preserve the rules of causal graphs at all, you must have an additional node - which would logically seem to represent one's logical uncertainty about the abstract computation 123 * 456 - which is the parent of both calculators. Seeing the Venusian calculator flash the result 56,088, this physical event sends a likelihood-message to its parent node representing the logical result of 123 * 456, which sends a prior-message to its child node, the physical message flashed on the screen at Mars.

A similar argument shows that if we have completely observed our own initial source code, and perhaps observed Omega's initial source code which contains a copy of our source code and the intention to simulate it, but we do not yet know our own decision, then the only way in which our uncertainty about our own physical act can possibly be correlated at all with Omega's past act to fill or leave empty the box B - given that neither act physically causes the other - is if there is some common ancestor node unobserved; and having already seen that our causal graph must include logical uncertainty if it is to stay factored, we can (must?) interpret this unobserved common node as the logical output of the known expected utility calculation.

From this, I would argue, TDT follows. But of course it's going to be difficult to exhibit an algorithm that computes this - guessing unknown causal networks is an extremely difficult problem in machine learning, and only small such networks can be learned. In general, determining the causal structure of reality is AI-complete. And by interjecting logical uncertainty into the problem, we really are heading far beyond the causal networks that known machine algorithms can learn. But it is the case that if you rely on humans to learn the causal algorithm, then it is pretty clear that the Newcomb's Problem setup, if it is to be analyzed in causal terms at all, must have nodes corresponding to logical uncertainty, on pain of violating the axioms governing causal graphs. Furthermore, in being told that Omega's leaving box B full or empty correlates to our decision to take only one box or both boxes, and that Omega's act lies in the past, and that Omega's act is not directly influencing us, and that we have not found any other property which would screen off this uncertainty even when we inspect our own source code / psychology in advance of knowing our actual decision, and that our computation is the only direct ancestor of our logical output, then we're being told in unambiguous terms (I think) to make our own physical act and Omega's act a common descendant of the unknown logical output of our known computation. (A counterexample in the form of another causal graph compatible with the same data is welcome.) And of course we could make the problem very clear by letting the agent be a computer program and letting Omega have a copy of the source code with superior computing power, in which case the logical interpretation is very clear.

So these are the facts which TDT takes into account, and the facts which it ignores. The Nesov-Dai updateless decision theory is even stranger - as far as I can make out, it ignores all facts except for the fact about which inputs have been received by the logical version of the computation it implements. If combined with TDT, we would interpret UDT as having a never-updated weighting on all possible universes, and a causal structure (causal graph, presumably) on those universes. Any given logical computation in UDT will count all instantiations of itself in all universes which have received exactly the same inputs - even if those instantiations are being imagined by Omega in universes which UDT would ordinarily be interpreted as "known to be logically inconsistent", like universes in which the third decimal digit of pi is 3. Then UDT calculates the counterfactual consequences, weighted across all imagined universes, using its causal graphs on each of those universes, of setting the logical act to A_i. Then it maximizes on A_i.

I would ask if, applying Metacircular Decision Theory from a "common-sense human base level", you see any case in which additional facts should be taken into account, or other facts ignored, apart from those facts used by TDT (UDT). If not, and if TDT (UDT) are reflectively consistent, then TDT (UDT) is the fixed point of MCDT starting from a human baseline decision theory. Of course this can't actually be the case because TDT (UDT) are incomplete with respect to the open problems cited earlier, like logical ordering of moves, and choice of conditional strategies in response to conditional strategies. But it would be the way I'd pose the problem to you, Gary Drescher - MCDT is an interesting way of looking things, but I'm still trying to wrap my mind around it.

Part 2: Metacircular Decision Theory as reflection criterion.

MCDT's proposed criterion is this: the agent makes a meta-choice about which facts to omit when making inferences about the hypothetical actions, and selects the set of facts which lead to the best outcome if the agent then evaluates the original candidate actions with respect to that choice of facts. The agent then iterates that meta-evaluation as needed (probably not very far) until a fixed point is reached, i.e. the same choice (as to which facts to omit) leaves the first-order choice unchanged. (It's ok if that's intractable or uncomputable; the agent can muddle through with some approximate algorithm.)

...In other words, metacircular consistency isn't just a test that we'd like the decision theory to pass. Metacircular consistency is the theory; it is the algorithm.

But it looks to me like MCDT has to start from some particular base theory, and different base theories may have different fixed points (or conceivably, cycles). In which case we can't yet call MCDT itself a complete theory specification. When you talk about which facts would be wise to take into account, or ignore, (or recompute counterfactually even if they already have known values?), then you're imagining different source codes (or MCDT specifications?) that an agent could have; and calculating the benefits of adopting these different source codes, relative to the way the current base theory computes "adopting" and "benefit"

For example, if you start with CDT and apply MCDT at 7am, it looks to me like "use TDT (UDT) for all cases where my source code has a physical effect after 7am, and use CDT for all cases where the source code had a physical effect before 7am or a correlation stemming from common ancestry" is a reflectively stable fixed point of MCDT. Whenever CDT asks "What if I took into account these different facts?", it will say, "But Omega would not be physically affected by my self-modification, so clearly it can't benefit me in any way." If the MCDT criterion is to be applied in a different and intuitively appealing way that has only one fixed point (up to different utility functions) then this would establish MCDT as a good candidate for the decision theory, but right now it does look to me like a reflective consistency test. But maybe this is because I haven't yet wrapped my mind around the MCDT's fact-treatment-based decomposition of decision theories, or because you've already specified further mandatory structure in the base theory how the effect of ignoring or taking into account some particular fact is to be computed.

Original with comments: Timeless Decision Theory and Meta-Circular Decision Theory

Working Mantras

Eliezer Yudkowsky, 24 August 2009 10:08PM

While working with Marcello on AI this summer, I've noticed that I have some standard mantras that I invoke in my inner dialogue (though only on appropriate occasions, not as a literal repeated mantra). This says something about which tropes are most often useful - in my own working life, anyway!

1. "If anyone could actually update on the evidence, they would have a power far beyond that of Nobel Prize winners."
(When encountering a need to discard some idea to which I was attached, or admit loss of a sunk cost on an avenue that doesn't seem to be working out.)

2. "The universe is already like [X], or not; if it is then I can only minimize the embarrassment by admitting it and adapting as fast as possible."
(If the first mantra doesn't work; then I actually visualize the universe already being a certain way, so that I can see the penalty for being a universe that works a certain way and yet believing otherwise.)

3. "First understand the problem, then solve it."
(If getting too caught up in proposing solutions, or discouraged when solutions don't work out - the immediate task at hand is just to understand the problem, and one may ask whether progress has been made on this. From full understanding a solution usually follows quickly.)

4. "Load the problem."
(Try to get your mind involved and processing the various aspects of it.)

5. "Five minutes is enough time to have an insight."
(If my mind seems to be going empty.)

6. "Ask only one thing of your mind and it may give it to you."
(Focusing during work, or trying to load the problem into memory before going to sleep each night, in hopes of putting the subconscious to work on it.)

7. "Run right up the mountain!"
(My general visualization of the FAI problem; a huge, blank, impossibly high wall, which I have to run up as quickly as possible. Used to accomodate the sense of the problem being much larger than whatever it is I'm working on right now.)

8. "When the problem is solved, that thought will be a wasted motion in retrospect."
(I first enunciated this as an explicit general principle when explaining to Marcello why e.g. one doesn't worry about people who have failed to solve a problem previously. When you actually solve the problem, those thoughts will predictably not have contributed anything in retrospect. So if your goal is to solve the problem, you should focus on the object-level problem, instead of worrying about whether you have sufficient status to solve it. The same rule applies to many other habitual worries, or reasoning effort expended to reassure against them, that would predictably appear as wasted motion in retrospect, after actually solving the problem.)

9. "There's always just enough time when you do something right, no more, no less."
(A quote from C. J. Cherryh's Paladin, used when feeling rushed. I don't think it's true literally or otherwise, but it seems to convey an important wordless sentiment.

10. "See the truth, not what you expect or hope."
(When expecting the answer to go a particular way, or hoping for the answer to go a particular way, is exerting detectable pressure on an ongoing inquiry.)

I don't listen to music while working, because of studies showing that, e.g., programmers listening to music are equally competent at implementing a given algorithm, but much less likely to notice that the algorithm's output is always equal to its input. However, I sometimes think of the theme Emiya #0 when feeling fatigued or trying to make a special demand on my mind.

Original with comments: Working Mantras

The Sword of Good

Eliezer Yudkowsky, 03 September 2009 12:53AM

..fragments of a book that would never be written...

*    * *

Captain Selena, late of the pirate ship Nemesis, quietly extended the very tip of her blade around the corner, staring at the tiny reflection on the metal. At once, but still silently, she pulled back the sword; and with her other hand made a complex gesture.

The translation spell told Hirou that the handsigns meant: "Orcs. Seven."

Dolf looked at Hirou. "My Prince," the wizard signed, "do not waste yourself against mundane opponents. Do not draw the Sword of Good as yet. Leave these to Selena."

Hirou's mouth was very dry. He didn't know if the translation spell could understand the difference between wanting to talk and wanting to make gestures; and so Hirou simply nodded.

Not for the first time, the thought occurred to Hirou that if he'd actually known he was going to be transported into a magical universe, informed he was the long-lost heir to the Throne of Bronze, handed the legendary Sword of Good, and told to fight evil, he would have spent less time reading fantasy novels. Joined the army, maybe. Taken fencing lessons, at least. If there was one thing that didn't prepare you for fantasy real life, it was sitting at home reading fantasy fiction.

Dolf and Selena were looking at Hirou, as if waiting for something more.

Oh. That's right. I'm the prince.

Hirou raised a finger and pointed it around the corner, trying to indicate that they should go ahead -

With a sudden burst of motion Selena plunged around the corner, Dolf following hard on her heels, and Hirou, startled and hardly thinking, moving after.

(This story ended up too long for a single LW post, so I put it on yudkowsky.net.
Do read the rest of the story there, before continuing to the Acknowledgments below.)

Acknowledgments:

I had the idea for this story during a conversation with Nick Bostrom and Robin Hanson about an awful little facet of human nature I call "suspension of moral disbelief". The archetypal case in my mind will always be the Passover Seder, watching my parents and family and sometimes friends reciting the Ten Plagues that God is supposed to have visited on Egypt. You take drops from the wine glass - or grape juice in my case - and drip them onto the plate, to symbolize your sadness at God slaughtering the first-born male children of the Egyptians. So the Seder actually points out the awfulness, and yet no one says: "This is wrong; God should not have done that to innocent families in retaliation for the actions of an unelected Pharaoh." I forget when I first realized how horrible that was - the real horror being not the Plagues, of course, since they never happened; the real horror is watching your family not notice that they're swearing allegiance to an evil God in a happy wholesome family Cthulhu-worshiping ceremony. Arbitrarily hideous evils can be wholly concealed by a social atmosphere in which no one is expected to point them out and it would seem awkward and out-of-place to do so.

In writing it's even simpler - the author gets to create the whole social universe, and the readers are immersed in the hero's own internal perspective. And so anything the heroes do, which no character notices as wrong, won't be noticed by the readers as unheroic. Genocide, mind-rape, eternal torture, anything.

Explicit inspiration was taken from this XKCD (warning: spoilers for The Princess Bride), this Boat Crime, and this Monty Python, not to mention that essay by David Brin and the entire Goblins webcomic. This Looking For Group helped inspire the story's title, and everything else flowed downhill from there.

Original with comments: The Sword of Good

Bay Area OBLW Meetup Sep 12

Eliezer Yudkowsky, 06 September 2009 12:00AM

Bay Area OB/LW meetup, 6pm September 12th 2009 at the SIAI House in Santa Clara.

This will also serve as my 2nd annual 29th birthday party.

Original with comments: Bay Area OBLW Meetup Sep 12

Forcing Anthropics: Boltzmann Brains

Eliezer Yudkowsky, 07 September 2009 07:02PM

Followup to: Anthropic Reasoning in UDT by Wei Dai

Suppose that I flip a logical coin - e.g. look at some binary digit of pi unknown to either of us - and depending on the result, either create a billion of you in green rooms and one of you in a red room if the coin came up 1; or, if the coin came up 0, create one of you in a green room and a billion of you in red rooms. You go to sleep at the start of the experiment, and wake up in a red room.

Do you reason that the coin very probably came up 0? Thinking, perhaps: "If the coin came up 1, there'd be a billion of me in green rooms and only one of me in a red room, and in that case, it'd be very surprising that I found myself in a red room."

What is your degree of subjective credence - your posterior probability - that the logical coin came up 1?

There are only two answers I can see that might in principle be coherent, and they are "50%" and "a billion to one against".

Tomorrow I'll talk about what sort of trouble you run into if you reply "a billion to one".

But for today, suppose you reply "50%". Thinking, perhaps: "I don't understand this whole consciousness rigamarole, I wouldn't try to program a computer to update on it, and I'm not going to update on it myself."

In that case, why don't you believe you're a Boltzmann brain?

Back when the laws of thermodynamics were being worked out, there was first asked the question: "Why did the universe seem to start from a condition of low entropy?" Boltzmann suggested that the larger universe was in a state of high entropy, but that, given a long enough time, regions of low entropy would spontaneously occur - wait long enough, and the egg will unscramble itself - and that our own universe was such a region.

The problem with this explanation is now known as the "Boltzmann brain" problem; namely, while Hubble-region-sized low-entropy fluctuations will occasionally occur, it would be far more likely - though still not likely in any absolute sense - for a handful of particles to come together in a configuration performing a computation that lasted just long enough to think a single conscious thought (whatever that means) before dissolving back into chaos. A random reverse-entropy fluctuation is exponentially vastly more likely to take place in a small region than a large one.

So on Boltzmann's attempt to explain the low-entropy initial condition of the universe as a random statistical fluctuation, it's far more likely that we are a little blob of chaos temporarily hallucinating the rest of the universe, than that a multi-billion-light-year region spontaneously ordered itself. And most such little blobs of chaos will dissolve in the next moment.

"Well," you say, "that may be an unpleasant prediction, but that's no license to reject it." But wait, it gets worse: The vast majority of Boltzmann brains have experiences much less ordered than what you're seeing right now. Even if a blob of chaos coughs up a visual cortex (or equivalent), that visual cortex is unlikely to see a highly ordered visual field - the vast majority of possible visual fields more closely resemble "static on a television screen" than "words on a computer screen". So on the Boltzmann hypothesis, highly ordered experiences like the ones we are having now, constitute an exponentially infinitesimal fraction of all experiences.

In contrast, suppose one more simple law of physics not presently understood, which forces the initial condition of the universe to be low-entropy. Then the exponentially vast majority of brains occur as the result of ordered processes in ordered regions, and it's not at all surprising that we find ourselves having ordered experiences.

But wait! This is just the same sort of logic (is it?) that one would use to say, "Well, if the logical coin came up heads, then it's very surprising to find myself in a red room, since the vast majority of people-like-me are in green rooms; but if the logical coin came up tails, then most of me are in red rooms, and it's not surprising that I'm in a red room."

If you reject that reasoning, saying, "There's only one me, and that person seeing a red room does exist, even if the logical coin came up heads" then you should have no trouble saying, "There's only one me, having a highly ordered experience, and that person exists even if all experiences are generated at random by a Boltzmann-brain process or something similar to it." And furthermore, the Boltzmann-brain process is a much simpler process - it could occur with only the barest sort of causal structure, no need to postulate the full complexity of our own hallucinated universe. So if you're not updating on the apparent conditional rarity of having a highly ordered experience of gravity, then you should just believe the very simple hypothesis of a high-volume random experience generator, which would necessarily create your current experiences - albeit with extreme relative infrequency, but you don't care about that.

Now, doesn't the Boltzmann-brain hypothesis also predict that reality will dissolve into chaos in the next moment? Well, it predicts that the vast majority of blobs who experience this moment, cease to exist after; and that among the few who don't dissolve, the vast majority of those experience chaotic successors. But there would be an infinitesimal fraction of a fraction of successors, who experience ordered successor-states as well. And you're not alarmed by the rarity of those successors, just as you're not alarmed by the rarity of waking up in a red room if the logical coin came up 1 - right?

So even though your friend is standing right next to you, saying, "I predict the sky will not turn into green pumpkins and explode - oh, look, I was successful again!", you are not disturbed by their unbroken string of successes. You just keep on saying, "Well, it was necessarily true that someone would have an ordered successor experience, on the Boltzmann-brain hypothesis, and that just happens to be us, but in the next instant I will sprout wings and fly away."

Now this is not quite a logical contradiction. But the total rejection of all science, induction, and inference in favor of an unrelinquishable faith that the next moment will dissolve into pure chaos, is sufficiently unpalatable that even I decline to bite that bullet.

And so I still can't seem to dispense with anthropic reasoning - I can't seem to dispense with trying to think about how many of me or how much of me there are, which in turn requires that I think about what sort of process constitutes a me. Even though I confess myself to be sorely confused, about what could possibly make a certain computation "real" or "not real", or how some universes and experiences could be quantitatively realer than others (possess more reality-fluid, as 'twere), and I still don't know what exactly makes a causal process count as something I might have been for purposes of being surprised to find myself as me, or for that matter, what exactly is a causal process.

Indeed this is all greatly and terribly confusing unto me, and I would be less confused if I could go through life while only answering questions like "Given the Peano axioms, what is SS0 + SS0?"

But then I have no defense against the one who says to me, "Why don't you think you're a Boltzmann brain? Why don't you think you're the result of an all-possible-experiences generator? Why don't you think that gravity is a matter of branching worlds in which all objects accelerate in all directions and in some worlds all the observed objects happen to be accelerating downward? It explains all your observations, in the sense of logically necessitating them."

I want to reply, "But then most people don't have experiences this ordered, so finding myself with an ordered experience is, on your hypothesis, very surprising. Even if there are some versions of me that exist in regions or universes where they arose by chaotic chance, I anticipate, for purposes of predicting my future experiences, that most of my existence is encoded in regions and universes where I am the product of ordered processes."

And I currently know of no way to reply thusly, that does not make use of poorly defined concepts like "number of real processes" or "amount of real processes"; and "people", and "me", and "anticipate" and "future experience".

Of course confusion exists in the mind, not in reality, and it would not be the least bit surprising if a resolution of this problem were to dispense with such notions as "real" and "people" and "my future". But I do not presently have that resolution.

(Tomorrow I will argue that anthropic updates must be illegal and that the correct answer to the original problem must be "50%".)

Referenced by: Outlawing Anthropics: An Updateless Dilemma • The Anthropic Trilemma

Original with comments: Forcing Anthropics: Boltzmann Brains

Why I'm Staying On Bloggingheads.tv

Eliezer Yudkowsky, 07 September 2009 08:15PM

Recently, Sean Carroll, Carl Zimmer, and Phil Plait have all decided to stop appearing on BloggingHeads.TV (BHTV), and PZ Myers announced he would not appear on it in the future, after a disastrous decision to have creationist Michael Behe interviewed by the linguist and non-biologist John McWhorter, who failed to call Behe on his standard BS.

I'm hereby publicly announcing that I intend to stay on BloggingHeads.TV.

Why? Two main reasons:

1) Robert Wright publicly said that this was foolish, apologized for the poor editorial oversight that led to it, and says they're going to try never to do this again. This looks sincere to me, and given that it's sincere, people really ought to be allowed more chance than this to recover from their mistakes.

2) Bloggingheads.TV has given me a forum to debate accomodationist atheists who are insufficiently condemning of religion - for example my diavlog with Adam Frank, author of "The Constant Fire". Adam Frank argues that, while of course we now know that God doesn't exist, nonetheless scientific wonder at the universe and its mysteries has a lot in common with the roots of religion. And I said this was wishful thinking, historically ignorant of how religions really arose and propagated themselves, and a continuation of such theistic bad habits as thinking that things of which we are temporarily ignorant are "sacred mysteries". And no one at BHTV complained that I was being too confrontational, or too anti-religious, or that it was unfair to have the diavlog be between two atheists.

If BHTV is willing to let me come on and (politely) kick hell out of atheists who aren't atheistic enough to suit me, then I don't believe that their unfortunate failure to have Behe interviewed by someone who could call his BS, represents any deep hidden agenda in favor of religion and against science.

Rather, I think it represents a commitment to having interesting discussions by people who intelligently disagree with each other and have something courteous to say about it - even if that discussion wanders into the fearsome death zones where science does ("does not!") clash with religion - and this commitment managed to go wrong on one or two occasions.

My friends and fellow antitheists, this is an important commitment while most of the world is continuing to pretend that there is no conflict between science and religion. It's not surprising if that commitment goes wrong now and then. It is not reasonable to expect that a commitment to repeatedly discuss a scary controversy will never go wrong. It may well go wrong again despite Robert Wright's best intentions. But unless it starts to go wrong systematically, I'm going to stay on BHTV, arguing that science and religion are not compatible.

Of course, if most other non-accomodationists jump ship from BHTV as a result of the Behe affair, then it will become a hangout for accomodationists only. "Evaporative Cooling of Group Beliefs" is another reason why you should put forth at least a little effort to "Tolerate Tolerance" - to not insist that all your potential trade-partners punish the same people you've labeled defectors, exactly the way you want them punished, before you cooperate. Yes, Behe is an enemy of science, but Wright is not; and Wright may also dislike Behe, yet not wish to implement exactly the same punishment-policy toward Behe that you advocate; and that needs to be all right, if we're all going to end up cooperating.

Original with comments: Why I'm Staying On Bloggingheads.tv

FHI postdoc at Oxford

Eliezer Yudkowsky, 08 September 2009 06:18PM

The Future of Humanity Institute at the University of Oxford is looking to fill a postdoctoral research fellowship in interdisciplinary science or philosophy. Current work areas include global catastrophic risks, probabilistic methodology and applied epistemology and rationality, impacts of future technologies, and ethical issues related to human enhancement. Apply by Oct 14.

Original with comments: FHI postdoc at Oxford

Outlawing Anthropics: An Updateless Dilemma

Eliezer Yudkowsky, 08 September 2009 06:31PM

Let us start with a (non-quantum) logical coinflip - say, look at the heretofore-unknown-to-us-personally 256th binary digit of pi, where the choice of binary digit is itself intended not to be random.

If the result of this logical coinflip is 1 (aka "heads"), we'll create 18 of you in green rooms and 2 of you in red rooms, and if the result is "tails" (0), we'll create 2 of you in green rooms and 18 of you in red rooms.

After going to sleep at the start of the experiment, you wake up in a green room.

With what degree of credence do you believe - what is your posterior probability - that the logical coin came up "heads"?

There are exactly two tenable answers that I can see, "50%" and "90%".

Suppose you reply 90%.

And suppose you also happen to be "altruistic" enough to care about what happens to all the copies of yourself. (If your current system cares about yourself and your future, but doesn't care about very similar xerox-siblings, then you will tend to self-modify to have future copies of yourself care about each other, as this maximizes your expectation of pleasant experience over future selves.)

Then I attempt to force a reflective inconsistency in your decision system, as follows:

I inform you that, after I look at the unknown binary digit of pi, I will ask all the copies of you in green rooms whether to pay $1 to every version of you in a green room and steal $3 from every version of you in a red room. If they all reply "Yes", I will do so.

(It will be understood, of course, that $1 represents 1 utilon, with actual monetary amounts rescaled as necessary to make this happen. Very little rescaling should be necessary.)

(Timeless decision agents reply as if controlling all similar decision processes, including all copies of themselves. Classical causal decision agents, to reply "Yes" as a group, will need to somehow work out that other copies of themselves reply "Yes", and then reply "Yes" themselves. We can try to help out the causal decision agents on their coordination problem by supplying rules such as "If conflicting answers are delivered, everyone loses $50". If causal decision agents can win on the problem "If everyone says 'Yes' you all get $10, if everyone says 'No' you all lose $5, if there are conflicting answers you all lose $50" then they can presumably handle this. If not, then ultimately, I decline to be responsible for the stupidity of causal decision agents.)

Suppose that you wake up in a green room. You reason, "With 90% probability, there are 18 of me in green rooms and 2 of me in red rooms; with 10% probability, there are 2 of me in green rooms and 18 of me in red rooms. Since I'm altruistic enough to at least care about my xerox-siblings, I calculate the expected utility of replying 'Yes' as (90% * ((18 * +$1) + (2 * -$3))) + (10% * ((18 * -$3) + (2 * +$1))) = +$5.60." You reply yes.

However, before the experiment, you calculate the general utility of the conditional strategy "Reply 'Yes' to the question if you wake up in a green room" as (50% * ((18 * +$1) + (2 * -$3))) + (50% * ((18 * -$3) + (2 * +$1))) = -$20. You want your future selves to reply 'No' under these conditions.

This is a dynamic inconsistency - different answers at different times - which argues that decision systems which update on anthropic evidence will self-modify not to update probabilities on anthropic evidence.

I originally thought, on first formulating this problem, that it had to do with double-counting the utilons gained by your variable numbers of green friends, and the probability of being one of your green friends.

However, the problem also works if we care about paperclips. No selfishness, no altruism, just paperclips.

Let the dilemma be, "I will ask all people who wake up in green rooms if they are willing to take the bet 'Create 1 paperclip if the logical coinflip came up heads, destroy 3 paperclips if the logical coinflip came up tails'. (Should they disagree on their answers, I will destroy 5 paperclips.)" Then a paperclip maximizer, before the experiment, wants the paperclip maximizers who wake up in green rooms to refuse the bet. But a conscious paperclip maximizer who updates on anthropic evidence, who wakes up in a green room, will want to take the bet, with expected utility ((90% * +1 paperclip) + (10% * -3 paperclips)) = +0.6 paperclips.

This argues that, in general, decision systems - whether they start out selfish, or start out caring about paperclips - will not want their future versions to update on anthropic "evidence".

Well, that's not too disturbing, is it? I mean, the whole anthropic thing seemed very confused to begin with - full of notions about "consciousness" and "reality" and "identity" and "reference classes" and other poorly defined terms. Just throw out anthropic reasoning, and you won't have to bother.

When I explained this problem to Marcello, he said, "Well, we don't want to build conscious AIs, so of course we don't want them to use anthropic reasoning", which is a fascinating sort of reply. And I responded, "But when you have a problem this confusing, and you find yourself wanting to build an AI that just doesn't use anthropic reasoning to begin with, maybe that implies that the correct resolution involves us not using anthropic reasoning either."

So we can just throw out anthropic reasoning, and relax, and conclude that we are Boltzmann brains. QED.

In general, I find the sort of argument given here - that a certain type of decision system is not reflectively consistent - to be pretty damned compelling. But I also find the Boltzmann conclusion to be, ahem, more than ordinarily unpalatable.

In personal conversation, Nick Bostrom suggested that a division-of-responsibility principle might cancel out the anthropic update - i.e., the paperclip maximizer would have to reason, "If the logical coin came up heads then I am 1/18th responsible for adding +1 paperclip, if the logical coin came up tails then I am 1/2 responsible for destroying 3 paperclips." I confess that my initial reaction to this suggestion was "Ewwww", but I'm not exactly comfortable concluding I'm a Boltzmann brain, either.

EDIT: On further reflection, I also wouldn't want to build an AI that concluded it was a Boltzmann brain! Is there a form of inference which rejects this conclusion without relying on any reasoning about subjectivity?

EDIT2: Psy-Kosh has converted this into a non-anthropic problem!

Original with comments: Outlawing Anthropics: An Updateless Dilemma

Let Them Debate College Students

Eliezer Yudkowsky, 09 September 2009 06:15PM

(EDIT: Woozle has an even better idea, which would apply to many debates in general if the true goal were seeking resolution and truth.)

Friends, Romans, non-Romans, lend me your ears. I have for you a modest proposal, in this question of whether we should publicly debate creationists, or freeze them out as unworthy of debate.

My fellow humans, I have two misgivings about this notion that there should not be a debate. My first misgiving is that - even though on this particular occasion scientific society is absolutely positively not wrong to dismiss creationism - this business of not having debates sounds like dangerous business to me. Science is sometimes wrong, you know, even if it is not wrong this time, and debating is part of the recovery process.

And my second misgiving is that, like it or not, the creationists are on the radio, in the town halls, and of course on the Web, and they are already talking to large audiences; and the idea that there is not going to be a debate about this, may be slightly naive.

"But," you cry, "when prestigious scientists lower themselves so far as to debate creationists, afterward the creationists smugly advertise that prestigious scientists are debating them!"

Ah, but who says that prestigious scientists are required to debate creationists?

Find some bright ambitious young college student working toward a biology degree, someone who's read Pharyngula and the talk.origins FAQ. Maybe have P. Z. Myers or someone run a test debate on them, to make sure they know how to answer all the standard lies and are generally good at debating and explaining. Then have the college student debate the creationists - if the creationists are still up for it. If not, of course, we can all make a big ruckus about how Michael Behe is afraid to debate a mere college student, and have the college student reply to all requests to debate Richard Dawkins or supply a scientific authority for the TV networks. And if Michael Behe manages to defeat the college student, then he can go on to debate a PhD, and if that doesn't work, Behe gets to talk to P. Z. Myers, and in the unlikely event Behe manages not to get his butt handed to him by P. Z. Myers, he would have earned the right to debate Richard Dawkins.

If we're dealing with young-earth creationists, then we add a bright 12-year-old at the start of the chain.

That way, anyone who wants to know the state of the debate and the status of the arguments, is welcome to watch creationists being beaten up by some college kid - armed with real science, mind!

But there will still be a debate. And if the scientific community, at some point in the future, manages to go astray on some issue where the opposing side seems "silly", then we can hope - if public debate is any use at all - that the challenger will gently defeat the 12-year-old, unravel the college student, score points against the PhD, and hold their own against senior scientists. There would still be a path to victory for worthy new ideas, and not a general license for a community to shut down all debate it thinks unworthy.

It's this notion of shutting down debate that I fear as dangerous; and it seems to me that you can get just the same strategic conservation of prestige, by endorsing the principle of debate, but sending out some bright college students to present the standard position. If the "controversy" as shown on CNN consists of some ID-er with a sober-looking business suit and an impressive-sounding title, versus a TA in jeans to represent the scientific community - but with accurate science, mind! - then I think this would viscerally answer what the scientific community thinks of creationism, and not create the false impression of an ongoing debate, while still giving airtime to the standard scientific replies. If CNN isn't interested in showing that "controversy" - well then, that tells us what CNN really wanted, doesn't it.

If an idea is so completely ridiculous as to be unworthy even of debate - then send out some bright un-titled college students to debate it! Do vet them for knowledge of standard replies, explanation ability, and debating ability against evil opponents, to make sure standard science is not needlessly embarrassed. But there should be plenty of ambitious young bright college students who can pass that filter and who would enjoy some TV exposure.

Original with comments: Let Them Debate College Students

The Lifespan Dilemma

Eliezer Yudkowsky, 10 September 2009 06:45PM

One of our most controversial posts ever was "Torture vs. Dust Specks". Though I can't seem to find the reference, one of the more interesting uses of this dilemma was by a professor whose student said "I'm a utilitarian consequentialist", and the professor said "No you're not" and told them about SPECKS vs. TORTURE, and then the student - to the professor's surprise - chose TORTURE. (Yay student!)

In the spirit of always making these things worse, let me offer a dilemma that might have been more likely to unconvince the student - at least, as a consequentialist, I find the inevitable conclusion much harder to swallow.

I'll start by briefly introducing Parfit's Repugnant Conclusion, sort of a little brother to the main dilemma. Parfit starts with a world full of a million happy people - people with plenty of resources apiece. Next, Parfit says, let's introduce one more person who leads a life barely worth living - but since their life is worth living, adding this person must be a good thing. Now we redistribute the world's resources, making it fairer, which is also a good thing. Then we introduce another person, and another, until finally we've gone to a billion people whose lives are barely at subsistence level. And since (Parfit says) it's obviously better to have a million happy people than a billion people at subsistence level, we've gone in a circle and revealed inconsistent preferences.

My own analysis of the Repugnant Conclusion is that its apparent force comes from equivocating between senses of barely worth living. In order to voluntarily create a new person, what we need is a life that is worth celebrating or worth birthing, one that contains more good than ill and more happiness than sorrow - otherwise we should reject the step where we choose to birth that person. Once someone is alive, on the other hand, we're obliged to take care of them in a way that we wouldn't be obliged to create them in the first place - and they may choose not to commit suicide, even if their life contains more sorrow than happiness. If we would be saddened to hear the news that such a person existed, we shouldn't kill them, but we should not voluntarily create such a person in an otherwise happy world. So each time we voluntarily add another person to Parfit's world, we have a little celebration and say with honest joy "Whoopee!", not, "Damn, now it's too late to uncreate them."

And then the rest of the Repugnant Conclusion - that it's better to have a billion lives slightly worth celebrating, than a million lives very worth celebrating - is just "repugnant" because of standard scope insensitivity. The brain fails to multiply a billion small birth celebrations to end up with a larger total celebration of life than a million big celebrations. Alternatively, average utilitarians - I suspect I am one - may just reject the very first step, in which the average quality of life goes down.

But now we introduce the Repugnant Conclusion's big sister, the Lifespan Dilemma, which - at least in my own opinion - seems much worse.

To start with, suppose you have a 20% chance of dying in an hour, and an 80% chance of living for 1010,000,000,000 years -

Now I know what you're thinking, of course. You're thinking, "Well, 10^(10^10) years may sound like a long time, unimaginably vaster than the 10^15 years the universe has lasted so far, but it isn't much, really. I mean, most finite numbers are very much larger than that. The realms of math are infinite, the realms of novelty and knowledge are infinite, and Fun Theory argues that we'll never run out of fun. If I live for 1010,000,000,000 years and then die, then when I draw my last metaphorical breath - not that I'd still have anything like a human body after that amount of time, of course - I'll go out raging against the night, for a life so short compared to all the experiences I wish I could have had. You can't compare that to real immortality. As Greg Egan put it, immortality isn't living for a very long time and then dying. Immortality is just not dying, ever."

Well, I can't offer you real immortality - not in this dilemma, anyway. However, on behalf of my patron, Omega, who I believe is sometimes also known as Nyarlathotep, I'd like to make you a little offer.

If you pay me just one penny, I'll replace your 80% chance of living for 10^(10^10) years, with a 79.99992% chance of living 10^(10^(10^10)) years. That's 99.9999% of 80%, so I'm just shaving a tiny fraction 10-6 off your probability of survival, and in exchange, if you do survive, you'll survive - not ten times as long, my friend, but ten to the power of as long. And it goes without saying that you won't run out of memory (RAM) or other physical resources during that time. If you feel that the notion of "years" is ambiguous, let's just measure your lifespan in computing operations instead of years. Really there's not much of a difference when you're dealing with numbers like 10^(1010,000,000,000).

My friend - can I call you friend? - let me take a few moments to dwell on what a wonderful bargain I'm offering you. Exponentiation is a rare thing in gambles. Usually, you put $1,000 at risk for a chance at making $1,500, or some multiplicative factor like that. But when you exponentiate, you pay linearly and buy whole factors of 10 - buy them in wholesale quantities, my friend! We're talking here about 1010,000,000,000 factors of 10! If you could use $1,000 to buy a 99.9999% chance of making $10,000 - gaining a single factor of ten - why, that would be the greatest investment bargain in history, too good to be true, but the deal that Omega is offering you is far beyond that! If you started with $1, it takes a mere eight factors of ten to increase your wealth to $100,000,000. Three more factors of ten and you'd be the wealthiest person on Earth. Five more factors of ten beyond that and you'd own the Earth outright. How old is the universe? Ten factors-of-ten years. Just ten! How many quarks in the whole visible universe? Around eighty factors of ten, as far as anyone knows. And we're offering you here - why, not even ten billion factors of ten. Ten billion factors of ten is just what you started with! No, this is ten to the ten billionth power factors of ten.

Now, you may say that your utility isn't linear in lifespan, just like it isn't linear in money. But even if your utility is logarithmic in lifespan - a pessimistic assumption, surely; doesn't money decrease in value faster than life? - why, just the logarithm goes from 10,000,000,000 to 1010,000,000,000.

From a fun-theoretic standpoint, exponentiating seems like something that really should let you have Significantly More Fun. If you can afford to simulate a mind a quadrillion bits large, then you merely need 2^(1,000,000,000,000,000) times as much computing power - a quadrillion factors of 2 - to simulate all possible minds with a quadrillion binary degrees of freedom so defined. Exponentiation lets you completely explore the whole space of which you were previously a single point - and that's just if you use it for brute force. So going from a lifespan of 10^(10^10) to 10^(10^(10^10)) seems like it ought to be a significant improvement, from a fun-theoretic standpoint.

And Omega is offering you this special deal, not for a dollar, not for a dime, but one penny! That's right! Act now! Pay a penny and go from a 20% probability of dying in an hour and an 80% probability of living 1010,000,000,000 years, to a 20.00008% probability of dying in an hour and a 79.99992% probability of living 10^(1010,000,000,000) years! That's far more factors of ten in your lifespan than the number of quarks in the visible universe raised to the millionth power!

Is that a penny, friend? - thank you, thank you. But wait! There's another special offer, and you won't even have to pay a penny for this one - this one is free! That's right, I'm offering to exponentiate your lifespan again, to 10^(10^(1010,000,000,000)) years! Now, I'll have to multiply your probability of survival by 99.9999% again, but really, what's that compared to the nigh-incomprehensible increase in your expected lifespan?

Is that an avaricious light I see in your eyes? Then go for it! Take the deal! It's free!

(Some time later.)

My friend, I really don't understand your grumbles. At every step of the way, you seemed eager to take the deal. It's hardly my fault that you've ended up with... let's see... a probability of 1/10^1000 of living 10^^(2,302,360,800) years, and otherwise dying in an hour. Oh, the ^^? That's just a compact way of expressing tetration, or repeated exponentiation - it's really supposed to be Knuth up-arrows, ↑↑, but I prefer to just write ^^. So 10^^(2,302,360,800) means 10^(10^(10^...^10)) where the exponential tower of tens is 2,302,360,800 layers high.

But, tell you what - these deals are intended to be permanent, you know, but if you pay me another penny, I'll trade you your current gamble for an 80% probability of living 1010,000,000,000 years.

Why, thanks! I'm glad you've given me your two cents on the subject.

Hey, don't make that face! You've learned something about your own preferences, and that's the most valuable sort of information there is!

Anyway, I've just received telepathic word from Omega that I'm to offer you another bargain - hey! Don't run away until you've at least heard me out!

Okay, I know you're feeling sore. How's this to make up for it? Right now you've got an 80% probability of living 1010,000,000,000 years. But right now - for free - I'll replace that with an 80% probability (that's right, 80%) of living 10^^10 years, that's 10^10^10^10^10^10^10^1010,000,000,000 years.

See? I thought that'd wipe the frown from your face.

So right now you've got an 80% probability of living 10^^10 years. But if you give me a penny, I'll tetrate that sucker! That's right - your lifespan will go to 10^^(10^^10) years! That's an exponential tower (10^^10) tens high! You could write that as 10^^^3, by the way, if you're interested. Oh, and I'm afraid I'll have to multiply your survival probability by 99.99999999%.

What? What do you mean, no? The benefit here is vastly larger than the mere 10^^(2,302,360,800) years you bought previously, and you merely have to send your probability to 79.999999992% instead of 10-1000 to purchase it! Well, that and the penny, of course. If you turn down this offer, what does it say about that whole road you went down before? Think of how silly you'd look in retrospect! Come now, pettiness aside, this is the real world, wouldn't you rather have a 79.999999992% probability of living 10^^(10^^10) years than an 80% probability of living 10^^10 years? Those arrows suppress a lot of detail, as the saying goes! If you can't have Significantly More Fun with tetration, how can you possibly hope to have fun at all?

Hm? Why yes, that's right, I am going to offer to tetrate the lifespan and fraction the probability yet again... I was thinking of taking you down to a survival probability of 1/(10^^^20), or something like that... oh, don't make that face at me, if you want to refuse the whole garden path you've got to refuse some particular step along the way.

Wait! Come back! I have even faster-growing functions to show you! And I'll take even smaller slices off the probability each time! Come back!

...ahem.

While I feel that the Repugnant Conclusion has an obvious answer, and that SPECKS vs. TORTURE has an obvious answer, the Lifespan Dilemma actually confuses me - the more I demand answers of my mind, the stranger my intuitive responses get. How are yours?

Based on an argument by Wei Dai. Dai proposed a reductio of unbounded utility functions by (correctly) pointing out that an unbounded utility on lifespan implies willingness to trade an 80% probability of living some large number of years for a 1/(3^^^3) probability of living some sufficiently longer lifespan. I looked at this and realized that there existed an obvious garden path, which meant that denying the conclusion would create a preference reversal. Note also the relation to the St. Petersburg Paradox, although the Lifespan Dilemma requires only a finite number of steps to get us in trouble.

Original with comments: The Lifespan Dilemma

The Finale of the Ultimate Meta Mega Crossover

Eliezer Yudkowsky, 20 September 2009 05:21PM

So I'd intended this story as a bit of utterly deranged fun, but it got out of control and ended up as a deep philosophical exploration, and now those of you who care will have to wade through the insanity. I'm sorry. I just can't seem to help myself.

I know that writing crossover fanfiction is considered one of the lower levels to which an author can sink. Alas, I've always been a sucker for audacity, and I am the sort of person who couldn't resist trying to top the entire... but never mind, you can see for yourself.

Click on to read my latest story and first fanfiction, a Vernor Vinge x Greg Egan crackfic.

Original with comments: The Finale of the Ultimate Meta Mega Crossover

The Anthropic Trilemma

Eliezer Yudkowsky, 27 September 2009 01:47AM

Speaking of problems I don't know how to solve, here's one that's been gnawing at me for years.

The operation of splitting a subjective worldline seems obvious enough - the skeptical initiate can consider the Ebborians, creatures whose brains come in flat sheets and who can symmetrically divide down their thickness. The more sophisticated need merely consider a sentient computer program: stop, copy, paste, start, and what was one person has now continued on in two places. If one of your future selves will see red, and one of your future selves will see green, then (it seems) you should anticipate seeing red or green when you wake up with 50% probability. That is, it's a known fact that different versions of you will see red, or alternatively green, and you should weight the two anticipated possibilities equally. (Consider what happens when you're flipping a quantum coin: half your measure will continue into either branch, and subjective probability will follow quantum measure for unknown reasons.)

But if I make two copies of the same computer program, is there twice as much experience, or only the same experience? Does someone who runs redundantly on three processors, get three times as much weight as someone who runs on one processor?

Let's suppose that three copies get three times as much experience. (If not, then, in a Big universe, large enough that at least one copy of anything exists somewhere, you run into the Boltzmann Brain problem.)

Just as computer programs or brains can split, they ought to be able to merge. If we imagine a version of the Ebborian species that computes digitally, so that the brains remain synchronized so long as they go on getting the same sensory inputs, then we ought to be able to put two brains back together along the thickness, after dividing them. In the case of computer programs, we should be able to perform an operation where we compare each two bits in the program, and if they are the same, copy them, and if they are different, delete the whole program. (This seems to establish an equal causal dependency of the final program on the two original programs that went into it. E.g., if you test the causal dependency via counterfactuals, then disturbing any bit of the two originals, results in the final program being completely different (namely deleted).)

So here's a simple algorithm for winning the lottery:

Buy a ticket. Suspend your computer program just before the lottery drawing - which should of course be a quantum lottery, so that every ticket wins somewhere. Program your computational environment to, if you win, make a trillion copies of yourself, and wake them up for ten seconds, long enough to experience winning the lottery. Then suspend the programs, merge them again, and start the result. If you don't win the lottery, then just wake up automatically.

The odds of winning the lottery are ordinarily a billion to one. But now the branch in which you win has your "measure", your "amount of experience", temporarily multiplied by a trillion. So with the brief expenditure of a little extra computing power, you can subjectively win the lottery - be reasonably sure that when next you open your eyes, you will see a computer screen flashing "You won!" As for what happens ten seconds after that, you have no way of knowing how many processors you run on, so you shouldn't feel a thing.

Now you could just bite this bullet. You could say, "Sounds to me like it should work fine." You could say, "There's no reason why you shouldn't be able to exert anthropic psychic powers." You could say, "I have no problem with the idea that no one else could see you exerting your anthropic psychic powers, and I have no problem with the idea that different people can send different portions of their subjective futures into different realities."

I find myself somewhat reluctant to bite that bullet, personally.

Nick Bostrom, when I proposed this problem to him, offered that you should anticipate winning the lottery after five seconds, but anticipate losing the lottery after fifteen seconds.

To bite this bullet, you have to throw away the idea that your joint subjective probabilities are the product of your conditional subjective probabilities. If you win the lottery, the subjective probability of having still won the lottery, ten seconds later, is ~1. And if you lose the lottery, the subjective probability of having lost the lottery, ten seconds later, is ~1. But we don't have p("experience win after 15s") = p("experience win after 15s"|"experience win after 5s")*p("experience win after 5s") + p("experience win after 15s"|"experience not-win after 5s")*p("experience not-win after 5s").

I'm reluctant to bite that bullet too.

And the third horn of the trilemma is to reject the idea of the personal future - that there's any meaningful sense in which I can anticipate waking up as myself tomorrow, rather than Britney Spears. Or, for that matter, that there's any meaningful sense in which I can anticipate being myself in five seconds, rather than Britney Spears. In five seconds there will be an Eliezer Yudkowsky, and there will be a Britney Spears, but it is meaningless to speak of the current Eliezer "continuing on" as Eliezer+5 rather than Britney+5; these are simply three different people we are talking about.

There are no threads connecting subjective experiences. There are simply different subjective experiences. Even if some subjective experiences are highly similar to, and causally computed from, other subjective experiences, they are not connected.

I still have trouble biting that bullet for some reason. Maybe I'm naive, I know, but there's a sense in which I just can't seem to let go of the question, "What will I see happen next?" I strive for altruism, but I'm not sure I can believe that subjective selfishness - caring about your own future experiences - is an incoherent utility function; that we are forced to be Buddhists who dare not cheat a neighbor, not because we are kind, but because we anticipate experiencing their consequences just as much as we anticipate experiencing our own. I don't think that, if I were really selfish, I could jump off a cliff knowing smugly that a different person would experience the consequence of hitting the ground.

Bound to my naive intuitions that can be explained away by obvious evolutionary instincts, you say? It's plausible that I could be forced down this path, but I don't feel forced down it quite yet. It would feel like a fake reduction. I have rather the sense that my confusion here is tied up with my confusion over what sort of physical configurations, or cascades of cause and effect, "exist" in any sense and "experience" anything in any sense, and flatly denying the existence of subjective continuity would not make me feel any less confused about that.

The fourth horn of the trilemma (as 'twere) would be denying that two copies of the same computation had any more "weight of experience" than one; but in addition to the Boltzmann Brain problem in large universes, you might develop similar anthropic psychic powers if you could split a trillion times, have each computation view a slightly different scene in some small detail, forget that detail, and converge the computations so they could be reunified afterward - then you were temporarily a trillion different people who all happened to develop into the same future self. So it's not clear that the fourth horn actually changes anything, which is why I call it a trilemma.

I should mention, in this connection, a truly remarkable observation: quantum measure seems to behave in a way that would avoid this trilemma completely, if you tried the analogue using quantum branching within a large coherent superposition (e.g. a quantum computer). If you quantum-split into a trillion copies, those trillion copies would have the same total quantum measure after being merged or converged.

It's a remarkable fact that the one sort of branching we do have extensive actual experience with - though we don't know why it behaves the way it does - seems to behave in a very strange way that is exactly right to avoid anthropic superpowers and goes on obeying the standard axioms for conditional probability.

In quantum copying and merging, every "branch" operation preserves the total measure of the original branch, and every "merge" operation (which you could theoretically do in large coherent superpositions) likewise preserves the total measure of the incoming branches.

Great for QM. But it's not clear to me at all how to set up an analogous set of rules for making copies of sentient beings, in which the total number of processors can go up or down and you can transfer processors from one set of minds to another.

To sum up:

- The first horn of the anthropic trilemma is to confess that there are simple algorithms whereby you can, indetectably to anyone but yourself, exert the subjective equivalent of psychic powers - use a temporary expenditure of computing power to permanently send your subjective future into particular branches of reality.

- The second horn of the anthropic trilemma is to deny that subjective joint probabilities behave like probabilities - you can coherently anticipate winning the lottery after five seconds, anticipate the experience of having lost the lottery after fifteen seconds, and anticipate that once you experience winning the lottery you will experience having still won it ten seconds later.

- The third horn of the anthropic trilemma is to deny that there is any meaningful sense whatsoever in which you can anticipate being yourself in five seconds, rather than Britney Spears; to deny that selfishness is coherently possible; to assert that you can hurl yourself off a cliff without fear, because whoever hits the ground will be another person not particularly connected to you by any such ridiculous thing as a "thread of subjective experience".

- The fourth horn of the anthropic trilemma is to deny that increasing the number of physical copies increases the weight of an experience, which leads into Boltzmann brain problems, and may not help much (because alternatively designed brains may be able to diverge and then converge as different experiences have their details forgotten).

- The fifth horn of the anthropic trilemma is to observe that the only form of splitting we have accumulated experience with, the mysterious Born probabilities of quantum mechanics, would seem to avoid the trilemma; but it's not clear how to have analogous rules could possibly govern information flows in computer processors.

I will be extremely impressed if Less Wrong solves this one.

Original with comments: The Anthropic Trilemma

Privileging the Hypothesis

Eliezer Yudkowsky, 29 September 2009 12:40AM

Suppose that the police of Largeville, a town with a million inhabitants, are investigating a murder in which there are few or no clues - the victim was stabbed to death in an alley, and there are no fingerprints and no witnesses.

Then, one of the detectives says, "Well... we have no idea who did it... no particular evidence singling out any of the million people in this city... but let's consider the hypothesis that this murder was committed by Mortimer Q. Snodgrass, who lives at 128 Ordinary Ln. It could have been him, after all."

I'll label this the fallacy of privileging the hypothesis. (Do let me know if it already has an official name - I can't recall seeing it described.)

Now the detective may perhaps have some form of rational evidence which is not legal evidence admissible in court - hearsay from an informant, for example. But if the detective does not have some justification already in hand for promoting Mortimer to the police's special attention - if the name is pulled entirely out of a hat - then Mortimer's rights are being violated.

And this is true even if the detective is not claiming that Mortimer "did" do it, but only asking the police to spend time pondering that Mortimer might have done it - unjustifiably promoting that particular hypothesis to attention. It's human nature to look for confirmation rather than disconfirmation. Suppose that three detectives each suggest their hated enemies, as names to be considered; and Mortimer is brown-haired, Frederick is black-haired, and Helen is blonde. Then a witness is found who says that the person leaving the scene was brown-haired. "Aha!" say the police. "We previously had no evidence to distinguish among the possibilities, but now we know that Mortimer did it!"

This is related to the principle I've started calling "locating the hypothesis", which is that if you have a billion boxes only one of which contains a diamond (the truth), and your detectors only provide 1 bit of evidence apiece, then it takes much more evidence to promote the truth to your particular attention - to narrow it down to ten good possibilities, each deserving of our individual attention - than it does to figure out which of those ten possibilities is true. 27 bits to narrow it down to 10, and just another 4 bits will give us better than even odds of having the right answer. (Again, let me know if there's a more standard name for this.)

Thus the detective, in calling Mortimer to the particular attention of the police, for no reason out of a million other people, is skipping over most of the evidence that needs to be supplied against Mortimer.

And the detective ought to have this evidence in their possession, at the first moment when they bring Mortimer to the police's attention at all. It may be mere rational evidence rather than legal evidence, but if there's no evidence then the detective is harassing and persecuting poor Mortimer.

During my recent diavlog with Scott Aaronson on quantum mechanics, I did manage to corner Scott to the extent of getting Scott to admit that there was no concrete evidence whatsoever that favors a collapse postulate or single-world quantum mechanics. But, said Scott, we might encounter future evidence in favor of single-world quantum mechanics, and many-worlds still has the open question of the Born probabilities.

This is indeed what I would call the fallacy of privileging the hypothesis. There must be a trillion better ways to answer the Born question without adding a collapse postulate that would be the only non-linear, non-unitary, discontinous, non-differentiable, non-CPT-symmetric, non-local in the configuration space, Liouville's-Theorem-violating, privileged-space-of-simultaneity-possessing, faster-than-light-influencing, acausal, informally specified law in all of physics. Something that unphysical is not worth saying out loud or even thinking about as a possibility without a rather large weight of evidence - far more than the current grand total of zero.

But because of a historical accident, collapse postulates and single-world quantum mechanics are indeed on everyone's lips and in everyone's mind to be thought of, and so the open question of the Born probabilities is offered up (by Scott Aaronson no less!) as evidence that many-worlds can't yet offer a complete picture of the world. Which is taken to mean that single-world QM is still in the running somehow.

In the minds of human beings, if you can get them to think about this particular hypothesis rather than the trillion other possibilities that are no more complicated or unlikely, you really have done a huge chunk of the work of persuasion. Anything thought about is treated as "in the running", and if other runners seem to fall behind in the race a little, it's assumed that this runner is edging forward or even entering the lead.

And yes, this is just the same fallacy committed, on a much more blatant scale, by the theist who points out that modern science does not offer an absolutely complete explanation of the entire universe, and takes this as evidence for the existence of Jehovah. Rather than Allah, the Flying Spaghetti Monster, or a trillion other gods no less complicated - never mind the space of naturalistic explanations!

To talk about "intelligent design" whenever you point to a purported flaw or open problem in evolutionary theory is, again, privileging the hypothesis - you must have evidence already in hand that points to intelligent design specifically in order to justify raising that particular idea to our attention, rather than a thousand others.

So that's the sane rule. And the corresponding anti-epistemology is to talk endlessly of "possibility" and how you "can't disprove" an idea, to hope that future evidence may confirm it without presenting past evidence already in hand, to dwell and dwell on possibilities without evaluating possibly unfavorable evidence, to draw glowing word-pictures of confirming observations that could happen but haven't happened yet, or to try and show that piece after piece of negative evidence is "not conclusive".

Just as Occam's Razor says that more complicated propositions require more evidence to believe, more complicated propositions also ought to require more work to raise to attention. Just as the principle of burdensome details requires that each part of a belief be separately justified, it requires that each part be separately raised to attention.

As discussed in Perpetual Motion Beliefs, faith and type 2 perpetual motion machines (water -> ice cubes + electricity) have in common that they purport to manufacture improbability from nowhere, whether the improbability of water forming ice cubes or the improbability of arriving at correct beliefs without observation. Sometimes most of the anti-work involved in manufacturing this improbability is getting us to pay attention to an unwarranted belief - thinking on it, dwelling on it. In large answer spaces, attention without evidence is more than halfway to belief without evidence.

Someone who spends all day thinking about whether the Trinity does or does not exist, rather than Allah or Thor or the Flying Spaghetti Monster, is more than halfway to Christianity. If leaving, they're less than half departed; if arriving, they're more than halfway there.

Added: An oft-encountered mode of privilege is to try to make uncertainty within a space, slop outside of that space onto the privileged hypothesis. For example, a creationist seizes on some (allegedly) debated aspect of contemporary theory, argues that scientists are uncertain about evolution, and then says, "We don't really know which theory is right, so maybe intelligent design is right." But the uncertainty is uncertainty within the realm of naturalistic theories of evolution - we have no reason to believe that we'll need to leave that realm to deal with our uncertainty, still less that we would jump out of the realm of standard science and land on Jehovah in particular. That is privileging the hypothesis - taking doubt within a normal space, and trying to slop doubt out of the normal space, onto a privileged (and usually discredited) extremely abnormal target.

Similarly, our uncertainty about where the Born statistics come from, should be uncertainty within the space of quantum theories that are continuous, linear, unitary, slower-than-light, local, causal, naturalistic, etcetera - the usual character of physical law. Some of that uncertainty might slop outside the standard space onto theories that violate one of these standard characteristics. It's indeed possible that we might have to think outside the box. But single-world theories violate all these characteristics, and there is no reason to privilege that hypothesis.

Wiki entry: Privilege the hypothesis.

Sequence: Overly Convenient Excuses

The Third Alternative

But There's Still A Chance, Right?

Referenced by: The Contrarian Status Catch-22 • The Correct Contrarian Cluster • "Outside View!" as Conversation-Halter

Original with comments: Privileging the Hypothesis

I'm Not Saying People Are Stupid

Eliezer Yudkowsky, 09 October 2009 04:23PM

Razib summarized my entire cognitive biases talk at the Singularity Summit 2009 as saying: "Most people are stupid."

Hey! That's a bit unfair. I never said during my talk that most people are stupid. In fact, I was very careful not to say, at any point, that people are stupid, because that's explicitly not what I believe.

I don't think that people who believe in single-world quantum mechanics are stupid. John von Neumann believed in a collapse postulate.

I don't think that philosophers who believe in the "possibility" of zombies are stupid. David Chalmers believes in zombies.

I don't even think that theists are stupid. Robert Aumann believes in Orthodox Judaism.

And in the closing sentence of my talk on cognitive biases and existential risk, I did not say that humanity was devoting more resources to football than existential risk prevention because we were stupid.

There's an old joke that runs as follows:

A motorist is driving past a mental hospital when he gets a flat tire.
He goes out to change the tire, and sees that one of the patients is watching him through the fence.
Nervous, trying to work quickly, he jacks up the car, takes off the wheel, puts the lugnuts into the hubcap -
And steps on the hubcap, sending the lugnuts clattering into a storm drain.
The mental patient is still watching him through the fence.
The motorist desperately looks into the storm drain, but the lugnuts are gone.
The patient is still watching.
The motorist paces back and forth, trying to think of what to do -
And the patient says,
"Take one lugnut off each of the other tires, and you'll have three lugnuts on each."
"That's brilliant!" says the motorist. "What's someone like you doing in an asylum?"
"I'm here because I'm crazy," says the patient, "not because I'm stupid."

Original with comments: I'm Not Saying People Are Stupid

PredictionBook.com - Track your calibration

Eliezer Yudkowsky, 14 October 2009 12:08AM

Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.

The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.

I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)

Original with comments: PredictionBook.com - Track your calibration

Waterloo, ON, Canada Meetup: 6pm Sun Oct 18 '09!

Eliezer Yudkowsky, 15 October 2009 04:30AM

Michael Vassar and I will be attending the Quantum to Cosmos Festival of the Perimeter Institute. Is anyone interested in meeting up at the Symposium Cafe on 4 King St N, Waterloo, ON, Canada, on Sunday at 6pm on October 18th 2009? I might duck out at around 8pm, but Michael Vassar seems more likely to stick around. If we get at least two more positive reply comments (plus the one person who suggested the meetup) then it'll be on and I'll take the question mark off the title. If that time doesn't work for you but you're in the area, feel free to email me about meeting informally.

Result: Okay, we have exactly two people RSVPing (I was hoping for three). We'll show up at the Symposium Cafe at 6pm, possibly walk around or head out to elsewhere by 6:30pm (i.e., join the meetup by 6:30pm if you want to be part of the walking-around group), and we've both got a talk we want to attend at 8:00pm. Either stuff will happen, or not.

Original with comments: Waterloo, ON, Canada Meetup: 6pm Sun Oct 18 '09!

Rationality Quotes: October 2009

Eliezer Yudkowsky, 22 October 2009 04:06PM

A monthly thread for posting rationality-related quotes you've seen recently (or had stored in your quotesfile for ages).

- Please post all quotes separately, so that they can be voted up/down separately. (If they are strongly related, reply to your own comments. If strongly ordered, then fine, post them together.)

- Do not quote yourself.

- Do not quote comments/posts on LW/OB.

- No more than 5 quotes per person per monthly thread, please.

Original with comments: Rationality Quotes: October 2009

Rolf Nelson's "The Rational Entrepreneur"

Eliezer Yudkowsky, 04 November 2009 09:31PM

New blog up by longtime OB/LWer Rolf Nelson, "The Rational Entrepreneur" at rolfnelson.com. On "the overlap between entrepreneurship and the modern tools of rationality". Rolf will post daily through November, and after that it will depend on how much traction the blog gets.

First posts: Pay more attention to the statistics! (which say to do what you know, not what you love) and Having co-founders is valuable but not crucial (according to a study).

Original with comments: Rolf Nelson's "The Rational Entrepreneur"

Hamster in Tutu Shuts Down Large Hadron Collider

Eliezer Yudkowsky, 06 November 2009 04:29PM

The Large Hadron Collider was shut down yesterday by a hamster in a tutu, weary scientists announced.

The Large Hadron Collider is the successor to the earlier Superconducting Super Collider, which was shut down by the US House of Representatives in 1993 after 14 miles of tunnel had been constructed at a cost of $2 billion. Since its inception, the Large Hadron Collider has been plagued by construction delays, dead technicians, broken magnet supports, electrical faults, helium containment failures, vacuum leaks, birds with baguettes, terrorists, ninjas, pirates, supervillains, hurricanes, asteroids, cosmic energy storms, and a runaway train. On one occasion it was discovered that the entire 17-mile circular tunnel had been built upside-down due to a sign error in the calculations, and the whole facility had to be carefully flipped by a giant spatula.

One year ago, hopes were raised for the first time in decades when it was discovered that all the incidents up until that point had been the work of a sinister globe-spanning conspiracy of religious fanatics who, inspired by the term "God Particle", had decided that no one could ever be allowed to look upon the hypothetical Higgs boson. This discovery was widely considered to have undermined the theory that Nature abhors a sufficiently powerful particle collider. Though some found it suspicious that the Higgs boson would even have a religious cult devoted to preventing its observation, the affair did have a patina of surface plausibility - after all, a giant plot to prevent physicists from observing the Higgs boson makes around as much sense as anything else religious people do.

After the conspiracy was shut down by heroic international detectives in an operation so hugely dramatic that it would be pointless to summarize it here, the world began to wonder whether the LHC might really, really work this time around. Scientists everywhere held their breaths as the bodies were cleared out, the tunnels reconditioned, and the broken magnets replaced, all without incident. The price of large hadrons held steady on the commodities market, permitting the LHC's reservoirs to be fully stocked. Proton beams were successfully formed and circulated through the giant tunnel.

Moments before the first collision was scheduled to probe the theretofore-unachieved energy of 3.5 TeV, a hamster in a tutu materialized from nowhere at the intended collision point. The poor creature didn't even have time for a terrified squeak before the two proton beams smashed into it, releasing the equivalent energy of 724 megajoules or 173 kilograms of TNT.

The dispirited scientists of the LHC have announced that this will create a 24-month delay while tiny bits of hamster are cleaned out of the tunnels and anti-hamster-materialization fields are installed in the collider.

At the poorly attended press conference, journalists asked whether it might finally be time to give up.

"Nature's just messing with you, man," said a reporter from the New York Times. "You need to admit this isn't going to work out."

Professor Nicholas von Shnicker, project leader of the LHC, responded.

"NEVER!" shrieked von Shnicker, spittle flying from his lips and spattering on his ragged beard. "Ve vill NEVER give up! My father spent his life trying to make the LHC vork, and his father! Even if it takes a century, if it takes a thousand years or ten thousand million years, VE VILL SEE THE HIGGS BOSON IN OUR LIFETIMES!"

Prof. Kill McBibben is the author of the recently released book Enough, which proposes a new theory of the mysterious Counter-Force that prevents the LHC from operating. "It's not the Higgs boson going back in time," says Prof. McBibben, "nor is it the anthropic principle preventing a black hole from forming. We've just hit the point that we all knew was coming - that we all knew had to happen someday. We've reached the limits of human science. We are just not allowed to build colliders at higher than a certain energy, or know more than a certain amount of particle physics. This is the end of the road. We're done."

Original with comments: Hamster in Tutu Shuts Down Large Hadron Collider

BHTV: Yudkowsky / Robert Greene

Eliezer Yudkowsky, 16 November 2009 08:26PM

Latest Bloggingheads up with Robert Greene, bestselling author of The 48 Laws of Power and most recently The 50th Law ("Fear nothing").

Excellent commentary/summary by Andy McKenzie here.

The most important piece of advice I got from The 50th Law was "always attack before you are ready".

Original with comments: BHTV: Yudkowsky / Robert Greene

Rationality Quotes November 2009

Eliezer Yudkowsky, 29 November 2009 11:36PM

A monthly thread for posting rationality-related quotes you've seen recently (or had stored in your quotesfile for ages).

- Please post all quotes separately, so that they can be voted up/down separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)

- Do not quote yourself.

- Do not quote comments/posts on LW/OB.

- No more than 5 quotes per person per monthly thread, please.

Original with comments: Rationality Quotes November 2009

Frequentist Statistics are Frequently Subjective

Eliezer Yudkowsky, 04 December 2009 08:22PM

Andrew Gelman recently responded to a commenter on the Yudkowsky/Gelman diavlog; the commenter complained that Bayesian statistics were too subjective and lacked rigor. I shall explain why this is unbelievably ironic, but first, the comment itself:

However, the fundamental belief of the Bayesian interpretation, that all probabilities are subjective, is problematic -- for its lack of rigor... One of the features of frequentist statistics is the ease of testability. Consider a binomial variable, like the flip of a fair coin. I can calculate that the probability of getting seven heads in ten flips is 11.71875%... At some point a departure from the predicted value may appear, and frequentist statistics give objective confidence intervals that can precisely quantify the degree to which the coin departs from fairness...

Gelman's first response is "Bayesian probabilities don't have to be subjective." Not sure I can back him on that; probability is ignorance and ignorance is a state of mind (although indeed, some Bayesian probabilities can correspond very directly to observable frequencies in repeatable experiments).

My own response is that frequentist statistics are far more subjective than Bayesian likelihood ratios. Exhibit One is the notion of "statistical significance" (which is what the above comment is actually talking about, although "confidence intervals" have almost the same problem). Steven Goodman offers a nicely illustrated example: Suppose we have at hand a coin, which may be fair (the "null hypothesis") or perhaps biased in some direction. So lo and behold, I flip the coin six times, and I get the result TTTTTH. Is this result statistically significant, and if so, what is the p-value - that is, the probability of obtaining a result at least this extreme?

Well, that depends. Was I planning to flip the coin six times, and count the number of tails? Or was I planning to flip the coin until it came up heads, and count the number of trials? In the first case, the probability of getting "five tails or more" from a fair coin is 11%, while in the second case, the probability of a fair coin requiring "at least five tails before seeing one heads" is 3%.

Whereas a Bayesian looks at the experimental result and says, "I can now calculate the likelihood ratio (evidential flow) between all hypotheses under consideration. Since your state of mind doesn't affect the coin in any way - doesn't change the probability of a fair coin or biased coin producing this exact data - there's no way your private, unobservable state of mind can affect my interpretation of your experimental results."

If you're used to Bayesian methods, it may seem difficult to even imagine that the statistical interpretation of the evidence ought to depend on a factor - namely the experimenter's state of mind - which has no causal connection whatsoever to the experimental result. (Since Bayes says that evidence is about correlation, and no systematic correlation can appear without causal connection; evidence requires entanglement.) How can frequentists manage even in principle to make the evidence depend on the experimenter's state of mind?

It's a complicated story. Roughly, the trick is to make yourself artificially ignorant of the data - instead of knowing the exact experimental result, you pick a class of possible results which includes the actual experimental result, and then pretend that you were told only that the result was somewhere in this class. So if the actual result is TTTTTH, for example, you can pretend that this is part of the class {TTTTTH, TTTTTTH, TTTTTTTH, ...}, a class whose total probability is 3% (1/32). Or if I preferred to have this experimental result not be statistically significant with p < 0.05, I could just as well pretend that some helpful fellow told me only that the result was in the class {TTTTTH, HHHHHT, TTTTTTH, HHHHHHT, ...}, so that the total probability of the class would be 6%, n.s. (In frequentism this question is known as applying a "two-tailed test" or "one-tailed test".)

The arch-Bayesian E. T. Jaynes ruled out this sort of reasoning by telling us that a Bayesian ought only to condition on events that actually happened, not events that could have happened but didn't. (This is not to be confused with the dog that doesn't bark. In this case, the dog was in fact silent; the silence of the dog happened in the real world, not somewhere else. We are rather being told that a Bayesian should not have to worry about alternative possible worlds in which the dog did bark, while estimating the evidence to take from the real world in which the dog did not bark. A Bayesian only worries about the experimental result that was, in fact, obtained; not other experimental results which could have been obtained, but weren't.)

The process of throwing away the actual experimental result, and substituting a class of possible results which contains the actual one - that is, deliberately losing some of your information - introduces a dose of real subjectivity. Colin Begg reports on one medical trial where the data was variously analyzed as having a significance level - that is, probability of the "experimental procedure" producing an "equally extreme result" if the null hypothesis were true - of p=0.051, p=0.001, p=0.083, p=0.28, and p=0.62. Thanks, but I think I'll stick with the conditional probability of the actual experiment producing the actual data.

Frequentists are apparently afraid of the possibility that "subjectivity" - that thing they were accusing Bayesians of - could allow some unspecified terrifying abuse of the scientific process. Do I need to point out the general implications of being allowed to throw away your actual experimental results and substitute a class you made up? In general, if this sort of thing is allowed, I can flip a coin, get 37 heads and 63 tails, and decide that it's part of a class which includes all mixtures with at least 75 heads plus this exact particular sequence. As if I only had the output of a fixed computer program which was written in advance to look at the coinflips and compute a yes-or-no answer as to whether the data is in that class.

Meanwhile, Bayesians are accused of being "too subjective" because we might - gasp! - assign the wrong prior probability to something. First of all, it's obvious from a Bayesian perspective that science papers should be in the business of reporting likelihood ratios, not posterior probabilities - likelihoods add up across experiments, so to get the latest posterior you wouldn't just need a "subjective" prior, you'd also need all the cumulative evidence from other science papers. Now, this accumulation might be a lot more straightforward for a Bayesian than a frequentist, but it's not the sort of thing a typical science paper should have to do. Science papers should report the likelihood ratios for any popular hypotheses - but above all, make the actual raw data available, so the likelihoods can be computed for any hypothesis. (In modern times there is absolutely no excuse for not publishing the raw data, but that's another story.)

And Bayesian likelihoods really are objective - so long as you use the actual exact experimental data, rather than substituting something else.

Meanwhile, over in frequentist-land... what if you told everyone that you had done 127 trials because that was how much data you could afford to collect, but really you kept performing more trials until you got a p-value that you liked, and then stopped? Unless I've got a bug in my test program, a limit of up to 500 trials of a "fair coin" would, 30% of the time, arrive on some step where you could stop and reject the null hypothesis with p<0.05. Or 9% of the time with p<0.01. Of course this requires some degree of scientific dishonesty... or, perhaps, some minor confusion on the scientist's part... since if this is what you are thinking, you're supposed to use a different test of "statistical significance". But it's not like we can actually look inside their heads to find out what the experimenters were thinking. If we're worried about scientific dishonesty, surely we should worry about that? (A similar test program done the Bayesian way, set to stop as soon as finding likelihood ratios of 20/1 and 100/1 relative to an alternative hypothesis that the coin was 55% biased, produced false positives of 3.2% and 0.3% respectively. Unless there was a bug; I didn't spend that much time writing it.)

The actual subjectivity of standard frequentist methods, the ability to manipulate "statistical significance" by choosing different tests, is not a minor problem in science. There are ongoing scandals in medicine and neuroscience from lots of "statistically significant" results failing to replicate. I would point a finger, not just at publication bias, but at scientists armed with powerful statistics packages with lots of complicated tests to run on their data. Complication is really dangerous in science - unfortunately, it looks like instead we have the social rule that throwing around big fancy statistical equations is highly prestigious. (I suspect that some of the opposition to Bayesianism comes from the fact that Bayesianism is too simple.) The obvious fix is to (a) require raw data to be published; (b) require journals to accept papers before the experiment is performed, with the advance paper including a specification of what statistics were selected in advance to be run on the results; (c) raising the standard "significance" level to p<0.0001; and (d) junking all the damned overcomplicated status-seeking impressive nonsense of classical statistics and going to simple understandable Bayesian likelihoods.

Oh, and this frequentist business of "confidence intervals"? Just as subjective as "statistical significance". Let's say I've got a measuring device which returns the true value plus Gaussian noise. If I know you're about to collect 100 results, I can write a computer program such that, before the experiment is run, it's 90% probable that the true value will lie within the interval output by the program.

So I write one program, my friend writes another program, and my enemy writes a third program, all of which make this same guarantee. And in all three cases, the guarantee is true - the program's interval will indeed contain the true value at least 90% of the time, if the experiment returns the true value plus Gaussian noise.

So you run the experiment and feed in the data; and the "confidence intervals" returned are [0.9-1.5], [2.0-2.2], and ["Cheesecake"-"Cheddar"].

The problem may be made clearer by considering the third program, which works as follows: 95% of the time, it does standard frequentist statistics to return an interval which will contain the true value 95% of the time, and on the other 5% of the time, it returns the interval ["Cheesecake"-"Cheddar"]. It is left as an exercise to the reader to show that this program will output an interval containing the true value at least 90% of the time.

BTW, I'm pretty sure I recall reading that "90% confidence intervals" as published in journal papers, in those cases where a true value was later pinned down more precisely, did not contain the true value 90% of the time. So what's the point, even? Just show us the raw data and maybe give us a summary of some likelihoods.

Parapsychology, the control group for science, would seem to be a thriving field with "statistically significant" results aplenty. Oh, sure, the effect sizes are minor. Sure, the effect sizes get even smaller (though still "statistically significant") as they collect more data. Sure, if you find that people can telekinetically influence the future, a similar experimental protocol is likely to produce equally good results for telekinetically influencing the past. Of which I am less tempted to say, "How amazing! The power of the mind is not bound by time or causality!" and more inclined to say, "Bad statistics are time-symmetrical." But here's the thing: Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored - that they are unfairly being held to higher standards than everyone else. I'm willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter. With two-thirds of medical studies in prestigious journals failing to replicate, getting rid of the entire actual subject matter would shrink the field by only 33%. We have to raise the bar high enough to exclude the results claimed by parapsychology under classical frequentist statistics, and then fairly and evenhandedly apply the same bar to the rest of science.

Michael Vassar has a theory that when an academic field encounters advanced statistical methods, it becomes really productive for ten years and then bogs down because the practitioners have learned how to game the rules.

For so long as we do not have infinite computing power, there may yet be a place in science for non-Bayesian statistics. The Netflix Prize was not won by using strictly purely Bayesian methods, updating proper priors to proper posteriors. In that acid test of statistical discernment, what worked best was a gigantic ad-hoc mixture of methods. It may be that if you want to get the most mileage out of your data, in this world where we do not have infinite computing power, you'll have to use some ad-hoc tools from the statistical toolbox - tools that throw away some of the data, that make themselves artificially ignorant, that take all sorts of steps that can't be justified in the general case and that are potentially subject to abuse and that will give wrong answers now and then.

But don't do that, and then turn around and tell me that - of all things! - Bayesian probability theory is too subjective. Probability theory is the math in which the results are theorems and every theorem is compatible with every other theorem and you never get different answers by calculating the same quantity in different ways. To resort to the ad-hoc variable-infested complications of frequentism while preaching your objectivity? I can only compare this with the politicians who go around preaching "Family values!" and then get caught soliciting sex in restrooms. So long as you deliver loud sermons and make a big fuss about painting yourself with the right labels, you get identified with that flag - no one bothers to look very hard at what you do. The case of frequentists calling Bayesians "too subjective" is worth dwelling on for that aspect alone - emphasizing how important it is to look at what's actually going on instead of just listening to the slogans, and how rare it is for anyone to even glance in that direction.

Original with comments: Frequentist Statistics are Frequently Subjective

The Contrarian Status Catch-22

Eliezer Yudkowsky, 19 December 2009 10:40PM

It used to puzzle me that Scott Aaronson still hasn't come to terms with the obvious absurdity of attempts to make quantum mechanics yield a single world.

I should have realized what was going on when I read Scott's blog post "The bullet-swallowers" in which Scott compares many-worlds to libertarianism. But light didn't dawn until my recent diavlog with Scott, where, at 50 minutes and 20 seconds, Scott says:

"What you've forced me to realize, Eliezer, and I thank you for this: What I'm uncomfortable with is not the many-worlds interpretation itself, it's the air of satisfaction that often comes with it."
-- Scott Aaronson, 50:20 in our Bloggingheads dialogue.

It doesn't show on my face (I need to learn to reveal my expressions more, people complain that I'm eerily motionless during these diavlogs) but at this point I'm thinking, Didn't Scott just outright concede the argument? (He didn't; I checked.) I mean, to me this sounds an awful lot like:

Sure, many-worlds is the simplest explanation that fits the facts, but I don't like the people who believe it.

And I strongly suspect that a lot of people out there who would refuse to identify themselves as "atheists" would say almost exactly the same thing:

What I'm uncomfortable with isn't the idea of a god-free physical universe, it's the air of satisfaction that atheists give off.

If you're a regular reader of Robin Hanson, you might essay a Hansonian explanation as follows:

Although the actual state of evidence favors many-worlds (atheism), I don't want to affiliate with other people who say so. They act all brash, arrogant, and offensive, and tend to believe and advocate other odd ideas like libertarianism. If I believed in many-worlds (atheism), that would make me part of this low-prestige group.

Or in simpler psychology:

I don't feel like I belong with the group that believes in many-worlds (atheism).

I think this might form a very general sort of status catch-22 for contrarian ideas.

When a correct contrarian idea comes along, it will have appealing qualities like simplicity and favorable evidence (in the case of factual beliefs) or high expected utility (in the case of policy proposals). When an appealing contrarian idea comes along, it will be initially supported by its appealing qualities, and opposed by the fact that it seems strange and unusual, or any other counterintuitive aspects it may have.

So initially, the group of people who are most likely to support the contrarian idea, are the people who are - among other things - most likely to break with their herd in support of an idea that seems true or right.

These first supporters are likely to be the sort of people who - rather than being careful to speak of the new idea in the cautious tones prudent to supplicate the many high-status insiders who believe otherwise - just go around talking as if the idea had a very high probability, merely because it seems to them like the simplest explanation that fits the facts. "Arrogant", "brash", and "condescending" are some of the terms used to describe people with this poor political sense.

The people first to speak out in support of the new idea will be those less sensitive to conformity; those with more confidence in their sense of truth or rightness; those less afraid of speaking out.

And to the extent these are general character traits, such first supporters are also more likely to advocate other contrarian beliefs, like libertarianism or strident atheism or cryonics.

And once that happens, the only people who'll be willing to believe the idea will be those willing to tar themselves by affiliating with a group of arrogant nonconformists - on top of everything else!

tl;dr: When a counterintuitive new idea comes along, the first people to support it will be contrarians, and so the new idea will become associated with contrarian traits and beliefs, and people will become even more reluctant to believe it because that would affiliate them with low-prestige people/traits/beliefs.

A further remark on "airs of satisfaction": Talk about how we don't understand the Born Probabilities and there are still open questions in QM, and hence we can't accept the no-worldeaters interpretation, sounds a good deal like the criticism given to atheists who go around advocating the no-God interpretation. "But there's so much we don't know about the universe! Why are you so self-satisfied with your disbelief in God?" There's plenty we don't understand about the universe, but that doesn't mean that future discoveries are likely to reveal Jehovah any more than they're likely to reveal a collapse postulate.

Furthermore, atheists are more likely than priests to hear "But we don't know everything about the universe" or "What's with this air of satisfaction?" Similarly, it looks to me like you can get away with speaking out strongly in favor of collapse postulates and against many-worlds, and the same people won't call you on an "air of satisfaction" or say "but what about the open questions in quantum mechanics?"

This is why I think that what we have here is just a sense of someone being too confident in an unusual belief given their assigned social status, rather than a genuine sense that we can't be too confident in any statement whatever. The instinctive status hierarchy treats factual beliefs in pretty much the same way as policy proposals. Just as you need to be extremely high-status to go off and say on your own that the tribe should do something unusual, there's a similar dissonance from a low-status person saying on their own to believe something unusual, without careful compromises with other factions. It shows the one has no sense of their appropriate status in the hierarchy, and isn't sensitive to other factions' interests.

The pure, uncompromising rejection merited by hypotheses like Jehovah or collapse postulates, socially appears as a refusal to make compromises with the majority, or a lack of sufficient humility when contradicting high-prestige people. (Also priests have higher social status to start with; it's understood that their place is to say and advocate these various things; and priests are better at faking humility while going on doing whatever they were going to do anyway.) The Copenhagen interpretation of QM - however ridiculous - is recognized as a conventional non-strange belief, so no one's going to call you insufficiently humble for advocating it. That would mark them as the contrarians.

Referenced by: The Correct Contrarian Cluster

Original with comments: The Contrarian Status Catch-22

Sufficiently Advanced Sanity

Eliezer Yudkowsky, 20 December 2009 06:11PM

Reply to: Shalmanese's Third Law

From an unpublished story confronting Vinge's Law, written in 2004, as abstracted a bit:

"If you met someone who was substantially saner than yourself, how would you know?"

"The obvious mistake that sounds like deep wisdom is claiming that sanity looks like insanity to the insane. I would expect to discover sanity that struck me as wonderfully and surprisingly sane, sanity that shocked me but that I could verify on deeper examination, sanity that sounded wrong but that I could not actually prove to be wrong, and sanity that seemed completely bizarre."

"Like a history of 20th-century science, presented to a scientist from 1900. Much of the future history would sound insane, and easy to argue against. It would take a careful mind to realize none of it was more inconsistent with present knowledge than the scientific history of the 19th century with the knowledge of 1800. Someone who wished to dismiss the whole affair as crackpot would find a thousand excuses ready to hand, plenty of statements that sounded obviously wrong. Yet no crackpot could possibly fake the parts that were obviously right. That is what it is like to meet someone saner. They are not infallible, are not future histories of anything. But no one could counterfeit the wonderfully and surprisingly sane parts; they would need to be that sane themselves."

Spot the Bayesian problem, anyone? It's obvious to me today, but not to the me of 2004. Eliezer_2004 would have seen the structure of the Bayesian problem the moment I pointed it out to him, but he might not have assigned it the same importance I would without a lot of other background.

Original with comments: Sufficiently Advanced Sanity

The Correct Contrarian Cluster

Eliezer Yudkowsky, 21 December 2009 10:01PM

Followup to: Contrarian Status Catch-22

Suppose you know someone believes that the World Trade Center was rigged with explosives on 9/11. What else can you infer about them? Are they more or less likely than average to believe in homeopathy?

I couldn't cite an experiment to verify it, but it seems likely that:

- There are persistent character traits which contribute to someone being willing to state a contrarian point of view.

- All else being equal, if you know that someone advocates one contrarian view, you can infer that they are more likely than average to have other contrarian views.

All sorts of obvious disclaimers can be included here. Someone who expresses an extreme-left contrarian view is less likely to have an extreme-right contrarian view. Different character traits may contribute to expressing contrarian views that are counterintuitive vs. low-prestige vs. anti-establishment etcetera. Nonetheless, it seems likely that you could usefully distinguish a c-factor, a general contrarian factor, in people and beliefs, even though it would break down further on closer examination; there would be a cluster of contrarian people and a cluster of contrarian beliefs, whatever the clusters of the subcluster.

(If you perform a statistical analysis of contrarian ideas and you find that they form distinct subclusters of ideologies that don't correlate with each other, then I'm wrong and no c-factor exists.)

Now, suppose that someone advocates the many-worlds interpretation of quantum mechanics. What else can you infer about them?

Well, one possible reason for believing in the many-worlds interpretation is that, as a general rule of cognitive conduct, you investigated the issue and thought about it carefully; and you learned enough quantum mechanics and probability theory to understand why the no-worldeaters advocates call their theory the strictly simpler one; and you're reflective enough to understand how a deeper theory can undermine your brain's intuition of an apparently single world; and you listen to the physicists who mock many-worlds and correctly assess that these physicists are not to be trusted. Then you believe in many-worlds out of general causes that would operate in other cases - you probably have a high correct contrarian factor - and we can infer that you're more likely to be an atheist.

It's also possible that you thought many-worlds means "all the worlds I can imagine exist" and that you decided it'd be cool if there existed a world where Jesus is Batman, therefore many-worlds is true no matter what the average physicist says. In this case you're just believing for general contrarian reasons, and you're probably more likely to believe in homeopathy as well.

A lot of what we do around here can be thought of as distinguishing the correct contrarian cluster within the contrarian cluster. In fact, when you judge someone's rationality by opinions they post on the Internet - rather than observing their day-to-day decisions or life outcomes - what you're trying to judge is almost entirely cc-factor.

It seems indubitable that, measured in raw bytes, most of the world's correct knowledge is not contrarian correct knowledge, and most of the things that the majority believes (e.g. 2 + 2 = 4) are correct. You might therefore wonder whether it's really important to try to distinguish the Correct Contrarian Cluster in the first place - why not just stick to majoritarianism? The Correct Contrarian Cluster is just the place where the borders of knowledge are currently expanding - not just that, but merely the sections on the border where battles are taking place. Why not just be content with the beauty of settled science? Perhaps we're just trying to signal to our fellow nonconformists, rather than really being concerned with truth, says the little copy of Robin Hanson in my head.

My primary personality, however, responds as follows:

- Religion

- Cryonics

- Diet

In other words, even though you would in theory expect the Correct Contrarian Cluster to be a small fringe of the expansion of knowledge, of concern only to the leading scientists in the field, the actual fact of the matter is that the world is *#$%ing nuts and so there's really important stuff in the Correct Contrarian Cluster. Dietary scientists ignoring their own experimental evidence have killed millions and condemned hundreds of millions more to obesity with high-fructose corn syrup. Not to mention that most people still believe in God. People are crazy, the world is mad. So, yes, if you don't want to bloat up like a balloon and die, distinguishing the Correct Contrarian Cluster is important.

Robin previously posted (and I commented) on the notion of trying to distinguish correct contrarians by "outside indicators" - as I would put it, trying to distinguish correct contrarians, not by analyzing the details of their arguments, but by zooming way out and seeing what sort of general excuse they give for disagreeing with the establishment. As I said in the comments, I am generally pessimistic about the chances of success for this project. Though, as I also commented, there are some general structures that make me sit up and take note; probably the strongest is "These people have ignored their own carefully gathered experimental evidence for decades in favor of stuff that sounds more intuitive." (Robyn Dawes/psychoanalysis, Robin Hanson/medical spending, Gary Taubes/dietary science, Eric Falkenstein/risk-return - note that I don't say anything like this about AI, so this is not a plea I have use for myself!) Mostly, I tend to rely on analyzing the actual arguments; meta should be spice, not meat.

However, failing analysis of actual arguments, another method would be to try and distinguish the Correct Contrarian Cluster by plain old-fashioned... clustering. In a sense, we do this in an ad-hoc way any time we trust someone who seems like a smart contrarian. But it would be possible to do it more formally - write down a big list of contrarian views (some of which we are genuinely uncertain about), poll ten thousand members of the intelligentsia, and look at the clusters. And within the Contrarian Cluster, we find a subcluster where...

...well, how do we look for the Correct Contrarian subcluster?

One obvious way is to start with some things that are slam-dunks, and use them as anchors. Very few things qualify as slam-dunks. Cryonics doesn't rise to that level, since it involves social guesses and values, not just physicalism. I can think of only three slam-dunks off the top of my head:

- Atheism: Yes.

- Many-worlds: Yes.

- "P-zombies": No.

These aren't necessarily simple or easy for contrarians to work through, but the correctness seems as reliable as it gets.

Of course there are also slam-dunks like:

- Natural selection: Yes.

- World Trade Center rigged with explosives: No.

But these probably aren't the right kind of controversy to fine-tune the location of the Correct Contrarian Cluster.

A major problem with the three slam-dunks I listed is that they all seem to have more in common with each other than any of them have with, say, dietary science. This is probably because of the logical, formal character which makes them slam dunks in the first place. By expanding the field somewhat, it would be possible to include slightly less slammed dunks, like:

- Rorschach ink blots: No.

But if we start expanding the list of anchors like this, we run into a much higher probability that one of our anchors is wrong.

So we conduct this massive poll, and we find out that if someone is an atheist and believes in many-worlds and does not believe in p-zombies, they are much more likely than the average contrarian to think that low-energy nuclear reactions (the modern name for cold fusion research) are real. (That is, among "average contrarians" who have opinions on both p-zombies and LENR in the first place!) If I saw this result I would indeed sit up and say, "Maybe I should look into that LENR stuff more deeply." I've never heard of any surveys like this actually being done, but it sounds like quite an interesting dataset to have, if it could be obtained.

There are much more clever things you could do with the dataset. If someone believes most things that atheistic many-worlder zombie-skeptics believe, but isn't a many-worlder, you probably want to know their opinion on infrequently considered topics. (The first thing I'd probably try would be SVD to see if it isolates a "correctness factor", since it's simple and worked famously well on the Netflix dataset.)

But there are also simpler things we could do using the same principle. Let's say we want to know whether the economy will recover, double-dip or crash. So we call up a thousand economists, ask each one "Do you have a strong opinion on whether the many-worlds interpretation is correct?", and see if the economists who have a strong opinion and answer "Yes" have a different average opinion from the average economist and from economists who say "No".

We might not have this data in hand, but it's the algorithm you're approximating when you notice that a lot of smart-seeming people assign much higher than average probabilities to cryonics technology working.

Referenced by: TakeOnIt: Database of Expert Opinions

Original with comments: The Correct Contrarian Cluster

Karma Changes

Eliezer Yudkowsky, 22 December 2009 12:17AM

As recently (re-)suggested by Kaj Sotala, posts now have much larger effects on karma than comments: Each up or down vote on a post is worth 10 karma.

Negative votes on posts have had karma effects all along, but for some reason Reddit's code imposed a display cap (not an actual cap) of 0. This violates a basic user interface principle: things with important effects should have visible effects. Since this just got 10x more important, we now show negative post totals rather than "0". This also provides some feedback to posters that was previously missing. Note that downvoting a post costs 10 karma from your downvote cap of 4x current karma.

The minimum karma to start posting has been raised to 50.

Thanks to our friends at Tricycle for implementing this request!

Original with comments: Karma Changes

The 9/11 Meta-Truther Conspiracy Theory

Eliezer Yudkowsky, 22 December 2009 06:59PM

Date: September 11th, 2001.
Personnel: Unknown [designate A], Unknown [designate B], Unknown [designate C].

A: It's done. The plane targeted at Congress was crashed by those on-board, but the Pentagon and Trade Center attacks occurred just as scheduled.

B: Congress seems sufficiently angry in any case. I don't think the further steps of the plan will meet with any opposition. We should gain the governmental powers we need, and the stock market should move as expected.

A: Good. Have you prepared the conspiracy theorists to accuse us?

B: Yes. All is in readiness. The first accusations will fly within the hour.

C: Er...

A: What is it?

C: Sorry, I know I'm a bit new to this sort of thing, but why are we sponsoring conspiracy theorists? Aren't they our arch-nemeses, tenaciously hunting down and exposing our lies?

A: No, my young apprentice, just the opposite. As soon as you pull off a conspiracy, the first thing you do is start a conspiracy theory about it. Day one.

C: What? We want to be accused of deliberately ignoring intelligence and assassinating that one agent who tried to forward specific information -

A: No, of course not! What you do in a case like this is start an accusation so ridiculous that nobody else wants to be associated with the accusers. You create a low-prestige conspiracy theory and staff it with as many vocal lunatics as you can. That way no one wants to be seen as affiliating with the conspiracy theorists by making a similar accusation.

C: That works? I know I'm not the brightest fish in the barrel - sometimes, hanging around you guys, I feel almost as dumb as I pretend to be - but even I know that "The world's stupidest man may say the sun is shining, but that doesn't make it dark out."

B: Works like a charm, in my experience. Like that business with the Section Magenta aircraft. All you need is a bunch of lunatics screaming about aliens and no one respectable will dream of reporting a "flying saucer" after that.

C: So what did you plan for the 9/11 cover conspiracy theory, by the way? Are the conspiracy theorists going to say the Jews were behind it? Can't get much lower-prestige than anti-Semitism!

B: You've got the right general idea, but you're not thinking creatively enough. Israel does have a clear motive here - even though they weren't in fact behind it - and if the conspiracy theorists cast a wide enough net, they're bound to turn up a handful of facts that seem to support their theory. The public doesn't understand how to discount that sort of "evidence", though, so they might actually be convinced.

C: So... the Illuminati planned the whole operation?

B: You know, for someone who reads as much science fiction as you do, you sure don't think outside the box.

C: ...okay, seriously, man. I don't see how a theory could get any more ridiculous than that and still acquire followers.

(A and B crack up laughing.)

B: Hah! What would you have done to cover up the Section Magenta aircraft, I wonder? Blamed it on Russia? To this day there are still people on the lookout for hidden aliens who overfly populated areas in gigantic non-nanotechnological aircraft with their lights on.

A: So what did we pick for the 9/11 cover conspiracy, by the way?

B: Hm? Oh, the World Trade Center wasn't brought down by planes crashing into it. It was pre-planted explosives.

C: You're kidding me.

B: Seriously, that's the cover conspiracy.

C: There are videos already on the Internet of the planes flying into the World Trade Center. It was on live television. There are thousands of witnesses on the ground who saw it with their own eyes -

B: Right, but the conspiracy theory is, the planes wouldn't have done it on their own - it took pre-planted explosives too.

C: No one is going to buy that. I don't care who you bought out in the conspiracy-theoretic community. This attack would've had the same political effect whether the buildings came down entirely or just the top floors burned. It's not like we spent a lot of time worrying about at what angle the planes would hit the building. The whole point was to keep our hands clean! That's why the al Qaeda plot was such a godsend compared to the original anthrax plan. All we had to do was let it happen. Once we arranged for the attack to go through, we were done, we had no conceivable motive to risk exposure by planting explosives on top of that -

B: Don't take this the wrong way. But one, you don't understand conspiracy theorists at all. Two, they bought the aliens, didn't they? And three, it's already online and the usual crowd of anti-establishment types are already snapping it up.

C: Are you joking?

B: Honest to Betsy. People are claiming that the buildings fell too quickly and that the video showed ejecta corresponding to controlled demolitions.

C: Wow. I don't suppose we actually planted some explosives, just to make sure that -

A: Oh, hell no, son. That sort of thing is never necessary. They'll turn up what looks to them like evidence. They always do.

C: Aren't they going to, um, suspect they're pawns?

A: Human nature 101. Once they've staked their identity on being part of the defiant elect who know the Hidden Truth, there's no way it'll occur to them that they're our catspaws.

B: One reason our fine fraternity has controlled the world for hundreds of years is that we've managed to make "conspiracy theories" look stupid. You know how often you've ever heard someone suggest that possibility? None. You know why? Because it would be a conspiracy theory.

A: Not to mention that the story would be too recursive to catch on. To conceal the truth, one need only make the reality complicated enough to exceed the stack depth of the average newspaper reader.

B: And I've saved the dessert for last.

C: Really?

B: Yeah. You can go totally overboard with these guys. They never notice and they never suspect they're being used.

C: Hit me.

B: We've arranged for them to be called "truthers".

I hereby dub any believers in this theory 9/11 meta-truthers.

I, Eliezer Yudkowsky, do now publicly announce that I am not planning to commit suicide, at any time ever, but particularly not in the next couple of weeks; and moreover, I don't take this possibility seriously myself at the moment, so you would merely be drawing attention to yourselves by assassinating me. However, I also hereby vow that if the Singularity Institute happens to receive donations from any sources totaling at least $3M in 2010, I will take down this post and never publicly speak of the subject again; and if anyone asks, I'll tell them honestly that it was probably a coincidence.

Original with comments: The 9/11 Meta-Truther Conspiracy Theory

Singularity Institute $100K Challenge Grant / 2009 Donations Reminder

Eliezer Yudkowsky, 30 December 2009 12:36AM

In case you missed the notice at the SIAI Blog, the Singularity Institute's next Challenge Grant is now running - $100,000 of matching funds for all donations until February 28th, 2010. If you want your donation to be tax-deductible in the U.S. for 2009, donations postmarked by December 31st will be counted for this tax year.

Original with comments: Singularity Institute $100K Challenge Grant / 2009 Donations Reminder

Drawing Two Aces

Eliezer Yudkowsky, 03 January 2010 10:33AM

Suppose I have a deck of four cards: The ace of spades, the ace of hearts, and two others (say, 2C and 2D).

You draw two cards at random.

Scenario 1: I ask you "Do you have the ace of spades?" You say "Yes." Then the probability that you are holding both aces is 1/3: There are three equiprobable arrangements of cards you could be holding that contain AS, and one of these is AS+AH.

Scenario 2: I ask you "Do you have an ace?" You respond "Yes." The probability you hold both aces is 1/5: There are five arrangements of cards you could be holding (all except 2C+2D) and only one of those arrangements is AS+AH.

Now suppose I ask you "Do you have an ace?"

You say "Yes."

I then say to you: "Choose one of the aces you're holding at random (so if you have only one, pick that one). Is it the ace of spades?"

You reply "Yes."

What is the probability that you hold two aces?

Argument 1: I now know that you are holding at least one ace and that one of the aces you hold is the ace of spades, which is just the same state of knowledge that I obtained in Scenario 1. Therefore the answer must be 1/3.

Argument 2: In Scenario 2, I know that I can hypothetically ask you to choose an ace you hold, and you must hypothetically answer that you chose either the ace of spades or the ace of hearts. My posterior probability that you hold two aces should be the same either way. The expectation of my future probability must equal my current probability: If I expect to change my mind later, I should just give in and change my mind now. Therefore the answer must be 1/5.

Naturally I know which argument is correct. Do you?

Original with comments: Drawing Two Aces

TakeOnIt: Database of Expert Opinions

Eliezer Yudkowsky, 05 January 2010 08:54PM

Ben Albahari wrote to tell us about TakeOnIt, which is trying to build a database of expert opinions. This looks very similar to the data that would be required to locate the Correct Contrarian Cluster - though currently they're building the expert database collaboratively, using quotes, rather than by directly polling the experts on standard topics. Searching for "many worlds" and "zombies" didn't turn up anything as yet; "God" was more productive.

The site is open to the public, you can help catalog expert opinions, and Ben says they're happy to export the data for the use of anyone interested in this research area.

Having this kind of database in standardized form is critical for assessing the track records of experts. TakeOnIt is aware of this.

Original with comments: TakeOnIt: Database of Expert Opinions

High Status and Stupidity: Why?

Eliezer Yudkowsky, 12 January 2010 04:36PM

Michael Vassar once suggested: "Status makes people effectively stupid, as it makes it harder for them to update their public positions without feeling that they are losing face."

To the extent that status does, in fact, make people stupid, this is a rather important phenomenon for a society like ours in which practically all decisions and beliefs pass through the hands of very-high-status individuals (a high "cognitive Gini coefficient").

Does status actually make people stupid? It's hard to say because I haven't tracked many careers over time. I do have a definite and strong impression, with respect to many high-status individuals, that it would have been a lot easier to have an intelligent conversation with them, if I'd approached them before they made it big. But where does that impression come from, since I haven't actually tracked them over time? (Fundamental question of rationality: What do you think you know and how do you think you know it?) My best guess for why my brain seems to believe this: I know it's possible to have intelligent conversations with smart grad students, and I get the strong impression that high-status people used to be those grad students, but now it's much harder to have intelligent conversations with them than with smart grad students.

Hypotheses:

1. Vassar's hypothesis: Higher status increases the amount of face you lose when you change your mind, or increases the cost of losing face.

2. The open-mindedness needed to consider interesting new ideas is (was) only an evolutionary advantage for low-status individuals seeking a good idea to ride to high status. Once high status is achieved, new ideas are high-risk gambles with less relative payoff - the optimal strategy is to be mainstream. I think Robin Hanson had a post about this but I can't recall the title.

3. Intelligence as such is a high-cost feature which is no longer necessary once status is achieved. We can call this the Llinas Hypothesis.

4. High-status individuals were intelligent when they were young; the observed disparity is due solely to the standard declines of aging.

5. High-status individuals spend more time on dinners and politics, and less time on problem-solving and reading; they exercise their minds less.

6. High-status individuals are under less pressure to perform, in general.

7. High-status individuals are just as smart as they ever were, but when you or I try to approach them, the status disparity makes it harder to converse with them - they would sound just as intelligent if we had higher status ourselves.

8. High-status individuals feel less social pressure to listen to your arguments, respond articulately to them, or change their minds when their own arguments are inadequate, which decreases their apparent or real intelligence.

9. High-status individuals become more convinced of their ideas' rightness or of their own competence.

10. High-status individuals get less honest advice from their friends, especially about their own failings.

Did I miss anything important?

Having achieved some small degree of status in certain very limited circles, here's what I do to try to avoid the status-makes-you-stupid effect:

- I try to feel a small flash of self-satisfaction whenever I publicly admit that I am wrong, over what a good rationalist I am being and what a good impression I am making. Not so much satisfaction that I forget that it's better to be correct in the first place, but enough to be a counter-force to the fear of losing face.

- I consistently refuse to be drawn into running the Singularity Institute. I have an overwhelming sense of doom about what happens if I start going down that road.

- I try in general to avoid sending my brain signals which tell it that I am high-status, just in case that causes my brain to decide it is no longer necessary. In fact I try to avoid sending my brain signals which tell it that I have achieved acceptance in my tribe. When my brain begins thinking something that generates a sense of high status within the tribe, I stop thinking that thought.

- I remember my low-prestige roots - for example, I remember what it was like to be a child squeezed through the horrors of the elementary school system. I haven't switched sides to declare that children are stupid and need to be overridden for their own good (as would signal my own adulthood and maturity). I remember the battles I fought then, and would still fight them now on behalf of another child if a target of opportunity arose. That is supporting and identifying with the downtrodden - not to be confused with the high-prestige activity of supporting trendy minority causes that other celebrities support. The vast majority of celebrities who "support the downtrodden" don't go so far as to support children against adults. (David Deutsch is a notable exception to this, and earned a huge amount of respect from me for it.)

- I refuse to conform to people's expectations of a wise sage who always speaks with kindness and sober deliberation, of which I have said: "I am not bloody Gandalf."

Original with comments: High Status and Stupidity: Why?

Meetup: Bay Area: Jan 15th, 7pm

Eliezer Yudkowsky, 13 January 2010 10:02PM

Overcoming Bias / Less Wrong meetup in the San Francisco Bay Area at SIAI House on January 15th, 2010, starting at 7PM.

Robin Hanson and possibly Michael Vassar will be present.

Original with comments: Meetup: Bay Area: Jan 15th, 7pm

Normal Cryonics

Eliezer Yudkowsky, 19 January 2010 07:08PM

I recently attended a small gathering whose purpose was to let young people signed up for cryonics meet older people signed up for cryonics - a matter of some concern to the old guard, for obvious reasons.

The young cryonicists' travel was subsidized. I suspect this led to a greatly different selection filter than usually prevails at conferences of what Robin Hanson would call "contrarians". At an ordinary conference of transhumanists - or libertarians, or atheists - you get activists who want to meet their own kind, strongly enough to pay conference fees and travel expenses. This conference was just young people who took the action of signing up for cryonics, and who were willing to spend a couple of paid days in Florida meeting older cryonicists.

The gathering was 34% female, around half of whom were single, and a few kids. This may sound normal enough, unless you've been to a lot of contrarian-cluster conferences, in which case you just spit coffee all over your computer screen and shouted "WHAT?" I did sometimes hear "my husband persuaded me to sign up", but no more frequently than "I pursuaded my husband to sign up". Around 25% of the people present were from the computer world, 25% from science, and 15% were doing something in music or entertainment - with possible overlap, since I'm working from a show of hands.

I was expecting there to be some nutcases in that room, people who'd signed up for cryonics for just the same reason they subscribed to homeopathy or astrology, i.e., that it sounded cool. None of the younger cryonicists showed any sign of it. There were a couple of older cryonicists who'd gone strange, but none of the young ones that I saw. Only three hands went up that did not identify as atheist/agnostic, and I think those also might have all been old cryonicists. (This is surprising enough to be worth explaining, considering the base rate of insanity versus sanity. Maybe if you're into woo, there is so much more woo that is better optimized for being woo, that no one into woo would give cryonics a second glance.)

The part about actually signing up may also be key - that's probably a ten-to-one or worse filter among people who "get" cryonics. (I put to Bill Faloon of the old guard that probably twice as many people had died while planning to sign up for cryonics eventually, than had actually been suspended; and he said "Way more than that.") Actually signing up is an intense filter for Conscientiousness, since it's mildly tedious (requires multiple copies of papers signed and notarized with witnesses) and there's no peer pressure.

For whatever reason, those young cryonicists seemed really normal - except for one thing, which I'll get to tomorrow. Except for that, then, they seemed like very ordinary people: the couples and the singles, the husbands and the wives and the kids, scientists and programmers and sound studio technicians.

It tears my heart out.

At some future point I ought to post on the notion of belief hysteresis, where you get locked into whatever belief hits you first. So it had previously occurred to me (though I didn't write the post) to argue for cryonics via a conformity reversal test:

If you found yourself in a world where everyone was signed up for cryonics as a matter of routine - including everyone who works at your office - you wouldn't be the first lonely dissenter to earn the incredulous stares of your coworkers by unchecking the box that kept you signed up for cryonics, in exchange for an extra $300 per year.

(Actually it would probably be a lot cheaper, more like $30/year or a free government program, with that economy of scale; but we should ignore that for purposes of the reversal test.)

The point being that if cryonics were taken for granted, it would go on being taken for granted; it is only the state of non-cryonics that is unstable, subject to being disrupted by rational argument.

And this cryonics meetup was that world. It was the world of the ordinary scientists and programmers and sound studio technicians who had signed up for cryonics as a matter of simple common sense.

It tears my heart out.

Those young cryonicists weren't heroes. Most of the older cryonicists were heroes, and of course there were a couple of other heroes among us young folk, like a former employee of Methuselah who'd left to try to put together a startup/nonprofit around a bright idea he'd had for curing cancer (note: even I think this is an acceptable excuse). But most of the younger cryonicists weren't there to fight a desperate battle against Death, they were people who'd signed up for cryonics because it was the obvious thing to do.

And it tears my heart out, because I am a hero and this was like seeing a ray of sunlight from a normal world, some alternate Everett branch of humanity where things really were normal instead of crazy all the goddamned time, a world that was everything this world could be and isn't.

Then there were the children, some of whom had been signed up for cryonics since the day they were born.

It tears my heart out. I'm having trouble remembering to breathe as I write this. My own little brother isn't breathing and never will again.

You know what? I'm going to come out and say it. I've been unsure about saying it, but after attending this event, and talking to the perfectly ordinary parents who signed their kids up for cryonics like the goddamn sane people do, I'm going to come out and say it: If you don't sign up your kids for cryonics then you are a lousy parent.

If you aren't choosing between textbooks and food, then you can afford to sign up your kids for cryonics. I don't know if it's more important than a home without lead paint, or omega-3 fish oil supplements while their brains are maturing, but it's certainly more important than you going to the movies or eating at nice restaurants. That's part of the bargain you signed up for when you became a parent. If you can afford kids at all, you can afford to sign up your kids for cryonics, and if you don't, you are a lousy parent. I'm just back from an event where the normal parents signed their normal kids up for cryonics, and that is the way things are supposed to be and should be, and whatever excuses you're using or thinking of right now, I don't believe in them any more, you're just a lousy parent.

Referenced by: That Magical Click

Original with comments: Normal Cryonics

That Magical Click

Eliezer Yudkowsky, 20 January 2010 04:35PM

Followup to: Normal Cryonics

Yesterday I spoke of that cryonics gathering I recently attended, where travel by young cryonicists was fully subsidized, leading to extremely different demographics from conventions of self-funded activists. 34% female, half of those in couples, many couples with kids - THAT HAD BEEN SIGNED UP FOR CRYONICS FROM BIRTH LIKE A GODDAMNED SANE CIVILIZATION WOULD REQUIRE - 25% computer industry, 25% scientists, 15% entertainment industry at a rough estimate, and in most ways seeming (for smart people) pretty damned normal.

Except for one thing.

During one conversation, I said something about there being no magic in our universe.

And an ordinary-seeming woman responded, "But there are still lots of things science doesn't understand, right?"

Sigh. We all know how this conversation is going to go, right?

So I wearily replied with my usual, "If I'm ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon itself; a blank map does not correspond to a blank territory -"

"Oh," she interrupted excitedly, "so the concept of 'magic' isn't even consistent, then!"

Click.

She got it, just like that.

This was someone else's description of how she got involved in cryonics, as best I can remember it, and it was pretty much typical for the younger generation:

"When I was a very young girl, I was watching TV, and I saw something about cryonics, and it made sense to me - I didn't want to die - so I asked my mother about it. She was very dismissive, but tried to explain what I'd seen; and we talked about some of the other things that can happen to you after you die, like burial or cremation, and it seemed to me like cryonics was better than that. So my mother laughed and said that if I still felt that way when I was older, she wouldn't object. Later, when I was older and signing up for cryonics, she objected."

Click.

It's... kinda frustrating, actually.

There are manifold bad objections to cryonics that can be raised and countered, but the core logic really is simple enough that there's nothing implausible about getting it when you're eight years old (eleven years old, in my case).

Freezing damage? I could go on about modern cryoprotectants and how you can see under a microscope that the tissue is in great shape, and there are experiments underway to see if they can get spontaneous brain activity after vitrifying and devitrifying, and with molecular nanotechnology you could go through the whole vitrified brain atom by atom and do the same sort of information-theoretical tricks that people do to recover hard drive information after "erasure" by any means less extreme than a blowtorch...

But even an eight-year-old can visualize that freezing a sandwich doesn't destroy the sandwich, while cremation does. It so happens that this naive answer remains true after learning the exact details and defeating objections (a few of which are even worth considering), but that doesn't make it any less obvious to an eight-year-old. (I actually did understand the concept of molecular nanotech at eleven, but I could be a special case.)

Similarly: yes, really, life is better than death - just because transhumanists have huge arguments with bioconservatives over this issue, doesn't mean the eight-year-old isn't making the right judgment for the right reasons.

Or: even an eight-year-old who's read a couple of science-fiction stories and who's ever cracked a history book can guess - not for the full reasons in full detail, but still for good reasons - that if you wake up in the Future, it's probably going to be a nicer place to live than the Present.

In short - though it is the sort of thing you ought to review as a teenager and again as an adult - from a rationalist standpoint, there is nothing alarming about clicking on cryonics at age eight... any more than I should worry about my first schism with Orthodox Judaism coming at age five, when they told me that I didn't have to understand the prayers in order for them to work so long as I said them in Hebrew. It really is obvious enough to see as a child, the right thought for the right reasons, no matter how much adult debate surrounds it.

And the frustrating thing was that - judging by this group - most cryonicists are people to whom it was just obvious. (And who then actually followed through and signed up, which is probably a factor-of-ten or worse filter for Conscientiousness.) It would have been convenient if I'd discovered some particular key insight that convinced people. If people had said, "Oh, well, I used to think that cryonics couldn't be plausible if no one else was doing it, but then I read about Asch's conformity experiment and pluralistic ignorance." Then I could just emphasize that argument, and people would sign up.

But the average experience I heard was more like, "Oh, I saw a movie that involved cryonics, and I went on Google to see if there was anything like that in real life, and found Alcor."

In one sense this shouldn't surprise a Bayesian, because the base rate of people who hear a brief mention of cryonics on the radio and have an opportunity to click, will be vastly higher than the base rate of people who are exposed to detailed arguments about cryonics...

Yet the upshot is that - judging from the generation of young cryonicists at that event I attended - cryonics is sustained primarily by the ability of a tiny, tiny fraction of the population to "get it" just from hearing a casual mention on the radio. Whatever part of one-in-a-hundred-thousand isn't accounted for by the Conscientiousness filter.

If I suffered from the sin of underconfidence, I would feel a dull sense of obligation to doubt myself after reaching this conclusion, just like I would feel a dull sense of obligation to doubt that I could be more rational about theology than my parents and teachers at the age of five. As it is, I have no problem with shrugging and saying "People are crazy, the world is mad."

But it really, really raises the question of what the hell is in that click.

There's this magical click that some people get and some people don't, and I don't understand what's in the click. There's the consequentialist/utilitarian click, and the intelligence explosion click, and the life-is-good/death-is-bad click, and the cryonics click. I myself failed to click on one notable occasion, but the topic was probably just as clickable.

(In fact, it took that particular embarrassing failure in my own history - failing to click on metaethics, and seeing in retrospect that the answer was clickable - before I was willing to trust non-click Singularitarians.)

A rationalist faced with an apparently obvious answer, must assign some probability that a non-obvious objection will appear and defeat it. I do know how to explain the above conclusions at great length, and defeat objections, and I would not be nearly as confident (I hope!) if I had just clicked five seconds ago. But sometimes the final answer is the same as the initial guess; if you know the full mathematical story of Peano Arithmetic, 2 + 2 still equals 4 and not 5 or 17 or the color green. And some people very quickly arrive at that same final answer as their best initial guess; they can swiftly guess which answer will end up being the final answer, for what seem even in retrospect like good reasons. Like becoming an atheist at eleven, then listening to a theist's best arguments later in life, and concluding that your initial guess was right for the right reasons.

We can define a "click" as following a very short chain of reasoning, which in the vast majority of other minds is derailed by some detour and proves strongly resistant to re-railing.

What makes it happen? What goes into that click?

It's a question of life-or-death importance, and I don't know the answer.

That generation of cryonicists seemed so normal apart from that...

What's in that click?

The point of the opening anecdote about the Mind Projection Fallacy (blank map != blank territory) is to show (anecdotal) evidence that there's something like a general click-factor, that someone who clicked on cryonics was able to click on mysteriousness=projectivism as well. Of course I didn't expect that I could just stand up amid the conference and describe the intelligence explosion and Friendly AI in a couple of sentences and have everyone get it. That high of a general click factor is extremely rare in my experience, and the people who have it are not otherwise normal. (Michael Vassar is one example of a "superclicker".) But it is still true AFAICT that people who click on one problem are more likely than average to click on another.

My best guess is that clickiness has something to do with failure to compartmentalize - missing, or failing to use, the mental gear that lets human beings believe two contradictory things at the same time. Clicky people would tend to be people who take all of their beliefs at face value.

The Hansonian explanation (not necessarily endorsed by Robin Hanson) would say something about clicky people tending to operate in Near mode. (Why?)

The naively straightforward view would be that the ordinary-seeming people who came to the cryonics did not have any extra gear that magically enabled them to follow a short chain of obvious inferences, but rather, everyone else had at least one extra insanity gear active at the time they heard about cryonics.

Is that really just it? Is there no special sanity to add, but only ordinary madness to take away? Where do superclickers come from - are they just born lacking a whole lot of distractions?

What the hell is in that click?

Original with comments: That Magical Click

Logical Rudeness

Eliezer Yudkowsky, 29 January 2010 06:48AM

The concept of "logical rudeness" (which I'm pretty sure I first found here, HT) is one that I should write more about, one of these days. One develops a sense of the flow of discourse, the give and take of argument. It's possible to do things that completely derail that flow of discourse without shouting or swearing. These may not be considered offenses against politeness, as our so-called "civilization" defines that term. But they are offenses against the cooperative exchange of arguments, or even the rules of engagement with the loyal opposition. They are logically rude.

Suppose, for example, that you're defending X by appealing to Y, and when I seem to be making headway on arguing against Y, you suddenly switch (without having made any concessions) to arguing that it doesn't matter if ~Y because Z still supports X; and when I seem to be making headway on arguing against Z, you suddenly switch to saying that it doesn't matter if ~Z because Y still supports X. This is an example from an actual conversation, with X = "It's okay for me to claim that I'm going to build AGI in five years yet not put any effort into Friendly AI", Y = "All AIs are automatically ethical", and Z = "Friendly AI is clearly too hard since SIAI hasn't solved it yet".

Even if you never scream or shout, this kind of behavior is rather frustrating for the one who has to talk to you. If we are ever to perform the nigh-impossible task of actually updating on the evidence, we ought to acknowledge when we take a hit; the loyal opposition has earned that much from us, surely, even if we haven't yet conceded. If the one is reluctant to take a single hit, let them further defend the point. Swapping in a new argument? That's frustrating. Swapping back and forth? That's downright logically rude, even if you never raise your voice or interrupt.

The key metaphor is flow. Consider the notion of "semantic stopsigns", words that halt thought. A stop sign is something that happens within the flow of traffic. Swapping back and forth between arguments might seem merely frustrating, or rude, if you take the arguments at face value - if you stay on the object level. If you jump back a level of abstraction and try to sense the flow of traffic, and imagine what sort of traffic signal this corresponds to... well, you wouldn't want to run into a traffic signal like that.

Another form of argumentus interruptus is when the other suddenly weakens their claim, without acknowledging the weakening as a concession. Say, you start out by making very strong claims about a God that answers prayers; but when pressed, you retreat back to talking about an impersonal beauty of the universe, without admitting that anything's changed. If you equivocated back and forth between the two definitions, you would be committing an outright logical fallacy - but even if you don't do so, sticking out your neck, and then quickly withdrawing it before anyone can chop it off, is frustrating; it lures someone into writing careful refutations which you then dance back from with a smile; it is logically rude. In the traffic metaphor, it's like offering someone a green light that turns yellow after half a second and leads into a dead end.

So, for example, I'm frustrated if I deal with someone who starts out by making vigorous, contestable, argument-worthy claims implying that the Singularity Institute's mission is unnecessary, impossible, futile, or misguided, and then tries to dance back by saying, "But I still think that what you're doing has a 10% chance of being necessary, which is enough to justify funding your project." Okay, but I'm not arguing with you because I'm worried about my funding getting chopped off, I'm arguing with you because I don't think that 10% is the right number. You said something that was worth arguing with, and then responded by disengaging when I pressed the point; and if I go on contesting the 10% figure, you are somewhat injured, and repeat that you think that what I'm doing is important. And not only is the 10% number still worth contesting, but you originally seemed to be coming on a bit more strongly than that, before you named a weaker-sounding number... It might not be an outright logical fallacy - not until you equivocate between strong claims and weak defenses in the course of the same argument - but it still feels a little frustrating over on the receiving end.

I try not to do this myself. I can't say that arguing with me will always be an enjoyable experience, but I at least endeavor not to be logically rude to the loyal opposition. I stick my neck out so that it can be chopped off if I'm wrong, and when I stick my neck out it stays stuck out, and if I have to withdraw it I'll do so as a visible concession. I may parry - and because I'm human, I may even parry when I shouldn't - but I at least endeavor not to dodge. Where I plant my standard, I have sent an invitation to capture that banner; and I'll stand by that invitation. It's hard enough to count up the balance of arguments without adding fancy dance footwork on top of that.

An awful lot of how people fail at changing their mind seems to have something to do with changing the subject. It might be difficult to point to an outright logical fallacy, but if we have community standards on logical rudeness, we may be able to organize our cognitive traffic a bit less frustratingly.

Added: Checking my notes reminds me to include offering a non-true rejection as a form of logical rudeness. This is where you offer up a reason that isn't really your most important reason, so that, if it's defeated, you'll just switch to something else (which still won't be your most important reason). This is a distinct form of failure from switching Y->Z->Y, but it's also frustrating to deal with; not a logical fallacy outright, but a form of logical rudeness. If someone else is going to the trouble to argue with you, then you should offer up your most important reason for rejection first - something that will make a serious dent in your rejection, if cast down - so that they aren't wasting their time.

Referenced by: You're Entitled to Arguments, But Not (That Particular) Proof • Conversation Halters

Original with comments: Logical Rudeness

You're Entitled to Arguments, But Not (That Particular) Proof

Eliezer Yudkowsky, 15 February 2010 07:58AM

Followup to: Logical Rudeness

"Modern man is so committed to empirical knowledge, that he sets the standard for evidence higher than either side in his disputes can attain, thus suffering his disputes to be settled by philosophical arguments as to which party must be crushed under the burden of proof."
-- Alan Crowe

There's a story - in accordance with Poe's Law, I have no idea whether it's a joke or it actually happened - about a creationist who was trying to claim a "gap" in the fossil record, two species without an intermediate fossil having been discovered. When an intermediate species was discovered, the creationist responded, "Aha! Now there are two gaps."

Since I'm not a professional evolutionary biologist, I couldn't begin to rattle off all the ways that we know evolution is true; true facts tend to leave traces of themselves behind, and evolution is the hugest fact in all of biology. My specialty is the cognitive sciences, so I can tell you of my own knowledge that the human brain looks just like we'd expect it to look if it had evolved, and not at all like you'd think it would look if it'd been intelligently designed. And I'm not really going to say much more on that subject. As I once said to someone who questioned whether humans were really related to apes: "That question might have made sense when Darwin first came up with the hypothesis, but this is the twenty-first century. We can read the genes. Human beings and chimpanzees have 95% shared genetic material. It's over."

Well, it's over, unless you're crazy like a human (ironically, more evidence that the human brain was fashioned by a sloppy and alien god). If you're crazy like a human, you will engage in motivated cognition; and instead of focusing on the unthinkably huge heaps of evidence in favor of evolution, the innumerable signs by which the fact of evolution has left its heavy footprints on all of reality, the uncounted observations that discriminate between the world we'd expect to see if intelligent design ruled and the world we'd expect to see if evolution were true...

...instead you search your mind, and you pick out one form of proof that you think evolutionary biologists can't provide; and you demand, you insist upon that one form of proof; and when it is not provided, you take that as a refutation.

You say, "Have you ever seen an ape species evolving into a human species?" You insist on videotapes - on that particular proof.

And that particular proof is one we couldn't possibly be expected to have on hand; it's a form of evidence we couldn't possibly be expected to be able to provide, even given that evolution is true.

Yet it follows illogically that if a video tape would provide definite proof, then, likewise, the absence of a videotape must constitute definite disproof. Or perhaps just render all other arguments void and turn the issue into a mere matter of personal opinion, with no one's opinion being better than anyone else's.

So far as I can tell, the position of human-caused global warming (anthropogenic global warming aka AGW) has the ball. I get the impression there's a lot of evidence piled up, a lot of people trying and failing to poke holes, and so I have no reason to play contrarian here. It's now heavily politicized science, which means that I take the assertions with a grain of skepticism and worry - well, to be honest I don't spend a whole lot of time worrying about it, because (a) there are worse global catastrophic risks and (b) lots of other people are worrying about AGW already, so there are much better places to invest the next marginal minute of worry.

But if I pretend for a moment to live in the mainstream mental universe in which there is nothing scarier to worry about than global warming, and a 6 °C (11 °F) rise in global temperatures by 2100 seems like a top issue for the care and feeding of humanity's future...

Then I must shake a disapproving finger at anyone who claims the state of evidence on AGW is indefinite.

Sure, if we waited until 2100 to see how much global temperatures increased and how high the seas rose, we would have definite proof. We would have definite proof in 2100, however, and that sounds just a little bit way the hell too late. If there are cost-effective things we can do to mitigate global warming - and by this I don't mean ethanol-from-corn or cap-and-trade, more along the lines of standardizing on a liquid fluoride thorium reactor design and building 10,000 of them - if there's something we can do about AGW, we need to do it now, not in a hundred years.

When the hypothesis at hand makes time valuable - when the proposition at hand, conditional on its being true, means there are certain things we should be doing NOW - then you've got to do your best to figure things out with the evidence that we have. Sure, if we had annual data on global temperatures and CO2 going back to 100 million years ago, we would know more than we do right now. But we don't have that time-series data - not because global-warming advocates destroyed it, or because they were neglectful in gathering it, but because they couldn't possibly be expected to provide it in the first place. And so we've got to look among the observations we can perform, to find those that discriminate between "the way the world could be expected to look if AGW is true / a big problem", and "the way the world would be expected to look if AGW is false / a small problem". If, for example, we discover large deposits of frozen methane clathrates that are released with rising temperatures, this at least seems like "the sort of observation" we might be making if we live in the sort of world where AGW is a big problem. It's not a necessary connection, it's not sufficient on its own, it's something we could potentially also observe in a world where AGW is not a big problem - but unlike the perfect data we can never obtain, it's something we can actually find out, and in fact have found out.

Yes, we've never actually experimented to observe the results over 50 years of artificially adding a large amount of carbon dioxide to the atmosphere. But we know from physics that it's a greenhouse gas. It's not a privileged hypothesis we're pulling out of nowhere. It's not like saying "You can't prove there's no invisible pink unicorn in my garage!" AGW is, ceteris paribus, what we should expect to happen if the other things we believe are true. We don't have any experimental results on what will happen 50 years from now, and so you can't grant the proposition the special, super-strong status of something that has been scientifically confirmed by a replicable experiment. But as I point out in "Scientific Evidence, Legal Evidence, Rational Evidence", if science couldn't say anything about that which has not already been observed, we couldn't ever make scientific predictions by which the theories could be confirmed. Extrapolating from the science we do know, global warming should be occurring; you would need specific experimental evidence to contradict that.

We are, I think, dealing with that old problem of motivated cognition. As Gilovich says: "Conclusions a person does not want to believe are held to a higher standard than conclusions a person wants to believe. In the former case, the person asks if the evidence compels one to accept the conclusion, whereas in the latter case, the person asks instead if the evidence allows one to accept the conclusion." People map the domain of belief onto the social domain of authority, with a qualitative difference between absolute and nonabsolute demands: If a teacher tells you certain things, and you have to believe them, and you have to recite them back on the test. But when a student makes a suggestion in class, you don't have to go along with it - you're free to agree or disagree (it seems) and no one will punish you.

And so the implicit emotional theory is that if something is not proven - better yet, proven using a particular piece of evidence that isn't available and that you're pretty sure is never going to become available - then you are allowed to disbelieve; it's like something a student says, not like something a teacher says.

You demand particular proof P; and if proof P is not available, then you're allowed to disbelieve.

And this is flatly wrong as probability theory.

If the hypothesis at hand is H, and we have access to pieces of evidence E1, E2, and E3, but we do not have access to proof X one way or the other, then the rational probability estimate is the result of the Bayesian update P(H|E1,E2,E3). You do not get to say, "Well, we don't know whether X or ~X, so I'm going to throw E1, E2, and E3 out the window until you tell me about X." I cannot begin to describe how much that is not the way the laws of probability theory work. You do not get to screen off E1, E2, and E3 based on your ignorance of X!

Nor do you get to ignore the arguments that influence the prior probability of H - the standard science by which, ceteris paribus and without anything unknown at work, carbon dioxide is a greenhouse gas and ought to make the Earth hotter.

Nor can you hold up the nonobservation of your particular proof X as a triumphant refutation. If we had time cameras and could look into the past, then indeed, the fact that no one had ever "seen with their own eyes" primates evolving into humans would refute the hypothesis. But, given that time cameras don't exist, then assuming evolution to be true we don't expect anyone to have witnessed humans evolving from apes with our own eyes, for the laws of natural selection require that this have happened far in the distant past. And so, once you have updated on the fact that time cameras don't exist - computed P(Evolution|~Camera) - and the fact that time cameras don't exist hardly seems to refute the theory of evolution - then you obtain no further evidence by observing ~Video, i.e., P(Evolution|~Video,~Camera) = P(Evolution|~Camera). In slogan-form, "The absence of unobtainable proof is not even weak evidence of absence." See appendix for details.

(And while we're on the subject, yes, the laws of probability theory are laws, rather than suggestions. It is like something the teacher tells you, okay? If you're going to ignore the Bayesian update you logically have to perform when you see a new piece of evidence, you might as well ignore outright mathematical proofs. I see no reason why it's any less epistemically sinful to ignore probabilities than to ignore certainties.)

Throwing E1, E2 and E3 out the window, and ignoring the prior probability of H, because you haven't seen unobtainable proof x; or holding up the nonobservation of X as a triumphant refutation, when you couldn't reasonably expect to see X even given that the underlying theory is true; all this is more than just a formal probability-theoretic mistake. It is logically rude.

After all - in the absence of your unobtainable particular proof, there may be plenty of other arguments by which you can hope to figure out whether you live in a world where the hypothesis of interest is true, or alternatively false. It takes work to provide you with those arguments. It takes work to provide you with extrapolations of existing knowledge to prior probabilities, and items of evidence with which to update those prior probabilities, to form a prediction about the unseen. Someone who does the work to provide those arguments is doing the best they can by you; throwing the arguments out the window is not just irrational, but logically rude.

And I emphasize this, because it seems to me that the underlying metaphor of demanding particular proof is to say as if, "You are supposed to provide me with a video of apes evolving into humans, I am entitled to see it with my own eyes, and it is your responsibility to make that happen; and if you do not provide me with that particular proof, you are deficient in your duties of argument, and I have no obligation to believe you." And this is, in the first place, bad math as probability theory. And it is, in the second place, an attitude of trying to be defensible rather than accurate, the attitude of someone who wants to be allowed to retain the beliefs they have, and not the attitude of someone who is honestly curious and trying to figure out which possible world they live in, by whatever signs are available. But if these considerations do not move you, then even in terms of the original and flawed metaphor, you are in the wrong: you are entitled to arguments, but not that particular proof.

Ignoring someone's hard work to provide you with the arguments you need - the extrapolations from existing knowledge to make predictions about events not yet observed, the items of evidence that are suggestive even if not definite and that fit some possible worlds better than others - and instead demanding proof they can't possibly give you, proof they couldn't be expected to provide even if they were right - that is logically rude. It is invalid as probability theory, foolish on the face of it, and logically rude.

And of course if you go so far as to act smug about the absence of an unobtainable proof, or chide the other for their credulity, then you have crossed the line into outright ordinary rudeness as well.

It is likewise a madness of decision theory to hold off pending positive proof until it's too late to do anything; the whole point of decision theory is to choose under conditions of uncertainty, and that is not how the expected value of information is likely to work out. Or in terms of plain common sense: There are signs and portents, smoke alarms and hot doorknobs, by which you can hope to determine whether your house is on fire before your face melts off your skull; and to delay leaving the house until after your face melts off, because only this is the positive and particular proof that you demand, is decision-theoretical insanity. It doesn't matter if you cloak your demand for that unobtainable proof under the heading of scientific procedure, saying, "These are the proofs you could not obtain even if you were right, which I know you will not be able to obtain until the time for action has long passed, which surely any scientist would demand before confirming your proposition as a scientific truth." It's still nuts.

Since this post has already gotten long, I've moved some details of probability theory, the subtext on cryonics, the sub-subtext on molecular nanotechnology, and the sub-sub-subtext on Artificial Intelligence, into:

Demands for Particular Proof: Appendices.

Referenced by: Demands for Particular Proof: Appendices • Undiscriminating Skepticism

Original with comments: You're Entitled to Arguments, But Not (That Particular) Proof

Demands for Particular Proof: Appendices

Eliezer Yudkowsky, 15 February 2010 07:58AM

Appendices to: You're Entitled to Arguments, But Not (That Particular) Proof

(The main article was getting long, so I decided to move the appendices to a separate article which wouldn't be promoted, thus minimizing the size of the article landing in a promoted-article-only-reader's feed.)

A. The absence of unobtainable proof is not even weak evidence of absence.

The wise will already know that absence of evidence actually is evidence of absence; and they may ask, "Since a time-lapse video record of apes evolving into humans would, in fact, be strong evidence in favor of the theory of evolution, is it not mandated by the laws of probability theory that the absence of this videotape constitute some degree of evidence against the theory of evolution?"

(Before you reject that proposition out of hand for containing the substring "evidence against the theory of evolution", bear in mind that grownups understand that evidence accumulates. You don't get to pick out just one piece of evidence and ignore all the rest; true hypotheses can easily generate a minority of weak pieces of evidence against themselves; conceding one point of evidence does not mean conceding the debate; and people who try to act as if it does are nitwits. Also there are probably no creationists reading this blog.)

The laws of probability theory do mandate that if P(H|E) > P(H), then P(H|~E) < P(H). So - even if absence of proof is by no means proof of absence, and even if we reject the philosophy that absence of a particular proof means you get to discard all the other arguments about evidence and priors - must we not at least concede that absence of proof is necessarily evidence of absence, even though it may be very weak evidence?

Actually, in cases like creationism, not even that much follows. Suppose we had a time camera - a device that lets us look into the distant past and see historical events with our own eyes. Let the proposition "we have a time camera" be labeled Camera. Then we would either be able to videotape the evolution of humans from apes; let the presence of this video record be labeled Video, and its absence ~Video. Let Evolution stand for the hypothesis that evolution is true. And let True and False stand for the epistemic states "Pretty much likely to be true" and "Pretty much likely to be false", respectively.

Then, given that evolution is true and that we have a time camera, we should expect to see Video:

P(Video|Evolution,Camera) = True   and   P(Video|~Evolution,Camera) = False

So if we had a time camera, if we could look into the past, then "no one has seen apes evolving into humans" would be strong evidence against the theory of natural selection:

P(Evolution|~Video,Camera) = False

But if we don't have a time camera, then regardless of whether evolution is true or false, we can't expect to have "seen apes evolving into humans":

P(Video|Evolution,~Camera) = False   and   P(Video|~Evolution,~Camera) = False

From which it follows that once you know ~Camera, observing ~Video tells you nothing further about Evolution:

P(Evolution|~Video,~Camera) = P(Evolution|~Camera)

If you didn't know whether or not we had time cameras, and I told you only that no one had ever seen apes evolving into humans, then you would have to evaluate P(Evolution|~Video) which includes some contributions from both P(Evolution|~Video,Camera) and P(Evolution|~Video,~Camera). You don't know whether the video is missing because evolution is false, or the video is missing because we don't have a time camera. And so, as the laws of probability require, P(Evolution|~Video) < P(Evolution) just as P(Evolution|Video) > P(Evolution). But once you know you don't have a time camera, you can start by evaluating P(Evolution|~Camera) - and it's hard to see why lack of time cameras should, in and of itself, be evidence against evolution. The process of natural selection hardly requires a time camera. So P(Evolution|~Camera) = P(Evolution), and then ~Video isn't evidence one way or another. The observation ~Camera screens off any evidence from observing ~Video.

And this is only what should be expected: once you know you don't have a time camera, and once you've updated your views on evolution in light of the fact that time cameras don't exist to begin with (which doesn't seem to have much of an impact on the matter of evolution), it makes no further difference when you learn that no one has ever witnessed apes evolving into humans.

B. Subtext on cryonics: Demanding that cryonicists produce a successful revival before you'll credit the possibility of cryonics, is logically rude; specifically, it is a demand for particular proof.

A successful cryonics revival performed with modern-day technology is not a piece of evidence you could possibly expect modern cryonicists to provide, even given that the proposition of interest is true. The whole point of cryonics is as an ambulance ride to the future; to take advantage of the asymmetry between the technology needed to successfully preserve a patient (cryoprotectants, liquid nitrogen storage) and the technology needed to revive a patient (probably molecular nanotechnology).

In particular, the screening-off condition (playing the role of ~Camera in the example above) is the observation that we presently lack molecular nanotechnology. Given that you don't currently have molecular nanotechnology, you can't reasonably expect to revive a cryonics patient today even given that they could in fact be revived using future molecular nanotechnology.

You are entitled to arguments, though not that particular proof, and cryonicists have done their best to provide you with whatever evidence can be obtained. For example:

A study on rat hippocampal slices showed that it is possible for vitrified slices cooled to a solid state at -130°C to have viability upon re-warming comparable to that of control slices that had not been vitrified or cryopreserved. Ultrastructure of the CA1 region (the region of the brain most vulnerable to ischemic damage) of the re-warmed slices is seen to be quite well preserved compared to the ultrastructure of control CA1 tissue (24). Cryonics organizations perfuse brains with vitrification solution until saturation is achieved...

A rabbit kidney has been vitrified, cooled to -135°C, re-warmed and transplanted into a rabbit. The formerly vitrified transplant functioned well enough as the sole kidney to keep the rabbit alive indefinitely (25)... The vitrification mixture used in preserving the rabbit kidney is known as M22. M22 is used by the cryonics organization Alcor for vitrifying cryonics subjects. Perfusion of rabbits with M22 has been shown to preserve brain ultrastructure without ice formation (26).

This is the sort of evidence we can reasonably expect to obtain today, and it took work to provide you with that evidence. Ignoring it in favor of demanding proof that you couldn't expect to see even if cryonicists were right, is (a) invalid as probability theory, (b) a sign of trying to defend an allowed belief rather than being honestly curious about which possible world we live in, and (c) logically rude.

Formally:

1. Even given that the proposition put forth by cryonicists is true - that people suspended with modern-day technology will be revivable by future technology - you cannot expect them to revive a cryonics patient using modern-day technology.

2. Cryonicists have put forth considerable effort, requiring years of work by many people, to provide you with such evidence as can be obtained today. The lack of that particular proof is not owing to any defect of diligence on the part of cryonicists, or disinterest in their part on doing the research.

3. The prediction that a properly cryoprotected patient does not suffer information-theoretical death is not a privileged hypothesis pulled out of nowhere; it is the default extrapolation from modern neuroscience. If we learn that a patient cryoprotected using current technologies has undergone erasure of critical brain information, we have learned something that is not in current neuroscience textbooks - and actually rather surprising, all things considered. The straight-line extrapolation from the science we do know is that if you can see the neurons nicely preserved under a microscope, the information sure ought to be there. (The idea that critical brain information is stored dynamically in spiking patterns has already been contraindicated by the evidence; dogs taken to very low (above-freezing) temperatures, sufficient to suppress brain activity, do not seem to suffer any memory loss or personality change.)

4. Given that the proposition of interest is true, there is something drastically urgent we ought to be doing RIGHT NOW, namely cryopreserving as many as possible of the 150,000 humans per day who undergo mind-state annihilation. (Economies of scale would very likely drive down costs by an order of magnitude or more; this is an entirely feasible goal economically and technologically, the only question is the political will.)

Given these points: to discard the straight-line extrapolation from modern science and all the hard work that cryonicists have done to provide further distinguishing evidence, in favor of a demand for particular proof that you know cannot possibly be obtained and which you couldn't expect to see even given that the underlying proposition is true, when there are things we ought to be doing NOW given the truth of the proposition and much value will be lost by waiting; all this is indefensible as decision theory in a formal sense, and is, in an informal sense, madness.

Which all goes to say only what Xiaoguang "Mike" Li observed to me some time ago: That saying you'll only sign up for cryonics when someone demonstrates a successful revival of a cryonics patient is sort of like saying that you won't get on the airplane until after it arrives at the destination. Only a very small amount of common sense is necessary to see this, and the objection really does demonstrate the degree to which, when most people feel an innate flinch away from an idea, they hardly feel obligated to come up with objections that make the slightest bit of sense.

This beautiful public service announcement, with only a slight change of metaphor, could serve as a PSA for cryonics. Stop making a big deal out of the decision. It's not that complicated.

C. Demanding the demonstration of a working nanomachine before you'll credit the possibility of molecular nanotechnology is logically rude, specifically, a demand for particular proof.

Given humanity's current level of technology, you can't reasonably expect a demonstration of molecular nanotechnology right now, even given that the proposition of interest is true: that molecular nanotechnology is physically possible to operate, physically possible to manufacture, and likely to be developed within some number of decades. Even if we live in that world, you can't necessarily expect to see nanotechnology now. And yet nonetheless the advocates of nanotechnology have gone to whatever extent possible to provide the arguments by which you could, today, figure out whether or not you live in a world where molecular nanotechnology is possible. Eric Drexler put forth around six years of hard work to produce Nanosystems, doing as much of the basic physics as one man working more or less alone could be expected to do; and since then Robert Freitas, in particular, has been designing and simulating molecular devices, and even trying to work out simple synthesis paths, which is about as much as one person could do, and the funding hasn't really been provided for more than that. To ignore all this hard work that has been put into providing you with such observations and arguments as can be reasonably obtained, and throw them out the window because of a demand for particular proof that you think they can't obtain and that they wouldn't be able to obtain even if the proposition at hand is true - this is not just invalid as probability theory, not just defensiveness rather than curiosity, it is logically rude.

Although actually, of course, you can see tiny molecularly-precise machines on parade, including freely rotating gears and general assemblers - just look inside a biological cell, particularly at ATP synthase and the ribosome. But by that power of cognitive chaos which generates the demand for unobtainable proof in the first place, there can be little doubt that, as soon as this overlooked demonstration is pointed out, the one will immediately find some clause by which to exclude it. To actually provide the unobtainable proof would hardly be fair, after all.

D. It is invalid as probability theory, suboptimal as decision theory, and generally insane, to insist that you want to see someone develop a full-blown Artificial General Intelligence before you'll credit that it's worth anyone's time to work on problems of Friendly AI.

Not precisely analogous to the above cases, but it is a demand for particular proof. Delineating the specifics is left as an exercise to the reader.

Referenced by: You're Entitled to Arguments, But Not (That Particular) Proof • Undiscriminating Skepticism

Original with comments: Demands for Particular Proof: Appendices

Conversation Halters

Eliezer Yudkowsky, 20 February 2010 03:00PM

Related to: Logical Rudeness, Semantic Stopsigns

While working on my book, I found in passing that I'd developed a list of what I started out calling "stonewalls", but have since decided to refer to as "conversation halters". These tactics of argument are distinguished by their being attempts to cut off the flow of debate - which is rarely the wisest way to think, and should certainly rate an alarm bell.

Here's my assembled list, on which I shall expand shortly:

- Appeal to permanent unknowability;

- Appeal to humility;

- Appeal to egalitarianism;

- Appeal to common guilt;

- Appeal to inner privacy;

- Appeal to personal freedom;

- Appeal to arbitrariness;

- Appeal to inescapable assumptions.

- Appeal to unquestionable authority;

- Appeal to absolute certainty.

Now all of these might seem like dodgy moves, some dodgier than others. But they become dodgier still when you take a step back, feel the flow of debate, observe the cognitive traffic signals, and view these as attempts to cut off the flow of further debate.

Hopefully, most of these are obvious, but to define terms:

Appeal to permanent unknowability - something along the lines of "Why did God allow smallpox? Well, no one can know the mind of God." Or, "There's no way to distinguish among interpretations of quantum mechanics, so we'll never know." Arguments like these can be refuted easily enough by anyone who knows the rules for reasoning under uncertainty and how they imply a correct probability estimate given a state of knowledge... but of course you'll probably have to explain the rules to the other, and the reason they appealed to unknowability is probably to cut off further discussion.

Appeal to humility - much the same as above, but said with a different emphasis: "How can we know?", where of course the speaker doesn't much want to know, and so the real meaning is "How can you know?" Of course one may gather entangled evidence in most such cases, and Occam's Razor or extrapolation from already-known facts takes care of the other cases. But you're not likely to get a chance to explain it, because by continuing to speak, you are committing the sin of pride.

Appeal to egalitarianism - something along the lines of "No one's opinion is better than anyone else's." Now if you keep talking you're committing an offense against tribal equality.

Appeal to common guilt - "everyone is irrational now and then", so if you keep talking, you're claiming to be better than them. An implicit subspecies of appeal to egalitarianism.

Appeal to inner privacy - "you can't possibly know how I feel!"  It's true that modern technology still encounters some slight difficulties in reading thoughts out of the brain, though work is underway as we speak. But it is rare that the exact details of how you feel are the key subject matter being disputed. Here the bony borders of the skull are being redeployed as a hard barrier to keep out further arguments.

Appeal to personal freedom - "I can define a word any way I want!" Now if you keep talking you're infringing on their civil rights.

Appeal to arbitrariness - again, the notion that word definitions are arbitrary serves as a good example (in fact I was harvesting some of these appeals from that sequence). It's not just that this is wrong, but that it serves to cut off further discourse. Generally, anything that people are motivated to argue about is not arbitrary. It is being controlled by invisible criteria of evaluation, it has connotations with consequences, and if that isn't true either, the topic of discourse is probably not "arbitrary" but just "meaningless". No map that corresponds to an external territory can be arbitrary.

Appeal to inescapable assumptions - closely related, the idea that you need some assumptions and therefore everyone is free to choose whatever assumptions they want. This again is almost never true. In the realm of physical reality, reality is one way or another and you don't get to make it that way by choosing an opinion, and so some "assumptions" are right and others wrong. In the realm of math, once you choose enough axioms to specify the subject matter, the remaining theorems are matters of logical implication. What I want you to notice is not just that "appeal to inescapable assumptions" is a bad idea, but that it is supposed to halt further conversation.

Appeal to unquestionable authority - for example, defending a definition by appealing to the dictionary, which is supposed to be a final settlement of the argument. Of course it is very rare that whatever is really at stake is something that ought to turn out differently if a Merriam-Webster editor writes a different definition. Only in matters of the solidest, most replicable science, do we have information so authoritative that there is no longer much point in considering other sources of evidence. And even then we shouldn't expect to see strong winds of evidence blowing in an opposing direction - under the Bayesian definition of evidence, strong evidence is just that sort of evidence which you only ever expect to find on at most one side of a factual question. More usually, this argument runs something along the lines of "How dare you argue with the dictionary?" or "How dare you argue with Professor Picklepumper of Harvard University?"

Appeal to absolute certainty - if you did have some source of absolute certainty, it would do no harm to cut off debate at that point. Needless to say, this usually doesn't happen.

And again: These appeals are all flawed in their separate ways, but what I want you to notice is the thing they have in common, the stonewall-effect, the conversation-halting cognitive traffic signal.

The only time it would actually be appropriate to use such a traffic signal is when you have information so strong, or coverage so complete, that there really is no point in further debate. This condition is rarely if ever met. A truly definite series of replicated experiments might settle an issue pending really surprising new experimental results, a la Newton's laws of gravity versus Einstein's GR. Or a gross prior improbability, combined with failure of the advocates to provide confirming evidence in the face of repeated opportunities to do so. Or you might simply run out of time.

But then you should state the stoppage condition outright and plainly, not package it up in one of these appeals. By and large, these traffic signals are simply bad traffic signals.

Referenced by: "Outside View!" as Conversation-Halter

Original with comments: Conversation Halters

"Outside View!" as Conversation-Halter

Eliezer Yudkowsky, 24 February 2010 05:53AM

Followup to: The Outside View's Domain, Conversation Halters
Reply to: Reference class of the unclassreferenceable

In "conversation halters", I pointed out a number of arguments which are particularly pernicious, not just because of their inherent flaws, but because they attempt to chop off further debate - an "argument stops here!" traffic sign, with some implicit penalty (at least in the mind of the speaker) for trying to continue further.

This is not the right traffic signal to send, unless the state of knowledge is such as to make an actual halt a good idea. Maybe if you've got a replicable, replicated series of experiments that squarely target the issue and settle it with strong significance and large effect sizes (or great power and null effects), you could say, "Now we know." Or if the other is blatantly privileging the hypothesis - starting with something improbable, and offering no positive evidence to believe it - then it may be time to throw up hands and walk away. (Privileging the hypothesis is the state people tend to be driven to, when they start with a bad idea and then witness the defeat of all the positive arguments they thought they had.) Or you could simply run out of time, but then you just say, "I'm out of time", not "here the gathering of arguments should end."

But there's also another justification for ending argument-gathering that has recently seen some advocacy on Less Wrong.

An experimental group of subjects were asked to describe highly specific plans for their Christmas shopping: Where, when, and how. On average, this group expected to finish shopping more than a week before Christmas. Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days. Both groups finished an average of 3 days before Christmas. Similarly, Japanese students who expected to finish their essays 10 days before deadline, actually finished 1 day before deadline; and when asked when they had previously completed similar tasks, replied, "1 day before deadline." (See this post.)

Those and similar experiments seem to show us a class of cases where you can do better by asking a certain specific question and then halting: Namely, the students could have produced better estimates by asking themselves "When did I finish last time?" and then ceasing to consider further arguments, without trying to take into account the specifics of where, when, and how they expected to do better than last time.

From this we learn, allegedly, that "the 'outside view' is better than the 'inside view'"; from which it follows that when you're faced with a difficult problem, you should find a reference class of similar cases, use that as your estimate, and deliberately not take into account any arguments about specifics. But this generalization, I fear, is somewhat more questionable...

For example, taw alleged upon this very blog that belief in the 'Singularity' (a term I usually take to refer to the intelligence explosion) ought to be dismissed out of hand, because it is part of the reference class "beliefs in coming of a new world, be it good or evil", with a historical success rate of (allegedly) 0%.

Of course Robin Hanson has a different idea of what constitutes the reference class and so makes a rather different prediction - a problem I refer to as "reference class tennis":

Taking a long historical long view, we see steady total growth rates punctuated by rare transitions when new faster growth modes appeared with little warning. We know of perhaps four such "singularities": animal brains (~600MYA), humans (~2MYA), farming (~1OKYA), and industry (~0.2KYA)...

Excess inside viewing usually continues even after folks are warned that outside viewing works better; after all, inside viewing better show offs inside knowledge and abilities. People usually justify this via reasons why the current case is exceptional. (Remember how all the old rules didn’t apply to the new dotcom economy?) So expect to hear excuses why the next singularity is also an exception where outside view estimates are misleading. Let’s keep an open mind, but a wary open mind.

If I were to play the game of reference class tennis, I'd put recursively self-improving AI in the reference class "huge mother#$%@ing changes in the nature of the optimization game" whose other two instances are the divide between life and nonlife and the divide between human design and evolutionary design; and I'd draw the lesson "If you try to predict that things will just go on sorta the way they did before, you are going to end up looking pathetically overconservative".

And if we do have a local hard takeoff, as I predict, then there will be nothing to say afterward except "This was similar to the origin of life and dissimilar to the invention of agriculture". And if there is a nonlocal economic acceleration, as Robin Hanson predicts, we just say "This was similar to the invention of agriculture and dissimilar to the origin of life". And if nothing happens, as taw seems to predict, then we must say "The whole foofaraw was similar to the apocalypse of Daniel, and dissimilar to the origin of life or the invention of agriculture". This is why I don't like reference class tennis.

But mostly I would simply decline to reason by analogy, preferring to drop back into causal reasoning in order to make weak, vague predictions. In the end, the dawn of recursive self-improvement is not the dawn of life and it is not the dawn of human intelligence, it is the dawn of recursive self-improvement. And it's not the invention of agriculture either, and I am not the prophet Daniel. Point out a "similarity" with this many differences, and reality is liable to respond "So what?"

I sometimes say that the fundamental question of rationality is "Why do you believe what you believe?" or "What do you think you know and how do you think you know it?"

And when you're asking a question like that, one of the most useful tools is zooming in on the map by replacing summary-phrases with the concepts and chains of inferences that they stand for.

Consider what inference we're actually carrying out, when we cry "Outside view!" on a case of a student turning in homework. How do we think we know what we believe?

Our information looks something like this:

- In January 2009, student X_1 predicted they would finish their homework 10 days before deadline, and actually finished 1 day before deadline.

- In February 2009, student X_1 predicted they would finish their homework 9 days before deadline, and actually finished 2 days before deadline.

- In March 2009, student X_1 predicted they would finish their homework 9 days before deadline, and actually finished 1 day before deadline.

- In January 2009, student X_2 predicted they would finish their homework 8 days before deadline, and actually finished 2 days before deadline.

- And so on through 157 other cases.

- Furthermore, in another 121 cases, asking students to visualize specifics actually made them more optimistic.

Therefore, when new student X_279 comes along, even though we've never actually tested them before, we ask:

"How long before deadline did you plan to complete your last three assignments?"

They say: "10 days, 9 days, and 10 days."

We ask: "How long before did you actually complete them?"

They reply: "1 day, 1 day, and 2 days".

We ask: "How long before deadline do you plan to complete this assignment?"

They say: "8 days."

Having gathered this information, we now think we know enough to make this prediction:

"You'll probably finish 1 day before deadline."

They say: "No, this time will be different because -"

We say: "Would you care to make a side bet on that?"

We now believe that previous cases have given us strong, veridical information about how this student functions - how long before deadline they tend to complete assignments - and about the unreliability of the student's planning attempts, as well. The chain of "What do you think you know and how do you think you know it?" is clear and strong, both with respect to the prediction, and with respect to ceasing to gather information. We have historical cases aplenty, and they are all as similar to each other as they are similar to this new case. We might not know all the details of how the inner forces work, but we suspect that it's pretty much the same inner forces inside the black box each time, or the same rough group of inner forces, varying no more in this new case than has been observed on the previous cases that are as similar to each other as they are to this new case, selected by no different a criterion than we used to select this new case. And so we think it'll be the same outcome all over again.

You're just drawing another ball, at random, from the same barrel that produced a lot of similar balls in previous random draws, and those previous balls told you a lot about the barrel. Even if your estimate is a probability distribution rather than a point mass, it's a solid, stable probability distribution based on plenty of samples from a process that is, if not independent and identically distributed, still pretty much blind draws from the same big barrel.

You've got strong information, and it's not that strange to think of stopping and making a prediction.

But now consider the analogous chain of inferences, the what do you think you know and how do you think you know it, of trying to take an outside view on self-improving AI.

What is our data? Well, according to Robin Hanson:

- Animal brains showed up in 550M BC and doubled in size every 34M years

- Human hunters showed up in 2M BC, doubled in population every 230Ky

- Farmers, showing up in 4700BC, doubled every 860 years

- Starting in 1730 or so, the economy started doubling faster, from 58 years in the beginning to a 15-year approximate doubling time now.

From this, Robin extrapolates, the next big growth mode will have a doubling time of 1-2 weeks.

So far we have an interesting argument, though I wouldn't really buy it myself, because the distances of difference are too large... but in any case, Robin then goes on to say: We should accept this estimate flat, we have probably just gathered all the evidence we should use. Taking into account other arguments... well, there's something to be said for considering them, keeping an open mind and all that; but if, foolishly, we actually accept those arguments, our estimates will probably get worse. We might be tempted to try and adjust the estimate Robin has given us, but we should resist that temptation, since it comes from a desire to show off insider knowledge and abilities.

And how do we know that? How do we know this much more interesting proposition that it is now time to stop and make an estimate - that Robin's facts were the relevant arguments, and that other arguments, especially attempts to think about the interior of an AI undergoing recursive self-improvement, are not relevant?

Well... because...

- In January 2009, student X_1 predicted they would finish their homework 10 days before deadline, and actually finished 1 day before deadline.

- In February 2009, student X_1 predicted they would finish their homework 9 days before deadline, and actually finished 2 days before deadline.

- In March 2009, student X_1 predicted they would finish their homework 9 days before deadline, and actually finished 1 day before deadline.

- In January 2009, student X_2 predicted they would finish their homework 8 days before deadline, and actually finished 2 days before deadline...

It seems to me that once you subtract out the scary labels "inside view" and "outside view" and look at what is actually being inferred from what - ask "What do you think you know and how do you think you know it?" - that it doesn't really follow very well. The Outside View that experiment has shown us works better than the Inside View, is pretty far removed from the "Outside View!" that taw cites in support of predicting against any epoch. My own similarity metric puts the latter closer to the analogies of Greek philosophers, actually. And I'd also say that trying to use causal reasoning to produce weak, vague, qualitative predictions like "Eventually, some AI will go FOOM, locally self-improvingly rather than global-economically" is a bit different from "I will complete this homework assignment 10 days before deadline". (The Weak Inside View.)

I don't think that "Outside View! Stop here!" is a good cognitive traffic signal to use so far beyond the realm of homework - or other cases of many draws from the same barrel, no more dissimilar to the next case than to each other, and with similarly structured forces at work in each case.

After all, the wider reference class of cases of telling people to stop gathering arguments, is one of which we should all be wary...

Original with comments: "Outside View!" as Conversation-Halter

The Last Days of the Singularity Challenge

Eliezer Yudkowsky, 26 February 2010 03:46AM

From Michael Anissimov on the Singularity Institute blog:

Thanks to generous contributions by our donors, we are only $11,840 away from fulfilling our $100,000 goal for the 2010 Singularity Research Challenge. For every dollar you contribute to SIAI, another dollar is contributed by our matching donors, who have pledged to match all contributions made before February 28th up to $100,000. That means that this Sunday is your final chance to donate for maximum impact.

Funds from the challenge campaign will be used to support all SIAI activities: our core staff, the Singularity Summit, the Visiting Fellows program, and more. Donors can earmark their funds for specific grant proposals, many of which are targeted towards academic paper-writing, or just contribute to our general fund

[Continue reading at the Singularity Institute blog.]

Original with comments: The Last Days of the Singularity Challenge

Undiscriminating Skepticism

Eliezer Yudkowsky, 14 March 2010 11:23PM

Tl;dr: Since it can be cheap and easy to attack everything your tribe doesn't believe, you shouldn't trust the rationality of just anyone who slams astrology and creationism; these beliefs aren't just false, they're also non-tribal among educated audiences. Test what happens when a "skeptic" argues for a non-tribal belief, or argues against a tribal belief, before you decide they're good general rationalists. This post is intended to be reasonably accessible to outside audiences.

I don't believe in UFOs. I don't believe in astrology. I don't believe in homeopathy. I don't believe in creationism. I don't believe there were explosives planted in the World Trade Center. I don't believe in haunted houses. I don't believe in perpetual motion machines. I believe that all these beliefs are not only wrong but visibly insane.

If you know nothing else about me but this, how much credit should you give me for general rationality?

Certainly anyone who was skillful at adding up evidence, considering alternative explanations, and assessing prior probabilities, would end up disbelieving in all of these.

But there would also be a simpler explanation for my views, a less rare factor that could explain it: I could just be anti-non-mainstream. I could be in the habit of hanging out in moderately educated circles, and know that astrology and homeopathy are not accepted beliefs of my tribe. Or just perceptually recognize them, on a wordless level, as "sounding weird". And I could mock anything that sounds weird and that my fellow tribesfolk don't believe, much as creationists who hang out with fellow creationists mock evolution for its ludicrous assertion that apes give birth to human beings.

You can get cheap credit for rationality by mocking wrong beliefs that everyone in your social circle already believes to be wrong. It wouldn't mean that I have any ability at all to notice a wrong belief that the people around me believe to be right, or vice versa - to further discriminate truth from falsity, beyond the fact that my social circle doesn't already believe in something.

Back in the good old days, there was a simple test for this syndrome that would get quite a lot of mileage: You could just ask me what I thought about God. If I treated the idea with deeper respect than I treated astrology, holding it worthy of serious debate even if I said I disbelieved in it, then you knew that I was taking my cues from my social surroundings - that if the people around me treated a belief as high-prestige, high-status, I wouldn't start mocking it no matter what the state of evidence.

On the other hand suppose I said without hesitation that my epistemic state on God was similar to my epistemic state on psychic powers: no positive evidence, lots of failed tests, highly unfavorable prior, and if you believe it under those circumstances then something is wrong with your mind. Then you would have heard a bit of skepticism that might cost me something socially, and that not everyone around me would have endorsed, even in educated circles. You would know it wasn't just a cheap way of picking up cheap points.

Today the God-test no longer works, because some people realized that the taking-it-seriously aura of religion is in fact the main thing left which prevents people from noticing the epistemic awfulness; there has been a concerted and, I think, well-advised effort to mock religion and strip it of its respectability. The upshot is that there are now quite wide social circles in which God is just another stupid belief that we all know we don't believe in, on the same list with astrology. You could be dealing with an adept rationalist, or you could just be dealing with someone who reads Reddit.

And of course I could easily go on to name some beliefs that others think are wrong and that I think are right, or vice versa, but would inevitably lose some of my audience at each step along the way - just as, a couple of decades ago, I would have lost a lot of my audience by saying that religion was unworthy of serious debate. (Thankfully, today this outright dismissal is at least considered a respectable, mainstream position even if not everyone holds it.)

I probably won't lose much by citing anti-Artificial-Intelligence views as an example of undiscriminating skepticism. I think a majority among educated circles are sympathetic to the argument that brains are not magic and so there is no obstacle in principle to building machines that think. But there are others, albeit in the minority, who recognize Artificial Intelligence as "weird-sounding" and "sci-fi", a belief in something that has never yet been demonstrated, hence unscientific - the same epistemic reference class as believing in aliens or homeopathy.

(This is technically a demand for unobtainable evidence. The asymmetry with homeopathy can be summed up as follows: First: If we learn that Artificial Intelligence is definitely impossible, we must have learned some new fact unknown to modern science - everything we currently know about neurons and the evolution of intelligence suggests that no magic was involved. On the other hand, if we learn that homeopathy is possible, we must have learned some new fact unknown to modern science; if everything else we believe about physics is true, homeopathy shouldn't work. Second: If homeopathy works, we can expect double-blind medical studies to demonstrate its efficacy right now; the absence of this evidence is very strong evidence of absence. If Artificial Intelligence is possible in theory and in practice, we can't necessarily expect its creation to be demonstrated using current knowledge - this absence of evidence is only weak evidence of absence.)

I'm using Artificial Intelligence as an example, because it's a case where you can see some "skeptics" directing their skepticism at a belief that is very popular in educated circles, that is, the nonmysteriousness and ultimate reverse-engineerability of mind. You can even see two skeptical principles brought into conflict - does a good skeptic disbelieve in Artificial Intelligence because it's a load of sci-fi which has never been demonstrated? Or does a good skeptic disbelieve in human exceptionalism, since it would require some mysterious, unanalyzable essence-of-mind unknown to modern science?

It's on questions like these where we find the frontiers of knowledge, and everything now in the settled lands was once on the frontier. It might seem like a matter of little importance to debate weird non-mainstream beliefs; a matter for easy dismissals and open scorn. But if this policy is implemented in full generality, progress goes down the tubes. The mainstream is not completely right, and future science will not just consist of things that sound reasonable to everyone today - there will be at least some things in it that sound weird to us. (This is certainly the case if something along the lines of Artificial Intelligence is considered weird!) And yes, eventually such scientific truths will be established by experiment, but somewhere along the line - before they are definitely established and everyone already believes in them - the testers will need funding.

Being skeptical about some non-mainstream beliefs is not a fringe project of little importance, not always a slam-dunk, not a bit of occasional pointless drudgery - though I can certainly understand why it feels that way to argue with creationists. Skepticism is just the converse of acceptance, and so to be skeptical of a non-mainstream belief is to try to contribute to the project of advancing the borders of the known - to stake an additional epistemic claim that the borders should not expand in this direction, and should advance in some other direction instead.

This is high and difficult work - certainly much more difficult than the work of mocking everything that sounds weird and that the people in your social circle don't already seem to believe.

To put it more formally, before I believe that someone is performing useful cognitive work, I want to know that their skepticism discriminates truth from falsehood, making a contribution over and above the contribution of this-sounds-weird-and-is-not-a-tribal-belief. In Bayesian terms, I want to know that p(mockery|belief false & not a tribal belief) > p(mockery|belief true & not a tribal belief).

If I recall correctly, the US Air Force's Project Blue Book, on UFOs, explained away as a sighting of the planet Venus what turned out to actually be an experimental aircraft. No, I don't believe in UFOs either; but if you're going to explain away experimental aircraft as Venus, then nothing else you say provides further Bayesian evidence against UFOs either. You are merely an undiscriminating skeptic. I don't believe in UFOs, but in order to credit Project Blue Book with additional help in establishing this, I would have to believe that if there were UFOs then Project Blue Book would have turned in a different report.

And so if you're just as skeptical of a weird, non-tribal belief that turns out to have pretty good support, you just blew the whole deal - that is, if I pay any extra attention to your skepticism, it ought to be because I believe you wouldn't mock a weird non-tribal belief that was worthy of debate.

Personally, I think that Michael Shermer blew it by mocking molecular nanotechnology, and Penn and Teller blew it by mocking cryonics (justification: more or less exactly the same reasons I gave for Artificial Intelligence). Conversely, Richard Dawkins scooped up a huge truckload of actual-discriminating-skeptic points, at least in my book, for not making fun of the many-worlds interpretation when he was asked about in an interview; indeed, Dawkins noted (correctly) that the traditional collapse postulate pretty much has to be incorrect. The many-worlds interpretation isn't just the formally simplest explanation that fits the facts, it also sounds weird and is not yet a tribal belief of the educated crowd; so whether someone makes fun of MWI is indeed a good test of whether they understand Occam's Razor or are just mocking everything that's not a tribal belief.

Of course you may not trust me about any of that. And so my purpose today is not to propose a new litmus test to replace atheism.

But I do propose that before you give anyone credit for being a smart, rational skeptic, that you ask them to defend some non-mainstream belief. And no, atheism doesn't count as non-mainstream anymore, no matter what the polls show. It has to be something that most of their social circle doesn't believe, or something that most of their social circle does believe which they think is wrong. Dawkins endorsing many-worlds still counts for now, although its usefulness as an indicator is fading fast... but the point is not to endorse many-worlds, but to see them take some sort of positive stance on where the frontiers of knowledge should change.

Don't get me wrong, there's a whole crazy world out there, and when Richard Dawkins starts whaling on astrology in "The Enemies of Reason" documentary, he is doing good and necessary work. But it's dangerous to let people pick up too much credit just for slamming astrology and homeopathy and UFOs and God. What if they become famous skeptics by picking off the cheap targets, and then use that prestige and credibility to go after nanotechnology? Who will dare to consider cryonics now that it's been featured on an episode of Penn and Teller's "Bullshit"? On the current system you can gain high prestige in the educated circle just by targeting beliefs like astrology that are widely believed to be uneducated; but then the same guns can be turned on new ideas like the many-worlds interpretation, even though it's being actively debated by physicists. And that's why I suggest, not any particular litmus test, but just that you ought to have to stick your neck out and say something a little less usual - say where you are not skeptical (and most of your tribemates are) or where you are skeptical (and most of the people in your tribe are not).

I am minded to pay attention to Robyn Dawes as a skillful rationalist, not because Dawes has slammed easy targets like astrology, but because he also took the lead in assembling and popularizing the total lack of experimental evidence for nearly all schools of psychotherapy and the persistence of multiple superstitions such as Rorschach ink-blot interpretation in the face of literally hundreds of experiments trying and failing to find any evidence for it. It's not that psychotherapy seemed like a difficult target after Dawes got through with it, but that, at the time he attacked it, people in educated circles still thought of it as something that educated people believed in. It's not quite as useful today, but back when Richard Feynman published "Surely You're Joking, Mr. Feynman" you could pick up evidence that he was actually thinking from the fact that he disrespected psychotherapists as well as psychics.

I'll conclude with some simple and non-trustworthy indicators that the skeptic is just filling in a cheap and largely automatic mockery template:

- The "skeptic" opens by remarking about the crazy true believers and wishful thinkers who believe in X, where there seem to be a surprising number of physicists making up the population of those wacky cult victims who believe in X. (The physicist-test is not an infallible indicator of rightness or even non-stupidity, but it's a filter that rapidly picks up on, say, strong AI, molecular nanotechnology, cryonics, the many-worlds interpretation, and so on.) Bonus point losses if the "skeptic" remarks on how easily physicists are seduced by sci-fi ideas. The reason why this is a particularly negative indicator is that when someone is in a mode of automatically arguing against everything that seems weird and isn't a belief of their tribe - of rejecting weird beliefs as a matter of naked perceptual recognition of weirdness - then they tend to perceptually fill-in-the-blank by assuming that anything weird is believed by wacky cult victims (i.e., people Not Of Our Tribe). And they don't backtrack, or wonder otherwise, even if they find out that the "cult" seems to exhibit a surprising number of people who go around talking about rationality and/or members with PhDs in physics. Roughly, they have an automatic template for mocking weird beliefs, and if this requires them to just swap in physicists for astrologers as gullible morons, that's what they'll do. Of course physicists can be gullible morons too, but you should be establishing that as a surprising conclusion, not using it as an opening premise!

- The "skeptic" offers up items of "evidence" against X which are not much less expected in the case that X is true than in the case that X is false; in other words, they fail to grasp the elementary Bayesian notion of evidence. I don't believe that UFOs are alien visitors, but my skepticism has nothing to do with all the crazy people who believe in UFOs - the existence of wacky cults is not much less expected in the case that aliens do exist, than in the case that they do not. (I am skeptical of UFOs, not because I fear affiliating myself with the low-prestige people who believe in UFOs, but because I don't believe aliens would (a) travel across interstellar distances AND (b) hide all signs of their presence AND THEN (c) fly gigantic non-nanotechnological aircraft over our military bases with their exterior lights on.)

- The demand for unobtainable evidence is a special case of the above, and of course a very common mode of skepticism gone wrong. Artificial Intelligence and molecular nanotechnology both involve beliefs in the future feasibility of technologies that we can't build right now, but (arguendo) seem to be strongly permitted by current scientific belief, i.e., the non-ineffability of the brain, or the basic physical calculations which seem to show that simple nanotechnological machines should work. To discard all the arguments from cognitive science and rely on the knockdown argument "no reliable reporter has ever seen an AI!" is blindly filling in the template from haunted houses.

- The "skeptic" tries to scare you away from the belief in their very first opening remarks: for example, pointing out how UFO cults beat and starve their victims (when this can just as easily happen if aliens are visiting the Earth). The negative consequences of a false belief may be real, legitimate truths to be communicated; but only after you establish by other means that the belief is factually false - otherwise it's the logical fallacy of appeal to consequences.

- They mock first and counterargue later or not at all. I do believe there's a place for mockery in the war on dumb ideas, but first you write the crushing factual counterargument, then you conclude with the mockery.

I'll conclude the conclusion by observing that poor skepticism can just as easily exist in a case where a belief is wrong as when a belief is right, so pointing out these flaws in someone's skepticism can hardly serve to establish a positive belief about where the frontiers of knowledge should move.

Original with comments: Undiscriminating Skepticism

Floating Festival Jul 22-25

Eliezer Yudkowsky, 21 July 2010 06:36AM

The Seasteading Institute had to cancel their annual on-water get-together, Ephemerisle, after being quoted ridiculous insurance costs, and it came back as the unofficial Floating Festival - no tickets, no organizers, you just show up with a boat. (Or find someone else who already has or is renting a boat and still has a spare spot, etc.) Jul 22-25 with an unconference (i.e., show up and give a talk) on Saturday the 24th. Posting here because of the large de facto overlap in the communities. The location is about a two-hour drive from the Bay Area.

Main info page.

Referenced by: Bay Area Events Roundup

Original with comments: Floating Festival Jul 22-25

Bay Area Events Roundup

Eliezer Yudkowsky, 24 July 2010 06:04AM

This Saturday (i.e., the 24th, that is, tomorrow) is the peak of the Floating Festival.

Michael Vassar says: "I'm going to be speaking tomorrow (= Saturday July 24th) at 4PM at Bay Area Mensa, in Mountain View, on the scientific method, the history of science, and how to think rationally about most scientific controversies including the Singularity. Less Wrongers are invited to attend. Interested people should email David Verdirame."

The Open Science Summit is July 29-31, in Berkeley.

And as ever, the Singularity Summit approaches on August 14-15 in San Francisco. Now featuring James Randi, Irene Pepperberg, and John Tooby.

Original with comments: Bay Area Events Roundup

LW's first job ad

Eliezer Yudkowsky, 16 September 2010 10:04AM

A friend of the Singularity Institute is seeking to hire someone to research trends and surprises in geopolitics, world economics, and technology - a brainstorming, think-tank type job at a for-profit company. No experience necessary, but strong math and verbal skills required; they're happy to hire out of college and would probably hire out of high school if they find a math-Olympiad type or polymath. This is a job that requires you to think all day and come up with interesting ideas, so they're looking for people who can come up with lots of ideas and criticize them without much external prompting, and enough drive to get their research done without someone standing over their shoulder. They pay well, and it obviously does not involve sales or marketing. They're interested in Less Wrong readers because rationality skills can help. Located in San Francisco. Send résumé and cover letter to yuanshotfirst@gmail.com. Writing sample optional.

Original with comments: LW's first job ad

Table of Contents

The Martial Art of Rationality

Why truth? And...

...What's a bias, again?

The Proper Use of Humility

The Modesty Argument

"I don't know."

A Fable of Science and Politics

Some Claims Are Just Too Extraordinary

Outside the Laboratory

Politics is the Mind-Killer

Just Lose Hope Already

You Are Not Hiring the Top 1%

Policy Debates Should Not Appear One-Sided

Burch's Law

The Scales of Justice, the Notebook of Rationality

Blue or Green on Regulation?

Superstimuli and the Collapse of Western Civilization

Useless Medical Disclaimers

Archimedes's Chronophone

Chronophone Motivations

Self-deception: Hypocrisy or Akrasia?

Tsuyoku Naritai! (I Want To Become Stronger)

Tsuyoku vs. the Egalitarian Instinct

"Statistical Bias"

Useful Statistical Biases

The Error of Crowds

The Majority Is Always Wrong

Knowing About Biases Can Hurt People

Debiasing as Non-Self-Destruction

"Inductive Bias"

Suggested Posts

Futuristic Predictions as Consumable Goods

Marginally Zero-Sum Efforts

Priors as Mathematical Objects

Lotteries: A Waste of Hope

New Improved Lottery

Your Rationality is My Business

Consolidated Nature of Morality Thread

Feeling Rational

Universal Fire

Universal Law

Think Like Reality

Beware the Unsurprised

The Third Alternative

Third Alternatives for Afterlife-ism

Scope Insensitivity

One Life Against the World

Risk-Free Bonds Aren't

Correspondence Bias

Are Your Enemies Innately Evil?

Open Thread

Two More Things to Unlearn from School

Making Beliefs Pay Rent (in Anticipated Experiences)

Belief in Belief

Bayesian Judo

Professing and Cheering

Belief as Attire

Religion's Claim to be Non-Disprovable

The Importance of Saying "Oops"

Focus Your Uncertainty

The Proper Use of Doubt

The Virtue of Narrowness

You Can Face Reality

The Apocalypse Bet

Your Strength as a Rationalist

I Defy the Data!

Absence of Evidence Is Evidence of Absence

Conservation of Expected Evidence

Update Yourself Incrementally

One Argument Against An Army

Hindsight bias

Hindsight Devalues Science

Scientific Evidence, Legal Evidence, Rational Evidence

Is Molecular Nanotechnology "Scientific"?

Fake Explanations

Guessing the Teacher's Password

Science as Attire

Fake Causality

Semantic Stopsigns

Mysterious Answers to Mysterious Questions

The Futility of Emergence

Positive Bias: Look Into the Dark

Say Not "Complexity"

My Wild and Reckless Youth

Failing to Learn from History

Making History Available

Stranger Than History

Explain/Worship/Ignore?

"Science" as Curiosity-Stopper

Absurdity Heuristic, Absurdity Bias

Availability

Why is the Future So Absurd?

Anchoring and Adjustment

The Crackpot Offer

Radical Honesty

We Don't Really Want Your Participation

Applause Lights

Rationality and the English Language

Human Evil and Muddled Thinking

Doublethink (Choosing to be Biased)

Why I'm Blooking

Planning Fallacy

Kahneman's Planning Anecdote

Conjunction Fallacy

Conjunction Controversy (Or, How They Nail It Down)

Burdensome Details

What is Evidence?

The Lens That Sees Its Flaws

How Much Evidence Does It Take?

Einstein's Arrogance

Occam's Razor

9/26 is Petrov Day

How to Convince Me That 2 + 2 = 3

The Bottom Line

What Evidence Filtered Evidence?

Rationalization

Recommended Rationalist Reading

A Rational Argument

We Change Our Minds Less Often Than We Think

Avoiding Your Belief's Real Weak Points

The Meditation on Curiosity

Singlethink

No One Can Exempt You From Rationality's Laws

A Priori

Priming and Contamination

Do We Believe Everything We're Told?

Cached Thoughts

The "Outside the Box" Box

Original Seeing

How to Seem (and Be) Deep

The Logical Fallacy of Generalization from Fictional Evidence

Hold Off On Proposing Solutions

"Can't Say No" Spending

Congratulations to Paris Hilton

Pascal's Mugging: Tiny Probabilities of Vast Utilities

Illusion of Transparency: Why No One Understands You

Self-Anchoring

Expecting Short Inferential Distances

Explainers Shoot High. Aim Low!

Double Illusion of Transparency

No One Knows What Science Doesn't Know

Why Are Individual IQ Differences OK?

Bay Area Bayesians Unite!

Motivated Stopping and Motivated Continuation

Torture vs. Dust Specks

A Case Study of Motivated Continuation

A Terrifying Halloween Costume

Fake Justification

An Alien God

The Wonder of Evolution

Evolutions Are Stupid (But Work Anyway)

Natural Selection's Speed Limit and Complexity Bound

Beware of Stephen J. Gould

The Tragedy of Group Selectionism

Fake Selfishness

Fake Morality

Fake Optimization Criteria

Adaptation-Executers, not Fitness-Maximizers

Evolutionary Psychology

Protein Reinforcement and DNA Consequentialism

Thou Art Godshatter

Terminal Values and Instrumental Values

Evolving to Extinction

No Evolutions for Corporations or Nanodevices

The Simple Math of Everything

Conjuring An Evolution To Serve You

Artificial Addition

Truly Part Of You

Not for the Sake of Happiness (Alone)

Leaky Generalizations

The Hidden Complexity of Wishes

Lost Purposes

Purpose and Pragmatism

The Affect Heuristic

Evaluability (And Cheap Holiday Shopping)

Unbounded Scales, Huge Jury Awards, & Futurism

The Halo Effect

Superhero Bias

Mere Messiahs

Affective Death Spirals

Resist the Happy Death Spiral

Uncritical Supercriticality

Fake Fake Utility Functions

Fake Utility Functions

Evaporative Cooling of Group Beliefs

When None Dare Urge Restraint

The Robbers Cave Experiment

Misc Meta

Every Cause Wants To Be A Cult

Reversed Stupidity Is Not Intelligence

Argument Screens Off Authority

Hug the Query

Guardians of the Truth

Guardians of the Gene Pool

Guardians of Ayn Rand

The Litany Against Gurus

Politics and Awful Art

Two Cult Koans

False Laughter

Effortless Technique

Zen and the Art of Rationality

The Amazing Virgin Pregnancy

Asch's Conformity Experiment

On Expressing Your Concerns

Lonely Dissent

To Lead, You Must Stand Up

Cultish Countercultishness

My Strange Beliefs

Posting on Politics

The Two-Party Swindle

The American System and Misleading Labels

Stop Voting For Nincompoops

Rational vs. Scientific Ev-Psych

A Failed Just-So Story

But There's Still A Chance, Right?

The Fallacy of Gray

Absolute Authority

Infinite Certainty

0 And 1 Are Not Probabilities

Beautiful Math

Expecting Beauty

Is Reality Ugly?

Beautiful Probability

Trust in Math

Rationality Quotes 1

Rationality Quotes 2

Rationality Quotes 3

The Allais Paradox

Zut Allais!

Rationality Quotes 4

Allais Malaise

Against Discount Rates

Circular Altruism

Rationality Quotes 5

Rationality Quotes 6

Rationality Quotes 7

Rationality Quotes 8

Rationality Quotes 9

The "Intuitions" Behind "Utilitarianism"

Trust in Bayes

Something to Protect

Newcomb's Problem and Regret of Rationality

OB Meetup: Millbrae, Thu 21 Feb, 7pm

The Parable of the Dagger

The Parable of Hemlock

Words as Hidden Inferences

Extensions and Intensions

Buy Now Or Forever Hold Your Peace

Similarity Clusters

Typicality and Asymmetrical Similarity

The Cluster Structure of Thingspace

Disguised Queries

Neural Categories

How An Algorithm Feels From Inside

Disputing Definitions

Feel the Meaning

The Argument from Common Usage

Empty Labels

Classic Sichuan in Millbrae, Thu Feb 21, 7pm

Taboo Your Words

Replace the Symbol with the Substance

Fallacies of Compression

Categorizing Has Consequences

Sneaking in Connotations

Arguing "By Definition"

Where to Draw the Boundary?

Entropy, and Short Codes

Mutual Information, and Density in Thingspace

Superexponential Conceptspace, and Simple Words

Leave a Line of Retreat

The Second Law of Thermodynamics, and Engines of Cognition

Perpetual Motion Beliefs

Searching for Bayes-Structure

Conditional Independence, and Naive Bayes

Words as Mental Paintbrush Handles

Rationality Quotes 10

Rationality Quotes 11

Variable Question Fallacies

37 Ways That Words Can Be Wrong

Gary Gygax Annihilated at 69

Dissolving the Question

Wrong Questions

Righting a Wrong Question

Mind Projection Fallacy

Probability is in the Mind

The Quotation is not the Referent

Penguicon & Blook

Qualitatively Confused

Reductionism

Explaining vs. Explaining Away

Fake Reductionism

Savanna Poets

Joy in the Merely Real

Joy in Discovery

Bind Yourself to Reality

If You Demand Magic, Magic Won't Help

New York OB Meetup (ad-hoc) on Monday, Mar 24, @6pm

The Beauty of Settled Science

Amazing Breakthrough Day: April 1st

Is Humanism A Religion-Substitute?

Scarcity

To Spread Science, Keep It Secret

Initiation Ceremony

Hand vs. Fingers

Angry Atoms

Heat vs. Motion

Brain Breakthrough! It's Made of Neurons!

Reductive Reference

Zombies! Zombies?

Zombie Responses

The Generalized Anti-Zombie Principle

GAZP vs. GLUT

Belief in the Implied Invisible

Quantum Explanations

Configurations and Amplitude

Joint Configurations

Distinct Configurations

Where Philosophy Meets Science

Can You Prove Two Particles Are Identical?

Classical Configuration Spaces

The Quantum Arena

Feynman Paths

No Individual Particles

Identity Isn't In Specific Atoms

Zombies: The Movie

Three Dialogues on Identity

Decoherence

The So-Called Heisenberg Uncertainty Principle

Which Basis Is More Fundamental?

Where Physics Meets Experience

Where Experience Confuses Physicists

On Being Decoherent

The Conscious Sorites Paradox

Decoherence is Pointless

Decoherent Essences

The Born Probabilities

Decoherence as Projection

Entangled Photons

Bell's Theorem: No EPR "Reality"

Spooky Action at a Distance: The No-Communication Theorem

Decoherence is Simple

Decoherence is Falsifiable and Testable

Quantum Non-Realism

Collapse Postulates

If Many-Worlds Had Come First

Many Worlds, One Best Guess

The Failures of Eld Science

The Dilemma: Science or Bayes?

Science Doesn't Trust Your Rationality

When Science Can't Help

Science Isn't Strict Enough

Do Scientists Already Know This Stuff?

No Safe Defense, Not Even Science

Changing the Definition of Science

Conference on Global Catastrophic Risks

Faster Than Science

Einstein's Speed

That Alien Message

My Childhood Role Model

Mach's Principle: Anti-Epiphenomenal Physics

A Broken Koan

Relative Configuration Space

Timeless Physics

Timeless Beauty

Timeless Causality

Einstein's Superpowers

Class Project

A Premature Word on AI

The Rhythm of Disagreement

Principles of Disagreement

Timeless Identity

Why Quantum?

Living in Many Worlds

Thou Art Physics

Timeless Control

Bloggingheads: Yudkowsky and Horgan

Against Devil's Advocacy

Eliezer's Post Dependencies; Book Notification; Graphic Designer Wanted

The Quantum Physics Sequence

An Intuitive Explanation of Quantum Mechanics

Quantum Physics Revealed As Non-Mysterious

And the Winner is... Many-Worlds!

Quantum Mechanics and Personal Identity

Causality and Moral Responsibility

Possibility and Could-ness

The Ultimate Source

Passing the Recursive Buck

Grasping Slippery Things

Ghosts in the Machine

LA-602 vs. RHIC Review

Heading Toward Morality

The Outside View's Domain

Surface Analogies and Deep Causes

Optimization and the Singularity

The Psychological Unity of Humankind

The Design Space of Minds-In-General

No Universally Compelling Arguments

2-Place and 1-Place Words

The Opposite Sex

What Would You Do Without Morality?

The Moral Void

Created Already In Motion

I'd take it

The Bedrock of Fairness

2 of 10, not 3 total

Moral Complexities

Is Morality Preference?

Is Morality Given?

Will As Thou Wilt

Where Recursive Justification Hits Bottom

The Fear of Common Knowledge

My Kind of Reflection

The Genetic Fallacy

Fundamental Doubts

Rebelling Within Nature

Probability is Subjectively Objective

Lawrence Watt-Evans's Fiction

Posting May Slow

Whither Moral Progress?

The Gift We Give To Tomorrow

Could Anything Be Right?

Existential Angst Factory

Touching the Old

Should We Ban Physics?

Fake Norms, or "Truth" vs. Truth

When (Not) To Use Probabilities

Can Counterfactuals Be True?

Math is Subjunctively Objective

Does Your Morality Care What You Think?

Changing Your Metaethics

Setting Up Metaethics

The Meaning of Right

Interpersonal Morality

Humans in Funny Suits

Detached Lever Fallacy

A Genius for Destruction

The Comedy of Behaviorism

No Logical Positivist I

Anthropomorphic Optimism

Contaminated by Optimism

Hiroshima Day

Morality as Fixed Computation

Inseparably Right; or, Joy in the Merely Good

Sorting Pebbles Into Correct Heaps

Moral Error and Moral Disagreement

Abstracted Idealized Dynamics

"Arbitrary"

Is Fairness Arbitrary?

The Bedrock of Morality: Arbitrary?

Hot Air Doesn't Disagree

When Anthropomorphism Became Stupid

The Cartoon Guide to Löb's Theorem

Dumb Deplaning

You Provably Can't Trust Yourself

No License To Be Human

Invisible Frameworks

Mirrors and Paintings

Unnatural Categories

Magical Categories

Three Fallacies of Teleology

Dreams of AI Design

Against Modal Logics

Harder Choices Matter Less

Qualitative Strategies of Friendliness

Dreams of Friendliness

Brief Break

Rationality Quotes 12

Rationality Quotes 13

The True Prisoner's Dilemma

The Truly Iterated Prisoner's Dilemma

Rationality Quotes 14

Rationality Quotes 15

Rationality Quotes 16

Singularity Summit 2008

Points of Departure

Rationality Quotes 17

Excluding the Supernatural

Psychic Powers

Optimization

My Childhood Death Spiral

My Best and Worst Mistake

Raised in Technophilia

A Prodigy of Refutation

The Sheer Folly of Callow Youth

Say It Loud

Ban the Bear

How Many LHC Failures Is Too Many?

Horrible LHC Inconsistency

That Tiny Note of Discord

Fighting a Rearguard Action Against the Truth

My Naturalistic Awakening

The Level Above Mine

Competent Elites

Above-Average AI Scientists

Friedman's "Prediction vs. Explanation"

The Magnitude of His Own Folly

Awww, a Zebra

Intrade and the Dow Drop

Trying to Try

Use the Try Harder, Luke

Rationality Quotes 18

Beyond the Reach of God

My Bayesian Enlightenment

Bay Area Meetup for Singularity Summit

On Doing the Impossible

Make an Extraordinary Effort

Shut up and do the impossible!

AIs and Gatekeepers Unite!

Crisis of Faith

The Ritual

Rationality Quotes 19

Why Does Power Corrupt?

Ends Don't Justify Means (Among Humans)

Entangled Truths, Contagious Lies

Traditional Capitalist Values

Dark Side Epistemology

Protected From Myself

Ethical Inhibitions

Ethical Injunctions

Prices or Bindings?

Ethics Notes

Which Parts Are "Me"?

Inner Goodness

San Jose Meetup, Sat 10/25 @ 7:30pm

Expected Creative Surprises

Belief in Intelligence

Aiming at the Target

Measuring Optimization Power

Efficient Cross-Domain Optimization

Economic Definition of Intelligence?

Intelligence in Economics

Mundane Magic

BHTV: Jaron Lanier and Yudkowsky

Building Something Smarter

Complexity and Intelligence

Today's Inspirational Tale

Hanging Out My Speaker's Shingle

Back Up and Ask Whether, Not Why

Recognizing Intelligence

Lawful Creativity

Ask OB: Leaving the Fold

Lawful Uncertainty

Worse Than Random

The Weighted Majority Algorithm

Bay Area Meetup: 11/17 8PM Menlo Park

Selling Nonapples

The Nature of Logic

Boston-area Meetup: 11/18/08 9pm MIT/Cambridge

Logical or Connectionist AI?

Whither OB?

Failure By Analogy

Failure By Affective Analogy

The Weak Inside View

The First World Takeover

Whence Your Abstractions?

Observing Optimization

Life's Story Continues

Surprised by Brains

Cascades, Cycles, Insight...

...Recursion, Magic

The Complete Idiot's Guide to Ad Hominem

Engelbart: Insufficiently Recursive

Total Nano Domination

Thanksgiving Prayer

Chaotic Inversion

Singletons Rule OK

Disappointment in the Future

Recursive Self-Improvement

Hard Takeoff

Permitted Possibilities, & Locality

Underconstrained Abstractions

Sustained Strong Recursion

Is That Your True Rejection?

Artificial Mysterious Intelligence

True Sources of Disagreement

Disjunctions, Antipredictions, Etc.

Bay Area Meetup Wed 12/10 @8pm

The Mechanics of Disagreement

What I Think, If Not Why

You Only Live Twice

BHTV: de Grey and Yudkowsky

For The People Who Are Still Alive

Not Taking Over the World

Visualizing Eutopia

Prolegomena to a Theory of Fun

High Challenge

Complex Novelty

Sensual Experience

Living By Your Own Strength

Rationality Quotes 20

Imaginary Positions

Harmful Options

Devil's Offers

Nonperson Predicates

Nonsentient Optimizers

Nonsentient Bloggers

Can't Unbirth a Child

Amputation of Destiny

Dunbar's Function

A New Day

Free to Optimize

The Uses of Fun (Theory)

Growing Up is Hard

Changing Emotions

Rationality Quotes 21

Emotional Involvement

Rationality Quotes 22

Serious Stories

Rationality Quotes 23

Continuous Improvement

Eutopia is Scary

Building Weirdtopia

She has joined the Conspiracy

Justified Expectation of Pleasant Surprises

Seduced by Imagination

Getting Nearer

In Praise of Boredom

Sympathetic Minds

Interpersonal Entanglement

Failed Utopia #4-2

Investing for the Long Slump

Higher Purpose

Rationality Quotes 24

The Fun Theory Sequence

BHTV: Yudkowsky / Wilkinson

31 Laws of Fun

OB Status Update

Rationality Quotes 25

Value is Fragile

Three Worlds Collide (0/8)

The Baby-Eating Aliens (1/8)

War and/or Peace (2/8)

The Super Happy People (3/8)

Interlude with the Confessor (4/8)

Three Worlds Decide (5/8)

Normal Ending: Last Tears (6/8)

True Ending: Sacrificial Fire (7/8)

Epilogue: Atonement (8/8)

The Thing That I Protect

...And Say No More Of It

(Moral) Truth in Fiction?

Informers and Persuaders

Cynicism in Ev-Psych (and Econ?)

The Evolutionary-Cognitive Boundary

An Especially Elegant Evpsych Experiment

Rationality Quotes 26

An African Folktale

Cynical About Cynicism

Good Idealistic Books are Rare

Against Maturity

Pretending to be Wise

Wise Pretensions v.0

Rationality Quotes 27

Fairness vs. Goodness

On Not Having an Advance Abyssal Plan

Formative Youth

Tell Your Rationalist Origin Story

Markets are Anti-Inductive

Tell Your Rationalist Origin Story... at Less Wrong

The Most Important Thing You Learned

The Most Frequently Useful Thing

That You'd Tell All Your Friends

Unteachable Excellence

Teaching the Unteachable

No, Really, I've Deceived Myself

Belief in Self-Deception

Posting now enabled

Slow down a little... maybe?

Moore's Paradox

Don't Believe You'll Self-Deceive

Striving to Accept

Wanted: Python open source volunteers

Raising the Sanity Waterline

A Sense That More Is Possible

Epistemic Viciousness

Schools Proliferating Without Evidence

3 Levels of Rationality Verification

What Do We Mean By "Rationality"?

Comments for "Rationality"

Rationalist Fiction

Why Our Kind Can't Cooperate

Tolerate Tolerance

You're Calling *Who* A Cult Leader?

BHTV: Yudkowsky & Adam Frank on "religious experience"

On Things that are Awesome

Hyakujo's Fox

The Sacred Mundane

Your Price for Joining

Can Humanism Match Religion's Output?

Church vs. Taskforce

Rationality: Common Interest of Many Causes

Helpless Individuals

Money: The Unit of Caring

Purchase Fuzzies and Utilons Separately

Wrong Tomorrow

Selecting Rationalist Groups

Rationality is Systematized Winning

Incremental Progress and the Valley

Real-Life Anthropic Weirdness

Newcomb's Problem standard positions

Extenuating Circumstances

Of Lies and Black Swan Blowups

Whining-Based Communities

Zero-based karma coming through

Mandatory Secret Identities

Beware of Other-Optimizing

That Crisis thing seems pretty useful

The Unfinished Mystery of the Shangri-La Diet

Akrasia and Shangri-La

Twelve Virtues booklet printing?

Bystander Apathy

Collective Apathy and the Internet

Bayesians vs. Barbarians

Of Gender and Rationality

My Way

Rationality Quotes - April 2009

Great Books of Failure

The Sin of Underconfidence

Well-Kept Gardens Die By Pacifism

Proposal: Use the Wiki for Concepts

Go Forth and Create the Art!

This Didn't Have To Happen

Practical Advice Backed By Deep Theories

Less Meta

The Craft and the Community

The End (of Sequences)

Final Words

Special Status Needs Special Support

Wiki.lesswrong.com Is Live

Willpower Hax #487: Execute by Default

"Open-Mindedness" - the video

"What Is Wrong With Our Thoughts"

This Failing Earth

Anime Explains the Epimenides Paradox

Do Fandoms Need Awfulness?

With whom shall I diavlog?

Honesty: Beyond Internal Truth

ESR's comments on some EY:OB/LW posts

Nonparametric Ethics

Atheism = Untheism + Antitheism

Jul 12 Bay Area meetup - Hanson, Vassar, Yudkowsky

The Strangest Thing An AI Could Tell You

Are You Anosognosic?

Article upvoting

Timeless Decision Theory: Problems I Can't Solve

Of Exclusionary Speech and Gender Politics

The Hero With A Thousand Chances

Unspeakable Morality

Dreams with Damaged Priors

Singularity Summit 2009 (quick post)

Ingredients of Timeless Decision Theory

Timeless Decision Theory and Meta-Circular Decision Theory

Working Mantras

The Sword of Good

Bay Area OBLW Meetup Sep 12

Forcing Anthropics: Boltzmann Brains

Why I'm Staying On Bloggingheads.tv

FHI postdoc at Oxford

Outlawing Anthropics: An Updateless Dilemma

Let Them Debate College Students

The Lifespan Dilemma

The Finale of the Ultimate Meta Mega Crossover

The Anthropic Trilemma

Privileging the Hypothesis

I'm Not Saying People Are Stupid

PredictionBook.com - Track your calibration

Waterloo, ON, Canada Meetup: 6pm Sun Oct 18 '09!

Rationality Quotes: October 2009

Rolf Nelson's "The Rational Entrepreneur"

Hamster in Tutu Shuts Down Large Hadron Collider

BHTV: Yudkowsky / Robert Greene

Rationality Quotes November 2009

Frequentist Statistics are Frequently Subjective

The Contrarian Status Catch-22

Sufficiently Advanced Sanity

The Correct Contrarian Cluster

Karma Changes

The 9/11 Meta-Truther Conspiracy Theory

Singularity Institute $100K Challenge Grant / 2009 Donations Reminder

Drawing Two Aces

TakeOnIt: Database of Expert Opinions

High Status and Stupidity: Why?

Meetup: Bay Area: Jan 15th, 7pm

Normal Cryonics

That Magical Click

Logical Rudeness

You're Entitled to Arguments, But Not (That Particular) Proof

Demands for Particular Proof: Appendices

Conversation Halters

"Outside View!" as Conversation-Halter

The Last Days of the Singularity Challenge

Undiscriminating Skepticism

Floating Festival Jul 22-25

Bay Area Events Roundup

LW's first job ad