Rationality doesn't mean immunity from ideology

[theo-cox]

As a former EAer even as I've diverged from their views I've always respected their seeming culture of critical self-reflection. Sparked by recent discussions I've dug out an article demonstrating that this commitment doesn't run as deep as it seems: https://www.vox.com/future-perfect/23569519/effective-altrusim-sam-bankman-fried-will-macaskill-ea-risk-decentralization-philanthropy

[theo-cox]

@rufuspollock see here

[rufuspollock]

Thanks Theo! Really great piece. Some initial reflections and excerpts (with first pass commentary)

Observations

• "For those inside of EA who are willing to look to new shores: Make the next decade in EA be that of the institutional turn. Here’s one: EA should offer itself as the testing ground for real innovation in institutional decision-making." **Sure ... and i'm dubious re institutional innovation as the solution (structural solutionism basically). The core issues with EA were "ontological / cultural" IMO. So what we need is a transformation there. Specifically, maybe it's time for pragmatic utopianism 😜 https://lifeitself.org/blog/2020/12/21/pragmatic-utopianism
  • TODO: maybe it's time we wrote up that analogy / vision we were working on back in early 2021!
• Longermism plus EV thinking (i.e. crude utilitarianism) is a dangerous combination
• Wisdom vs Intelligence: EA / rationalists ended up with a rather reductive form of intelligence / rationality (caricaturing: it's bayesian updating with a bit of epistemic humility). True rationality is in the Socratic mold and tends to what one could term wisdom, or at least one form of it (hence, of course the very term philosophy - love of wisdom). However, that isn't what one ended up with. Given the meaning of intelligence (and even rationality) in our culture i think an explicit emphasis on wisdom and the qualities it has in addition to intelligence and reason would be important. See https://lifeitself.org/blog/2020/10/24/collective-wisdom-reflections

Excerpts

Point about longtermism and the techno

How exactly did well-intentioned, studious young people once more set out to fix the world only to come back with dirty hands? Unlike others, I do not believe that longtermism — the EA label for caring about the future, which particularly drove Bankman-Fried’s donations — or a too-vigorous attachment to utilitarianism is the root of their miscalculations. A postmortem of the marriage between crypto and EA holds more generalizable lessons and solutions. For one, the approach of doing good by relying on individuals with good intentions — a key pillar of EA — appears ever more flawed. The collapse of FTX is a vindication of the view that institutions, not individuals, must shoulder the job of keeping excessive risk-taking at bay. Institutional designs must shepherd safe collective risk-taking and help navigate decision-making under uncertainty.

Key point about the techno-solutionist SV bent of the movement

Disagreeing with some core assumptions in EAs became rather exhausting. By 2021, my co-author Luke Kemp of the Centre for the Study of Existential Risk at the University of Cambridge and I thought that much of the methodology used in the field of existential risk — a field funded, populated, and driven by EAs — made no sense. So we attempted to publish an article titled “Democratising Risk,” hoping that criticism would give breathing space to alternative approaches. We argued that the idea of a good future as envisioned in Silicon Valley might not be shared across the globe and across time, and that risk had a political dimension. People reasonably disagree on what risks are worth taking, and these political differences should be captured by a fair decision process.

The paper proved to be divisive: Some EAs urged us not to publish, because they thought the academic institutions we were affiliated with might vanish and that our paper could prevent vital EA donations. We spent months defending our claims against surprisingly emotional reactions from EAs, who complained about our use of the term “elitist” or that our paper wasn’t “loving enough.” More concerningly, I received a dozen private messages from EAs thanking me for speaking up publicly or admitting, as one put it: “I was too cowardly to post on the issue publicly for fear that I will get ‘canceled.’”

Comment: i've often spoken about how techo-oriented and weird some of the utopian futures seem to be (e.g. some of the rather "woah there" that's kind of weird pieces by Holden Karnofsky - cf https://www.cold-takes.com/most-important-century/ and specifically e.g. https://www.cold-takes.com/visualizing-utopia/ **UPDATE: she actually flags this as well further down the piece: "They get to project themselves for hours on the 80,000 Hours podcast, their sacrificial superiority in good-doing is hailed as the next level of what it means to be “value-aligned,” and their often incomprehensible fantasies about the future are considered too brilliant to fully grasp.")

People not able to speak out

Maybe I should not have been surprised about the pushback from EAs. One private message to me read: “I’m really disillusioned with EA. There are about 10 people who control nearly all the ‘EA resources.’ However, no one seems to know or talk about this. It’s just so weird. It’s not a disaster waiting to happen, it’s already happened. It increasingly looks like a weird ideological cartel where, if you don’t agree with the power holders, you’re wasting your time trying to get anything done.”

The commonsense foundations

Doing right by your EA peers in 2015 meant that you check out a randomized controlled trial before donating 10 percent of your student budget to combating poverty. I had always refused to assign myself the cringe-worthy label of “effective altruist,” but I too had my few months of a love affair with what I naively thought was my generation’s attempt to apply science to “making the world a better place.” It wasn’t groundbreaking — just commonsensical.

Here, here. This was the kind of stuff i was into as a teenager and just at the end of university. Wanting to do good and thought rationality/science could help. e.g. setting up "What is to be done" to decide on the priorities for the 21st century and how we should fund them (oh the hubris - and in 2002 - well before givewell 😉).

A key point, is the significance of EA from a sociological point of view: it exists and thrives in the purpose vacuum of neoliberalism, where all faith in the state and politics has been dissolved (weirdly given that states are actually doing pretty well - though perhaps suffering badly from a wisdom gap). It is strongly connected both socially (in terms of money and participants) and ideologically to the techno-engineering mindset prevalent in tech. A world where because A/B testing a website works we should be able to solve complex social problems with similar tools.¹

Weird assessment methodologies applied to people

But this changed fast. In 2019, I was leaked a document circulating at the Centre for Effective Altruism, the central coordinating body of the EA movement. Some people in leadership positions were testing a new measure of value to apply to people: a metric called PELTIV, which stood for “Potential Expected Long-Term Instrumental Value.” It was to be used by CEA staff to score attendees of EA conferences, to generate a “database for tracking leads” and identify individuals who were likely to develop high “dedication” to EA — a list that was to be shared across CEA and the career consultancy 80,000 Hours. There were two separate tables, one to assess people who might donate money and one for people who might directly work for EA.

Individuals were to be assessed along dimensions such as “integrity” or “strategic judgment” and “acting on own direction,” but also on “being value-aligned,” “IQ,” and “conscientiousness.” Real names, people I knew, were listed as test cases, and attached to them was a dollar sign (with an exchange rate of 13 PELTIV points = 1,000 “pledge equivalents” = 3 million “aligned dollars”).

Is this creepy / bad because assessing people is just wrong ... or because of what they were prioritizing on (and it's disturbing instrumentalism towards people).

This is a deep point for Life Itself and the whole stage debate and growth hierarchies vs dominator hierarchies. A topic which @theo-cox and I (and others) have gone round a few times - aside/todo: i think getting this out in a discussion thread would be valuable.

Shift in EA principles and priorities over time?

The list showed just how much what it means to be “a good EA” has changed over the years. Early EAs were competing for status by counting the number of mosquito nets they had funded out of their own pocket; later EAs competed on the number of machine learning papers they co-authored at big AI labs.

Going slow is a feature not a bug

All metrics are imperfect. But a small error between a measure of that which is good to do and that which is actually good to do suddenly makes a big difference fast if you’re encouraged to optimize for the proxy. It’s the difference between recklessly sprinting or cautiously stepping in the wrong direction. Going slow is a feature, not a bug.

👍👍 this is a such an important point. And something I / we at Life Itself are seeking to cultivate more. (It also goes to the contention i felt in my short conversation with Daniel Schmachtenberger - where there seemed to be this urgency to do lots of stuff fast because things are bad. In some ways the worse things are, the more we need to ground and go slow ... ²

The dangers of longtermism and EV calculation: you don't know how wrong you can be (there is no updating)

Longtermism and expected value calculations merely provided room for the measure of goodness to wiggle and shape-shift. Futurism gives rationalization air to breathe because it decouples arguments from verification. You might, by chance, be right on how some intervention today affects humans 300 years from now. But if you were wrong, you’ll never know — and neither will your donors. For all their love of Bayesian inference, their endless gesturing at moral uncertainty, and their norms of superficially signposting epistemic humility, EAs became more willing to venture into a far future where they were far more likely to end up in a space so vast and unconstrained that the only feedback to update against was themselves.

EA needs new ideas

For those inside of EA who are willing to look to new shores: Make the next decade in EA be that of the institutional turn. The Economist has argued that EAs now “need new ideas.” Here’s one: EA should offer itself as the testing ground for real innovation in institutional decision-making.

She calls for institutional innovation. Of course useful but rather structural solutionist. The issues with EA were "ontological / cultural" IMO. So what we need is a transformation there. Specifically, maybe it's time for pragmatic utopianism 😜 https://lifeitself.org/blog/2020/12/21/pragmatic-utopianism

And it gets more techno-structural solutionist as it goes on:

Researchers have amassed hundreds of design options for procedures that vary in when, where, and how they elicit experts, deliberate, predict, and vote. There are numerous available technological platforms, such as loomio, panelot, decidim, rxc voice, or pol.is, that facilitate online deliberations at scale and can be adapted to specific contexts. New projects, like the AI Objectives Institute or the Collective Intelligence Project, are brimming with startup energy and need a user base to pilot and iterate with. Let EA groups be a lab for amassing empirical evidence behind what actually works.

Sure you can do this stuff but without "wiser" participants, dealing with our egos etc it probably won't work that well. We need more collective wisdom not just collective intelligence - https://lifeitself.org/blog/2020/10/24/collective-wisdom-reflections

Footnotes

1. Of course, there is some truth to this -- more evidence based policy, medicine etc would be good. But it can blind one to the things that don't fit into that paradigm. ↩

2. I should also cite Bayo Akomolafe's The Times are Urgent: Let’s Slow Down ↩