And why would a rich guy be against a “we are trying to convince rich guys to spend their money differently” organization.
Well when they are just passively trying to convince the rich guys, they can use the organization to launder reputation or boost ideologies they are in favor of. When the organization actually tries to get regulations passed, even ineffectually, well, that is a threat to the likes of Thiel.
The quirky eschatologist that you’re looking for is René Girard, who he personally met at some point. For more details, check out the Behind the Bastards on him.
Thanks for the references. The quirky theology was so outside the range of even the weirder Fundamentalist Christian stuff I didn’t recognize it as such. (And didn’t trust the EA summary because they try so hard to charitably make sense of Thiel).
In this context, Thiel fears the spectre of AGI because it can’t be influenced by his normal approach to power, which is to hide anything that can be hidden and outspend everybody else talking in the open.
Except the EAs are, on net, opposed to the creation of AGI (albeit they are ineffectual in their opposition). So going after the EAs doesn’t make sense if Thiel is genuinely opposed to inventing AGI faster. So I still think Thiel is just going after the EA’s because he’s libertarian and EA has shifted in the direction of trying to get more government regulation. (As opposed to a coherent theological goal beyond libertarianism). I’ll check out the BtB podcast and see if it changes my mind as to his exact flavor of insanity.
So… apparently Peter Thiel has taken to co-opting fundamentalist Christian terminology to go after Effective Altruism? At least it seems that way from this EA post (warning, I took psychic damage just skimming the lunacy). As far as I can tell, he’s merely co-opting the terminology, Thiel’s blather doesn’t have any connection to any variant of Christian eschatology (whether mainstream or fundamentalist or even obscure wacky fundamentalist), but of course, the majority of the EAs don’t recognize that, or the fact that he is probably targeting them for their (kind of weak to be honest) attempts at getting AI regulated at all, and instead they charitably try to steelman him and figure out if he was a legitimate point. …I wish they could put a tenth of this effort into understanding leftist thought.
Some of the comments are… okay actually, at least by EA standards, but there are still plenty of people willing to defend Thiel
One comment notes some confusion:
I’m still confused about the overall shape of what Thiel believes.
He’s concerned about the antichrist opposing Jesus during Armageddon. But afaik standard theology says that Jesus will win for certain. And revelation says the world will be in disarray and moral decay when the Second Coming happens.
If chaos is inevitable and necessary for Jesus’ return, why is expanding the pre-apocalyptic era with growth/prosperity so important to him?
Yeah, its because he is simply borrowing Christian Fundamentalists Eschatological terminology… possibly to try to turn the Christofascists against EA?
I’m dubious Thiel is actually an ally to anyone worried about permanent dictatorship. He has connections to openly anti-democratic neoreactionaries like Curtis Yarvin, he quotes Nazi lawyer and democracy critic Carl Schmitt on how moments of greatness in politics are when you see your enemy as an enemy, and one of the most famous things he ever said is “I no longer believe that freedom and democracy are compatible”. Rather I think he is using “totalitarian” to refer to any situation where the government is less economically libertarian than he would like, or “woke” ideas are popular amongst elite tastemakers, even if the polity this is all occurring in is clearly a liberal democracy, not a totalitarian state.
Note this commenter still uses non-confrontational language (“I’m dubious”) even when directly calling Thiel out.
The top comment, though, is just like the main post, extending charitability to complete technofascist insanity. (Warning for psychic damage)
Nice post! I am a pretty close follower of the Thiel Cinematic Universe (ie his various interviews, essays, etc)
I think Thiel is also personally quite motivated (understandably) by wanting to avoid death. This obviously relates to a kind of accelerationist take on AI that sets him against EA, but again, there’s a deeper philosophical difference here. Classic Yudkowsky essays (and a memorable Bostrom short story, video adaptation here) share this strident anti-death, pro-medical-progress attitude (cryonics, etc), as do some philanthropists like Vitalik Buterin. But these days, you don’t hear so much about “FDA delenda est” or anti-aging research from effective altruism. Perhaps there are valid reasons for this (low tractability, perhaps). But some of the arguments given by EAs against aging’s importance are a little weak, IMO (more on this later) – in Thiel’s view, maybe suspiciously weak. This is a weird thing to say, but I think to Thiel, EA looks like a fundamentally statist / fascist ideology, insofar as it is seeking to place the state in a position of central importance, with human individuality / agency / consciousness pushed aside.
As for my personal take on Thiel’s views – I’m often disappointed at the sloppiness (blunt-ness? or low-decoupling-ness?) of his criticisms, which attack the EA for having a problematic “vibe” and political alignment, but without digging into any specific technical points of disagreement. But I do think some of his higher-level, vibe-based critiques have a point.
This was discussed last week but I looked at the comments and noticed someone in the comments getting slammed for… checks notes… noting that Eliezer wasn’t clear on what research paper he was actually responding to (multiple other comments are kind of confused, because they assume he means one paper then other comments correct them that he obviously meant another). The commenter of course edits to back-peddle.
One of the comments really annoyed me:
The “genetics is meaningless at the individual level” argument has always struck me as a bit of an ivory-tower oversimplification.
No, its pushing back at eugenicist with completely fallacious ideas. See for example Genesmith’s posts on Lesswrong. They are like concentrated Genetics Dunning-Kruger and the lesswrongers eat them up.
No one is promising perfect prediction.
Yes they are, see Kelsey Piper’s comments about superbabies, or Eliezer worldbuilding about dath Ilan’s eugenics, or Genesmith’s totally wacko ideas.
The numbers that get thrown about don’t mean what the people throwing them around think them to mean
That describes a common rationalist failure mode. They reach for a false sense of quantification by throwing lots of numbers at things, but the numbers are already approximations of much more nuanced, complex, and/or continuous things, so by overemphasizing the numbers, they actually get further from properly understanding. See for example… fixation on IQ; slapping probabilities everywhere; extrapolating trend lines (METR task length); and prediction markets.
that couple
I hate that I know what is being talked about the instant I see it.
Also, they’ve appeared on 3 separate top posts in the stubstack this week, so yeah another PR blitz. I find it kind of funny/stupid the news media can’t even bother to find a local eugenicist couple to talk to. I guess having a “story” served up to you is enticing enough to utterly fail to provide pushback or question if the story is even relevant to your audience in the first place.
They are going with the 50% success rate because the “time horizons” for something remotely reasonable like 99% or even just 95% are still so tiny they can’t extrapolate a trend out of it and it tears a massive hole in their whole AGI agents soon scenarios().
I would give it credit for being better than the absolutely worthless approach of “scoring well on a bunch of multiple choice question tests”. And it is possibly vaguely relevant for the pipe-dream end goal of outright replacing programmers. But overall, yeah, it is really arbitrary.
Also, given how programming is perceived as one of the more in-demand “potential” killer-apps for LLMs and how it is also one of the applications it is relatively easy to churn out and verify synthetic training data for (write really precise detailed test cases, then you can automatically verify attempted solutions and synthetic data), even if LLMs are genuinely improving at programming it likely doesn’t indicate general improvement in capabilities.
Eliezer’s response is especially stupid given that he has cited his fictional worldbuilding project like it is evidence (we have examples saved over on the reddit sneerclub).
Saw this posted to the Reddit Sneerclub, this essay has some excellent zingers and a good overall understanding of rationalists. A few highlights…
Rationalism is the notion that the universe is a collection of true facts, but since the human brain is an instrument for detecting lions in the undergrowth, almost everyone is helplessly confused about the world, and if you want to believe as many true things and disbelieve as many false things as possible—and of course you do—you must use various special techniques to discipline your brain into functioning more like a computer. (In practice, these techniques mostly consist of calling your prejudices ‘Bayesian priors,’ but that’s not important right now.)
We’re all very familiar with this phenoma, but this author has a pithy way of summarizing it.
The story is not a case study in how rationality will help you understand the world, it’s a case study in how rationality will give you power over other people. It might have been overtly signposted as fiction, with all the necessary content warnings in place. That doesn’t mean it’s not believed. Despite being genuinely horrible, this story does have one important use: it makes sense out of the rationalist fixation on the danger of a superhuman AI. According to HPMOR, raw intelligence gives you direct power over other people; a recursively self-improving artificial general intelligence is just our name for the theoretical point where infinite intelligence transforms into infinite power.
Yep, the author nails the warped view Rationalists have about intelligence.
We’re supposedly dealing with a group of idiosyncratic weirdos, all of them trying to independently reconstruct the entirety of human knowledge from scratch. Their politics run all the way from the furthest fringes of the far right to the furthest fringes of the liberal centre.
That is a concise summary of their warped Overton Window, yeah.
And some people are crediting Eliezer as if he predicted this devastating damage and not something completely different. Or they compare LLMs spewing shit to his scenarios of agentically and intellignetly dangerous and manipulative AGIs.
Should we give up on all altruist causes because the AGI God is nearly here? the answer may surprise you!
tldr; actually you shouldn’t give because the AGI God might not be quite omnipotent and thus would still benefit from your help and maybe there will be multiple Gods, some used for Good and some for Evil so your efforts are still needed. Shrimp are getting their eyeballs cut off right now!
So this blog post was framed positively towards LLM’s and is too generous in accepting many of the claims around them, but even so, the end conclusions are pretty harsh on practical LLM agents: https://utkarshkanwat.com/writing/betting-against-agents/
Basically, the author has tried extensively, in multiple projects, to make LLM agents work in various useful ways, but in practice:
The dirty secret of every production agent system is that the AI is doing maybe 30% of the work. The other 70% is tool engineering: designing feedback interfaces, managing context efficiently, handling partial failures, and building recovery mechanisms that the AI can actually understand and use.
The author strips down and simplifies and sanitizes everything going into the LLMs and then implements both automated checks and human confirmation on everything they put out. At that point it makes you question what value you are even getting out of the LLM. (The real answer, which the author only indirectly acknowledges, is attracting idiotic VC funding and upper management approval).
Even as critcal as they are, the author doesn’t acknowledge a lot of the bigger problems. The API cost is a major expense and design constraint on the LLM agents they have made, but the author doesn’t acknowledge the prices are likely to rise dramatically once VC subsidization runs out.
Is this “narrative” in the room with us right now?
I actually recall recently someone pro llm trying to push that sort of narrative (that it’s only already mentally ill people being pushed over the edge by chatGPT)…
Where did I see it… oh yes, lesswrong! https://www.lesswrong.com/posts/f86hgR5ShiEj4beyZ/on-chatgpt-psychosis-and-llm-sycophancy
This has all the hallmarks of a moral panic. ChatGPT has 122 million daily active users according to Demand Sage, that is something like a third the population of the United States. At that scale it’s pretty much inevitable that you’re going to get some real loonies on the platform. In fact at that scale it’s pretty much inevitable you’re going to get people whose first psychotic break lines up with when they started using ChatGPT. But even just stylistically it’s fairly obvious that journalists love this narrative. There’s nothing Western readers love more than a spooky story about technology gone awry or corrupting people, it reliably rakes in the clicks.
The call narrative is coming from inside the house forum. Actually, this is even more of a deflection, not even trying to claim they were already on the edge but that the number of delusional people is at the base rate (with no actual stats on rates of psychotic breaks, because on lesswrong vibes are good enough).
Some of the comments are, uh, really telling:
The main effects of the sort of “AI Safety/Alignment” movement Eliezer was crucial in popularizing have been OpenAI, which Eliezer says was catastrophic, and funding for “AI Safety/Alignment” professionals, whom Eliezer believes to predominantly be dishonest grifters. This doesn’t seem at all like what he or his sincere supporters thought they were trying to do.
The irony is completely lost on them.
I wasn’t sure what you meant here, where two guesses are “the models/appeal in Death with Dignity are basically accurate, but, should prompt a deeper 'what went wrong with LW or MIRI’s collective past thinking and decisionmaking?, '” and “the models/appeals in Death with Dignity are suspicious or wrong, and we should be halt-melting-catching-fire about the fact that Eliezer is saying them?”
The OP replies that they meant the former… the later is a better answer, Death with Dignity is kind of a big reveal of a lot of flaws with Eliezer and MIRI. To recap, Eliezer basically concluded that since he couldn’t solve AI alignment, no one could, and everyone is going to die. It is like a microcosm of Eliezer’s ego and approach to problem solving.
“Trigger the audience into figuring out what went wrong with MIRI’s collective past thinking and decision-making” would be a strange purpose from a post written by the founder of MIRI, its key decision-maker, and a long-time proponent of secrecy in how the organization should relate to outsiders (or even how members inside the organization should relate to other members of MIRI).
Yeah, no shit secrecy is bad for scientific inquiry and open and honest reflections on failings.
…You know, if I actually believed in the whole AGI doom scenario (and bought into Eliezer’s self-hype) I would be even more pissed at him and sneer even harder at him. He basically set himself up as a critical savior to mankind, one of the only people clear sighted enough to see the real dangers and most important question… and then he totally failed to deliver. Not only that he created the very hype that would trigger the creation of the unaligned AGI he promised to prevent!
Ha, even by the standards of SCP fanfiction, the slop Geoff Lewis got it to churn out was bad and silly.
He knows the connectionist have basically won (insofar as you can construe competing scientific theories and engineering paradigms as winning or losing… which is kind of a bad framing), so that is why he pushing the “neurosymbolic” angle so hard.
(And I do think Gary Marcus is right that the neurosymbolic approaches has been neglected by the big LLM companies because they are narrower and you can’t “guarantee” success just by dumping a lot of compute on them, you need actual domain expertise to do the symbolic half.)
I can imagine it clear… a chart showing minimum feature size decreasing over time (using cherry picked data points) with a dotted line projection of when 3d printers would get down nanotech scale. 3d printer related companies would warn of dangers of future nanotech and ask for legislation regulating it (with the language of the legislation completely failing to effect current 3d printing technology). Everyone would be buying 3d printers at home, and lots of shitty startups would be selling crappy 3d printed junk.
It is kind of sad. They are missing the ideological pieces that would let them carry out activism effectually so instead they’ve gotten used as a free source of crit-hype in the LLM bubble. …except not that sad because they would ignore real AI dangers in favor of their sci-fi scenarios, so I don’t feel too bad for them.