My colleagues in math are now frightened about the (very expensive) mathematical theorem proving ability of these AIs, and many of them really do think that if they can do math, they can do all cognitive tasks. Running a store like this should be so easy! Every single conversation about AI with them has become more frustrating. They are so confused when I still say that the AI companies will die a painful death. When I give my usual points about their expense and their failures in other domains, I am given the usual spiel of “it’ll get better in other areas” and “it’ll get cheaper”.

Unlike them, I have actually been paying attention to this stuff from the beginning. What they think is going on is AI solving math first and shortly getting around to all the other stuff, but what I’ve seen is that AI labs had already tried all the other stuff first and only managed to win the booby prize of theorem proving, which doesn’t pay the bills. And what’s the point of spending thousands or millions to output random blobs of Lean that technically compile if there is no one around to bother making sense of them?

One example I gave is when Anthropic vibe coded an entire C compiler from scratch back in February, which turned out to be a pile of shit. I’ve said that if AI had made similarly rapid progress on software engineering, we would have seen Anthropic continue to put out these demonstrations, and they would have become truly high quality. They would release a compiler more efficient than gcc one week, and a browser better than Chrome the next. (OpenAI’s actual attempt at a browser didn’t go so well.) And if they could do this, they would actually have a shot of making money!

If they could do this, they would have already. The theorem proving stuff actually works (for certain things, in certain ways, at enormous expense), and look at how OpenAI and Anthropic do not hesitate to snipe mathematicians for results rather than being content as tool vendors. But lately I haven’t heard of any software demonstrations. Silence is much louder than noise. More Millennium prize problems bashed with tens of millions in compute costs are not going to change my mind very much.

The counterargument I got was that AI can already one-shot most programming tasks and I shouldn’t be cherry-picking the failures. I am far too tired to argue at this point.

Despite all the doomer propaganda that has been swirling around for many years, it’s still easier to imagine the end of the world than the end of capitalism.

Signed.

Terry Tao talks about how he used to try to cooperate with the AI industry to achieve a positive outcome, but now he finally sees their true colors. Link

During this event, OpenAI requested an interview concerning my vision of the future of AI and mathematics. I accepted, and spoke with them for perhaps an hour. I had done similar interviews in various venues, and I assumed that, as with these other cases, they would eventually post the entire interview online, which talked about both the possibilities and risks of AI much as I have done in these other interviews. As it turned out, they only used a few snippets of that interview for that infamous advertisement instead. In retrospect, I should have pushed back harder on their decision; but I decided at the time that even a selective release of my commentary would help raise awareness of the potential for AI, and in particular on the possibility of the “best of both worlds”.

Since then, the situation has deterioriated markedly. Many of the people in the industry that shared my views have left or become sidelined, with most major tech companies now increasingly focused on the race to develop extremely powerful, autonomous AI technologies regardless of their actual value to society. The current drama surrounding the Navier-Stokes global regularity problem is the most dramatic and visible instance of this, but there have been multiple other such examples, and much of my commentary in the last few months has been aimed that the increasingly severe divergence between the current objectives of the AI industry, and of mathematics in general.

Much respect to artists for seeing all this coming from the very beginning, and holding the line.

One detail that seems to have not spread around: the announcement was so rushed that when the 165 page output was released, it only included 16 citations, and those citations did not even include the work of Diego Córdoba and Luis Martínez-Zoroa, who introduced the overarching framework for attacking the problem (that Buckmaster-Alpöge and OpenAI used). After the announcement and after these redditors pointed it out, they threw in a few more citations to include them, but come on.

Of course there are people trying to find a silver lining to this by conjuring up the hypothetical scenario where a student only uses the AI to aid in learning the material instead of just doing all the work.

First, any convenience in learning the material just reduces your ability to learn it. The friction involved with learning may seem like an inconvenience to be smoothed away, but it turns out that the friction is how learning happens. It’s called engaging with the material. This has been the case with previous technologies: handwriting is better for retaining memory than typing (https://pmc.ncbi.nlm.nih.gov/articles/PMC11943480/), although it seems like AI is on an entire new level. (I guess there is some commentary about the sadly common worldview that life is about avoiding inconveniences. I feel like this mindset draws a lot of people to AI.)

Second, there is a very thin line between “helping” you learn the material and just doing the work for you. The temptation to cut corners is always there, and when you have the Corner Cutting Machine at your disposal, you are kidding yourself if you think you will have perfect discipline. Tools influence behavior.

Glad to see that OpenAI has not changed in their scummy ways. Despite all that has changed in the meantime, they have kept their time-honored tradition of passing off other people’s work as their own.

One of OpenAI’s math announcements a month ago claimed that their results cost only $2000 worth of tokens, which frustrated me because they were likely sweeping away many inconvenient details and almost certainly misrepresenting their true costs. But people took this as a gotcha. This is the same bullshit as the water usage arguments. We are literally seeing city council members signing motherfucking NDAs about this, and you think that water usage numbers provided by the tech companies themselves are going to sway me?

I am also questioning OpenAI’s strategy of strip-mining math for PR, since it seems like advances in math do not actually register that well in the public. From what I remember, the Hugging Face incident got a lot more press than any of the math results.

Is this the post you’re referring to? https://davidbessis.substack.com/p/the-fall-of-the-theorem-economy

https://www.math.brown.edu/reschwar/Stories/AI.pdf

Phil Aroneanu, Irreplaceable’s executive director and a co-founder of the climate organization 350.org, told me he believes that climate advocates initially floundered because they acted as policy wonks. They thought that making evidence-based arguments about the dangers of melting ice sheets and rising sea levels to receptive congresspeople would be enough to pass nationwide climate legislation. Today, Aroneanu said, movements are built not necessarily on what people think, but instead on what they feel. For climate, that meant fossil-fuel-divestment campaigns, protesting oil pipelines, and the school strikes led by Greta Thunberg. People are already “feeling the squeeze,” Aroneanu told me. “And we should be pointing that anxiety and that anger in the right direction.”

What is with this attitude of dismissing regular people’s opinions as “feelings” in comparison to their own rationality? This is like those people who would really like to be anti-AI but find it more important to nitpick the water usage numbers and point out how agriculture uses so much more water anyway, in order to form a “better” opposition.

Some systems like SynthID (for Google’s AI) get around this problem. In fact you don’t need to know the LLM’s internal state, and defeating it would likely involve breaking up most blocks of 3 words. The oversimplified explanation is that it introduces a function g that gives a score to each word, with the score being (pseudo)randomly determined by your secret key. For each next word the LLM generates, the LLM produces a small list of candidate next words, and the one with the highest score according to g is selected. You should expect that the LLM will generally pick words with a high score, but the score itself is independent of the LLM. To detect a watermark, you need to know g and the secret key, and you check if the average score is much higher than expected from normal text.

Now, one question is, will this bias to the LLM to favor certain words? The solution is that for each next word, you append the last 3 words (nothing special about 3, just a small number) to the secret key for g, and this repeatedly scrambles which words have a high score. To defeat the watermark, you would need to break up most blocks of 3 words. I’m sure there are deeper issues with this, but I have not studied the topic that much.

From a technical perspective, I don’t really know how to let it track specific users. The goal is that if some AI vendor provides their key, it is easy for them to prove that their AI produced the text. So I guess if a user wanted to show that they generated the text, they could prove it. But usually AI users don’t want people knowing that they used the slop cannon. In any case, Anthropic’s own announcement states that their watermark does not contain identifying information for users.

In any case, some proud AI users are now frightened that there might be a way for people to identify that they use such a wonderful tool. How will I ever deliver low quality slop for my contracts now?

He said he uses AI for code reviews and translations, and worried an AI label on code shipped to clients could raise questions about authorship or trigger contract penalties.

Having code attributed to Claude “definitely would be not desirable,” he said, as he would not want professionally shipped code to carry an invisible marker that could raise questions about authorship, compliance, or client policies.

My pet theory is that their reading comprehension has degraded, so this is all they can handle now.

Anthropic is now watermarking the outputs of its AI. For once this is some AI news that doesn’t completely piss me off, and it’s amusing to see all the uninformed boosters get in a tizzy about this.

I actually understand at a reasonable level how this watermarking works. A year ago, I watched Scott Aaronson give a talk about it, and from what I know he was somewhat involved in developing the theory behind it while working for OpenAI. But at the time my thought was, “He is naive if he thinks these companies would ever implement this out of the goodness of their hearts.” And I was right; Anthropic is only doing watermarking now thanks to the EU AI Act, even though the theory has long been developed.

Watermarking doesn’t mean adding an extra watermark that can be easily removed. It instead directly affects the output of the chatbot itself. Fundamentally, an LLM is still a most-likely-next-word-predictor. More precisely, an LLM produces a probability distribution of what the next word can be. For example, “my pet is a …” could give a distribution of 60% dog, 30% cat, and 10% axolotl. Normally, an LLM would randomly choose the next word based on this distribution, and this is one reason why LLMs are nondeterministic (there’s another parameter called “temperature” that affects this, but no need to get into that).

With watermarking, instead of a truly random choice, the randomness instead comes from a cryptographic pseudorandom generator seeded with a secret key from the AI company. If you don’t know the secret key, then you can’t really tell that watermarking was used. But if you do know the secret key, then the idea is you can tell when the text was generated by the LLM because you know exactly what word should be next. It would be a freak coincidence if some non-AI text just happened to choose the correct next word every time. Thus, you can provide a service to tell if some text was generated by the LLM. (This technically makes the LLM “deterministic”, in a completely useless sense.)

Now, I think this is a step in the right direction, but it has its limits. The biggest problem is that you don’t want people to just move to a different LLM without watermarking, and that’s exhibit #832593 why government regulation is important. Another issue is that sometimes there is very little randomness in what the next word should be (“The first president of the USA is George …”). Finally, watermarking can be defeated by editing the output, although you would have to break up most of the blocks of consecutive words. I have a feeling most AI users are not the type to put in extra effort after copy-pasting the output directly from the chat window.

I suppose it will discourage some of the “use cases” of LLMs, such as drowning the world with spam Slopstack essays. Ah, who am I kidding? Everyone could already tell it’s AI generated, they don’t care!

Mathematicians are not entirely happy about OpenAI’s ten mathematical results. As it turns out, the proofs did not cite many of the existing techniques in the literature. The plagiarism bot is still a plagiarism bot. https://www.scientificamerican.com/article/openais-latest-math-breakthroughs-commit-research-misconduct-experts-say/

“They are running roughshod over the work of others who came before them in a deliberate way,” says Steven Miller, a mathematician at Yeshiva University, who argues that OpenAI has effectively plagiarized his own research. “It seems completely systematic to me, and it points to research misconduct.”

The LLM-generated proof hinges on a particular mathematical argument that it presented as its own but that actually first appeared in a 2016 paper by Miller and a collaborator.

The discovery stunned Francesco Fournier-Facio, a mathematician at the University of Cambridge, who studies group theory—at least until he “engaged with this breakthrough as I would if a human had written it,” he says. The result, he and some of his colleagues found, wasn’t as novel as it first appeared. Like a number of recent AI breakthroughs, it pasted together ideas from the mathematical literature to build a new theorem. Once again, the LLM’s trick is its superhuman patience for assembling puzzle pieces, not the ability to make some profound intellectual leap.

In particular, Astra’s key mathematical step combined ideas first found in two papers from 2016 and 2019. Andreas Thom, a mathematician at the Dresden University of Technology, who co-authored the 2019 paper, summarized the result on MathOverflow.com, calling it “creative and at the same time elementary.”

He is spending the equivalent of $1 million a year on tokens just to develop this game with 6 concurrent players on a good day. Of course, he isn’t actually paying that much. Instead, he just has a dozen different $200/mo Max subscriptions to get the same amount of tokens for $33k a year.

AI economics, baby!

Clammy Sammy acknowledges Ed Zitron for the first time! https://xcancel.com/sama/status/2084663673570971990

His response is the Live Laugh Love of 2015 Silicon Valley platitudes:

i would rather be an optimist and work hard than a pessimist posting about why things won’t work.

it’s much more difficult and the most likely path is failure, but society fails if people don’t try.

no amount of “it will never work” essays will drive society forward.

Of course, building data centers that use untold amounts of rare earth metals, electricity, and water while spewing out tons of pollution and scraping every last bit of human content in order to remove people from their jobs by doing them badly is what drives society forward. Profits fail if oligarchs don’t try. Actually, the AI companies don’t even need profits.

The OpenAI hacking incident has put everyone on edge. Anthropic, not to be outdone, decides to publicly announce that they have committed not one, not two, but three crimes! Come on! I was told this wasn’t a marketing campaign!

Before the days of AI 2027, he became famous in 2024 for posting one of the original pieces of writing in the line-go-up genre: Situational Awareness. It seems that like any good grifter, he used this opportunity to make money (in this case by starting a hedge fund).

There is something about this picture, a warehouse full of books to be destroyed by Anthropic for training data, that just makes me sick. Knowing that they are destroying books in the abstract is bad enough, but seeing the sheer scale of it makes me viscerally angry.

Karen Hao called her book Empire of AI for a very good reason. The empire has all these noble narratives of bringing civilization and progress to the colonies, but in reality the empire only sees resources to be extracted, repackaged, and sold back to their subjects at a fee. Kudos to those who saw this immediately and fought against it. Shame on those who believed in the narrative of progress, to the point where many of them are collaborators with the empire.

Whenever a smarmy fellow on Linkedin posts about how AI is making so much progress in coding or math, I want them to see this picture. I want them to see the videos of families in tears after a data center takes away their home with eminent domain, and other families who have to deal with the resulting pollution. Go beyond the bloodless math of tokens and instead see for yourself what this all truly costs. I hope it was worth it.