The Sequences: What’s good isn’t original and what’s original isn’t good… but at least they have some original thought in them!

Yeah, definitely. Even if you are heavily modifying an idea or synthesizing multiple ideas, not citing your sources is still major academic misconduct.

I bet Elon could have kept the plates spinning for decades longer if he was happy with just Tesla and SpaceX, but he had to keep doubling down on grifting investors so maybe, finally, we will see it all come crashing down for him. (What happens once his assets no longer cover his debts? Do they actually force him to liquidate?)

I’ll go one step wider. It is not just Sam Altman or his social circle, but all the people most enriched by capitalism (and thus most able to exploit and benefit from a technology that shifts power away from labor and to capital) that I don’t want to see further empowered.

I believe the main ingredient is Lean, which is a formal language resembling a programming language. Math proofs written in Lean can be verified deterministically with a computer, which really helps mitigate the hallucination problems of LLMs.

100% this. Also, looking back at an earlier example that was actually written up in more detail, AlphaGeometry 1 got 28/30 problems, but entirely stripping out the LLM from the system, the symbolic logic proportion alone could get 14/30, and replacing the LLM with different heuristic methods could get 18/30 and 21/30 (for different methods).

Even if math research works out perfectly well (which is a still big if), it’s not going to pay the bills. They would need to find a use case in the real world, where hallucinations can cause serious damage and cannot be formally prevented. And they have certainly tried. Math will not change the fact that all of this will collapse.

The boosters and LLM companies still believe LLMs get their current level of performance by generalizing and not just memorizing facts (and maybe a wide shallow pool of weak heuristics). So they are hoping by pushing the LLM performance up in some narrow domain they can churn out synthetic data for, they will see some large general improvements in LLM performance.

Nick Bostrom Took Many of His “Best” Ideas from Other People

My first reaction was that many thinkers and academics synthesize ideas from older sources and that this doesn’t sound that bad (well except for some of the ideas being eugenics or AI doomerism)… some of the examples Emile points out look like just straight up plagiarism.

that would be a signal to drop AI and nuke data centers from orbit

On Lesswrong I’ve seen some posts contemplating why this “warning shot” isn’t making any political waves. Also, in response to the fact the OpenAI has released basically no details, I read one poster realizing that actually they need external governance and regulation on the AI companies.

I had two main guesses even before reading into the details…

The obvious, they set this up for the doom crit-hype value. That is how basically every one of these stories like this to come out of Anthropic turn out to be once you read the details of the setup. The lack of details is to keep the mystique and hype up.

The other guess… their coding agent actually did manage to vibe code its way out of its ‘sandbox’ and past Huggingface’s security, but the security on both ends (OpenAI’s"sandbox" and huggingface’s code) was itself laughably bad vibe coded garbage. The lack of details is to obfuscate just how garbage they are. I think this guess is pretty well supported by the linked Mastadon posts (although I still wouldn’t discount my first guess until more details are pried out). My “favorite” quote:

“prepend some text with processing instructions and an SGML tag, and then hand that blob to an llm.”

I fucking hate this timeline.

and they turned it into a mutual marketing stunt

Their marketing actually seem to be trying to take different spins on it…

HuggingFace stressed that it worked out the attack was going on using an open weight model it hosted itself — because the guard rails on the commercial model HuggingFace tried wouldn’t let them do security work. So you should use open weight models from HuggingFace.

Yeah, its an opposite spin than Anthropic and OpenAI have been trying lately, where they want to shut down all open weights model because something something China something something too dangerous something something regulate our competition out of existence. (It is funny, some of the boosters on hackernews and /r/singularity are actually acknowledging without an artificial moat to help boost OpenAI’s and Anthropic’s capex for bigger and bigger models will stop. Even though they still deny Ed Zitron’s calculations of the financials.)

Anyway, I guess even if they have some very opposed strategic directions, huggingface and OpenAI are aligned on maximizing the hype.

You know, given automated proof checkers, I was naively assuming mathematics was one field that gen-AI would have a hard time screwing up. Even programming is too difficult to write thorough testing for. But a proof (or counter example to a conjecture) seems like it would have to be solid if it passes lean or whatever system for validating it.

But the threats #3 and #5 the declaration lists make me consider the bigger picture. Academic fields without clear capitalist payouts are already underfunded and under respected. Pure mathematics could, at least up until now, draw on the respect STEM gets, but with math proofs getting used as fuel for the LLM hype machine, there are a variety of unpleasant ways things could twist.

Threat #1 makes me wonder… if LLMs+formal verification systems manage to pluck lots of low hanging fruit, and we are left with harder stuff that not enough literature exists as training data for LLMs, it seems like the entire educational pipeline for producing mathematicians could end up screwed up.

he chickened out at the last minute and self-censored it

You know, on net, the internet would be a better place if more people did this, but I wonder why the author even bothered posting anything at all?

2017, three years post gamergate

You know, I was wondering if the author elided the gap between early 2000s and 2017 because of the cringey and sometimes outright awful stuff that came out of the rationalists… but you’ve reminded me the skeptic movement also had some nasty bits both within it and as offshoots.

That was kind of a nothing post? I think the author is misrepresenting the growth (or at least failing to connect the most important dots) of the lesswrong rationalist by portraying them as a natural evolution of the skeptics movement and not the deliberate cultivation of an audience by Eliezer looking for people to spread his ideology to and by Thiel looking to influence farm and create a cult incubator. And then the post is titled “my falling out with the rationalist”, and they write they "wanted to offer a more personal perspective " but they don’t actually discuss that much of the personal angle either…

Even /r/singularity, which has a high percentage of boosters and often buys OpenAI/Anthropic talking points uncritically (or with only a few un-upvoted naysayers) eviscerated his comments as the shallow attempt to beg for regulations on China to protect OpenAI that they are. Also, China’s consistent open sourcing and lower relative cost per performance (well insofar as you believe the benchmarks and such indicate performance) has the /r/singularity audience questioning the “China bad” narrative US media tries to sell.

That is an unfair simplification! Some of this audience is there for the closeted (well not so closeted over the past few years) race science and are annoyed by his AI doomer posting!

I read through it myself, but I was too busy laughing at the absurd parts (secret Chinese data centers inside mountains) and suspending disbelief at the overall premise to track all the individually nonsensical bits.

That’s not how international cooperation works.

They also completely ignore the fact that the US has burned away half a century of soft power and (relative) credibility. A lot of Scott’s (and the other AI:2027/2040 authors) readers are Trumpers or at least so libertarian-brain-rotted they view Trump as not that bad a tradeoff, so it’s pretty obvious to me why they would avoid dwelling on this little problem for their scenario. (I think this is also lowkey why the put the scenario off until 2029, even their insane imaginations can’t imagine international cooperation under Trump, but they don’t want to draw attention to this.)

That’s not how economics works.

Yeah, everyone is somehow superrich because carefully limited AIs are producing so much value? Even with the lesswrong favorite trope of magic nanotech (which I was disappointed to see AI: 2040 fail to include), equitable distribution is an extreme problem.

nothing that AI is revolutionizing is really impacting the kinds of stuff that people would be buying save that there’s more of it and that people are no longer being paid to produce it

I actually know the rationalist canon answer to this from other crap Scott has posted! He expects humanoid robots to takeoff exponentially basically, with an initial bootstrapping of buying some car companies’ manufacturing lines solely off the insane valuations of the robot companies.

and of a kind that isn’t even interesting

It is also boring because there is not enough magic-scifi super tech, or competing rival God-AGIs playing 40 dimensional chess against each other as they jockey for power and control over the humans, but your complaints are also true!

utopia they outline is not meaningfully connected to the policies and values they advocate for

Their doom/salvation scenario really needs centralized planning acting for the common good to land on the salvation side, and they are deeply allergic to the idea, hence all the complicated neoliberal workarounds.

apparently he’s having a bit of a hard time selling his crowd on the really obvious totalitarian implications of instituting a global surveillance system to track and remotely disable chipsets

I thought I would agree with his libertarian audience about this much at least, but reading their comments… some of the angles they take and line of argument they make actually makes me more in favor of totalitarian dystopian regulation on computer chips.

Awww, but I like the pure science fiction.

Even as sci-fi, Ai: 2040 is lackluster! Where is the classic lesswrong trope of drexler-style nanotech? Where are the cybernetic augmentations!?

The second is, things happen because of ideas and agreements rather than actions. You can speak into existence billion year plans for what happens to whole galaxies, and it happens because that is what you said.

It is the same flaw that underlies the cryptobros delusions about blockchains and NFTs. You can make some cryptography ledger, it doesn’t mean anyone actually follows it or that the ledger tracks reality.

Oh hey, a full top level post complaining AI-2027 wasn’t sci-fi enough (they literally use the words “I don’t think it is science-fictional enough”)! https://www.lesswrong.com/posts/Lc8v92pGPtjF3gsCv/some-quick-thoughts-on-ai-2027

The impression I’ve gotten it that it is relatively less toothless than some of the other rationalist spin-off movements aimed at preventing doom. And it is willing to deign to try to coalition build with normies that want to stop AI for normie reasons (except apparently they’ve made even that a bad thing, from the tweet it is willing to unapologetically coalition build with fascists!) But yes, it still has a lot of rationalist hangups and quirks getting in the way of actually acting effectively.

Lesswrong has some funny objections to AI: 2040. They buy the overall premise, sure, but even that much still leaves lots of gaps in the “scenario”. And they have even more galaxy brain takes… My favorite complaint: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=54dZes9uNqDdbNGBu

tldr; AI: 2040 is leaving out all the other sci-fi singularity tech just around the corner like genetic augmentations and cybernetic brain implants!