Random Thread

[malloc]

Having researched AI and the debates around it, I find the commentary of Ed Zitron perhaps the most vexing. Despite writing numerous polemics against AI over the years, he conspicuously avoids talking about its capabilities and focuses almost entirely on the expense of building and operating it. Imagine you were protesting nuclear proliferation and the threat of nuclear Armageddon and then someone shows up and complains in a British accent about the extraordinary cost of nukes and how they strain government budgets. He wouldn't necessarily be wrong but he's clearly putting the emphásis on the wrong sylláble.

[Raphael]

Having researched AI and the debates around it, I find the commentary of Ed Zitron perhaps the most vexing. Despite writing numerous polemics against AI over the years, he conspicuously avoids talking about its capabilities and focuses almost entirely on the expense of building and operating it. Imagine you were protesting nuclear proliferation and the threat of nuclear Armageddon and then someone shows up and complains in a British accent about the extraordinary cost of nukes and how they strain government budgets. He wouldn't necessarily be wrong but he's clearly putting the emphásis on the wrong sylláble.

It because he thinks the people talking about the amazing capabilities of "AI" are bullshitting people. He may be right on that, he may be wrong on that, but in any case it shouldn't be difficult to understand his position.

PS: In case you didn't notice it yet, that, rather than "AI is great and should be supported", is the actual position of most of the people against whom you've been arguing here.

[malloc]

It because he thinks the people talking about the amazing capabilities of "AI" are bullshitting people. He may be right on that, he may be wrong on that, but in any case it shouldn't be difficult to understand his position.

That position is clearly rubbish though. One could perhaps argue that when AI was just starting to invade the arts and flooding the internet with six-fingered hands. But then it got better and drawing and learned to write literature and compose music and mastered coding and now it's proving decades-old mathematical conjectures. The amazing capabilities of AI are now an indisputable fact and any analysis of AI that does not center them and their consequences for the rest of us makes no sense to me.

[zompist]

indisputable

That word, I don't think it means what you think it means.

[bradrn]

indisputable

That word, I don't think it means what you think it means.

I mean… I’m going to side with malloc on this one — it is a demonstrable fact that LLMs today have quite startling capabilities, most easily noticeable in the field of mathematics, but examples abound in other fields too. To me this makes it a uniquely dangerous technology, especially in its so-called ‘agentic’ uses (which strike me as being unethical to start with).

That does not, of course, mean that malloc’s persistent doomerism is rational or indeed apposite. I’m going to recommend this recent article from the field of cryptography as a nicely-balanced take — yes, LLMs are highly capable, but their achievements get hyped up even beyond that, and they’re not nearly superintelligent yet.

[zompist]

I mean… I’m going to side with malloc on this one — it is a demonstrable fact that LLMs today have quite startling capabilities, most easily noticeable in the field of mathematics, but examples abound in other fields too. To me this makes it a uniquely dangerous technology, especially in its so-called ‘agentic’ uses (which strike me as being unethical to start with).

That does not, of course, mean that malloc’s persistent doomerism is rational or indeed apposite. I’m going to recommend this recent article from the field of cryptography as a nicely-balanced take — yes, LLMs are highly capable, but their achievements get hyped up even beyond that, and they’re not nearly superintelligent yet.

A good quote from that article:

Which brings me to the real problem: just because a model spits out an apparent new result, this does not mean the result is real. Even if models are good at producing real results, they’re much better at producing results that look real but are misleading. This can be enormously frustrating, and often means that human attention is more necessary than ever.

Which is pretty much the whole problem with AI. It has not "mastered" literature or coding or mathematics as malloc claims. It produces outputs that look like what humans do. Sure, you can be startled by this. But relying on it is questionable at best. (And no, you can't just ask another AI to do the testing.)

As Cory Doctorow has pointed out, computers were traditionally used to boost human capabilities— what he calls being a centaur. The use of AI promotes instead being a "reverse centaur": delegate the interesting and creative work to AIs, while the human is reduced to checking over the results (no one knows how you train for this in an AI world) and taking the blame for the AI's failures. The problem, as ever, is not "OMIGOD AI", it's how corporations, under the shaky worldview of CEOs, will use it to worsen the working environment.

Note, the AI companies have a strong interest in overhyping the AIs' capabilities— that's how they bring in the investment they need to keep going, since they're massive unprofitable. That in itself is a good reason to distrust the hype. malloc is repeating techbro marketing as if it were "indisputable" fact.

In most every area, the AI impresses noobs and is seen through by the experts. Beginning programmers are dazzled by what AI lets them do; top-level programmers are dubious, and don't want to maintain AI's messy code. Artists are less impressed than people who can't draw a cartoon. Judges are not happy when lawyers submit brief with hallucinated citations.... somehow the lawyers involved didn't care.

As for agentic AI, I'm sure you're read about the very recent case where one of OpenAI's latest AIs mounted an attack on another AI company that wasn't noticed for four days. I'm an old curmudgeon who was taught not to run programs from sources you don't trust. But the latest thing is apparently to let any machine anywhere run a mishmash of every program ever written. That's a recipe for security disasters, and hey look, a security disaster happened. But caring about security is so last-century.

[bradrn]

Which is pretty much the whole problem with AI. It has not "mastered" literature or coding or mathematics as malloc claims. It produces outputs that look like what humans do. Sure, you can be startled by this. But relying on it is questionable at best. (And no, you can't just ask another AI to do the testing.)

I think that ‘outputs that look like what humans do’ is awfully ambiguous… it sounds pretty innocuous, but can encompass quite advanced things like ‘find a counterexample like a mathematician does’. Or, for that matter, ‘find an exploit like a hacker does’ (in the case of the HuggingFace incident you referred to). LLMs can do and have done some pretty impressive/startling/frightening stuff, and their unreliability doesn’t change that fact (though of course it does complicate the picture somewhat).

As Cory Doctorow has pointed out, computers were traditionally used to boost human capabilities— what he calls being a centaur. The use of AI promotes instead being a "reverse centaur": delegate the interesting and creative work to AIs, while the human is reduced to checking over the results (no one knows how you train for this in an AI world) and taking the blame for the AI's failures. The problem, as ever, is not "OMIGOD AI", it's how corporations, under the shaky worldview of CEOs, will use it to worsen the working environment.

Agreed — though personally I think that LLMs would be an intrinsically risky technology even without big corporations making things worse.

As for agentic AI, I'm sure you're read about the very recent case where one of OpenAI's latest AIs mounted an attack on another AI company that wasn't noticed for four days. I'm an old curmudgeon who was taught not to run programs from sources you don't trust. But the latest thing is apparently to let any machine anywhere run a mishmash of every program ever written. That's a recipe for security disasters, and hey look, a security disaster happened. But caring about security is so last-century.

Indeed. The fact that anyone at all thinks this is a good idea strikes me as quite, quite mad.

[malloc]

Just as I predicted and many others have denied would happen, AI has continued to advance and now seems poised to match the human mind in capability. Zompist called my predictions an induction fallacy and compared it to imagining that modes of transportation would continue to advance until we are all flying around in hypersonic craft. Yet his implied counter-prediction that AI advancement would stall has conspicuously failed thus far. We must now ask who will control the reins of civilization in the future: humans or machines. Will humans retain their role as the dreamers, thinkers, and administrators of civilization or will they allow tech corporations to seize that role from them and give it to machines?

[zompist]

Just as I predicted and many others have denied would happen, AI has continued to advance and now seems poised to match the human mind in capability. Zompist called my predictions an induction fallacy and compared it to imagining that modes of transportation would continue to advance until we are all flying around in hypersonic craft. Yet his implied counter-prediction that AI advancement would stall has conspicuously failed thus far.

You dirty liar. "Implied prediction" means you are making up shit. Don't even pretend to be reading or quoting me when you're stroking your ego.

You need to seriously up your social skills, or your reading skills, or you research skills, before pretending that you know better than everyone else..

We must now ask who will control the reins of civilization in the future: humans or machines.

Ironically, you'd have done better to stick with Marxism rather than doomerism. You parrot techbro talking points, especially the techbro fear/boast that AI is as good as humans. Why are you so enamored of fascist ideas and fascist actors?

[malloc]

You dirty liar. "Implied prediction" means you are making up shit. Don't even pretend to be reading or quoting me when you're stroking your ego.

My apologies. I was not trying to put words in your mouth, only making my best guess about your own predictions. It seems obvious that the opposite of AI improving indefinitely is AI hitting a wall in improvements.

Ironically, you'd have done better to stick with Marxism rather than doomerism. You parrot techbro talking points, especially the techbro fear/boast that AI is as good as humans. Why are you so enamored of fascist ideas and fascist actors?

Because the evidence clearly points in that direction. Time and time again, artificial intelligence has mastered something once thought exclusive to humans, as my meme with the grim reaper at the door illustrates. I hardly support the goals of fascism or the tech industry, but neither can I ignore their obvious victories. If we want any chance of defeating them, we must first acknowledge their strengths so we can counter them effectively. WWII was not won by dismissing the Nazis as hapless losers when they were steamrolling through Poland and France after all.

[zompist]

You dirty liar. "Implied prediction" means you are making up shit. Don't even pretend to be reading or quoting me when you're stroking your ego.

My apologies. I was not trying to put words in your mouth, only making my best guess about your own predictions. It seems obvious that the opposite of AI improving indefinitely is AI hitting a wall in improvements.

The opposite of "this is happening, so it will happen forever" is "This may not happen forever."

Because the evidence clearly points in that direction. Time and time again, artificial intelligence has

I've already pointed out that AI has not in fact done what you think it does. So this isn't even "this is happening", it's "although this is not happening, AI marketers say it is and I really want to believe them".

But the proper reaction to it being pointed out that induction does not always work is not to simply repeat the fallacy. There is no "evidence" that induction always works.

We've been over the history of AI, which is also not what you think it is; go read the earlier monster thread.

[malloc]

The opposite of "this is happening, so it will happen forever" is "This may not happen forever."

So you're agnostic on whether AI will continue to improve indefinitely. You don't reject the possibility that it will continue to improve indefinitely, but rather feel that I'm too confident that it will.

I've already pointed out that AI has not in fact done what you think it does. So this isn't even "this is happening", it's "although this is not happening, AI marketers say it is and I really want to believe them".

Except that it clearly is happening. We now have AIs that can prove conjectures that have stumped mathematicians for decades. That comes on the heels of AIs capable of imitating human-made drawings, literature, and music, writing source code, and so forth. Do you dispute any of those achievements?

But the proper reaction to it being pointed out that induction does not always work is not to simply repeat the fallacy. There is no "evidence" that induction always works.

But you can hardly deny that the trend motivating my induction exists. Climate scientists model global warming based on current trends and nobody reasonable accused them of induction. What makes my predictions on AI so different?

[zompist]

But you can hardly deny that the trend motivating my induction exists. Climate scientists model global warming based on current trends and nobody reasonable accused them of induction. What makes my predictions on AI so different?

False analogy. With climate change, the idea is that if you pour CO₂ into the atmosphere long enough, temperatures will rise. For decades people did pour CO₂ into the atmosphere and it looked like they'd keep increasing the rate. (Some countries are now decreasing the rate, so the doomiest scenarios are now unlikely.)

There's no such simple feedback loop with AI.

Look at this page. Look at it long ago to absorb its main lesson. (Is AI profitable? No.)

Since you like inductive arguments, note that AI has never been profitable.

[malloc]

False analogy. With climate change, the idea is that if you pour CO2 into the atmosphere long enough, temperatures will rise. For decades people did pour CO2 into the atmosphere and it looked like they'd keep increasing the rate. (Some countries are now decreasing the rate, so the doomiest scenarios are now unlikely.)

There's no such simple feedback loop with AI.

Perhaps not, but there have been consistent advances in computing power over the past eighty years and AI has grown explosively in the past four years alone. Sure that progress could stall and AI might never advance beyond its savant skills in specific subjects, but it seems incredibly foolish to base AI policy on that assumption. We need plans for the current trajectory, not just hopes that everything will work out in our favor.

[bradrn]

We need plans for the current trajectory, not just hopes that everything will work out in our favor.

Correct, and I don’t see anyone disagreeing with that…

[malloc]

We need plans for the current trajectory, not just hopes that everything will work out in our favor.

Correct, and I don’t see anyone disagreeing with that…

Zompist effectively does disagree, though, given that he doubts the current trajectory will hold much longer and considers that reason enough not to worry about AI getting too powerful.

[bradrn]

We need plans for the current trajectory, not just hopes that everything will work out in our favor.

Correct, and I don’t see anyone disagreeing with that…

Zompist effectively does disagree, though, given that he doubts the current trajectory will hold much longer and considers that reason enough not to worry about AI getting too powerful.

Where did he say that he believes that?

[malloc]

Zompist effectively does disagree, though, given that he doubts the current trajectory will hold much longer and considers that reason enough not to worry about AI getting too powerful.

Where did he say that he believes that?

He has consistently dismissed my concerns about AI continuing to advance as the fallacy of induction. He once compared them to expecting modes of transportation to continually improve until everyone has hypersonic vehicles. While he does not necessarily rule out the possibility that AI will continue to improve, he evidently considers it unlikely enough that it doesn't make sense to develop active policies against it.

Based on his past comments, it sounds like he has lived through multiple waves of AI enthusiasm that all ended in failure, leaving him with inherent skepticism toward the field. Fair enough, but that doesn't change the fact that current AIs are clearly working in ways that previous ones could not.

[bradrn]

Zompist effectively does disagree, though, given that he doubts the current trajectory will hold much longer and considers that reason enough not to worry about AI getting too powerful.

Where did he say that he believes that?

He has consistently dismissed my concerns about AI continuing to advance as the fallacy of induction. He once compared them to expecting modes of transportation to continually improve until everyone has hypersonic vehicles. While he does not necessarily rule out the possibility that AI will continue to improve, he evidently considers it unlikely enough that it doesn't make sense to develop active policies against it.

Based on his past comments, it sounds like he has lived through multiple waves of AI enthusiasm that all ended in failure, leaving him with inherent skepticism toward the field. Fair enough, but that doesn't change the fact that current AIs are clearly working in ways that previous ones could not.

You didn’t answer my question — this is all supposition. If you’re making statements about what zompist believes you should back them up with specific things that he’s written saying so.

[Raphael]

Unrelated to the current discussion, I would like you to take a moment to admire the perfection of this social media comment on a local news story. From Lindsay Beyerstein on Bluesky:

This is the most ABQ story ever. The cocaine dealing was a front for the snake-hoarding.

https://bsky.app/profile/beyerstein.bsk ... uppgpn5k2r