zompist's Essay on "Advanced Civs"

[Raphael]

Having them in our bodies...

Well, we've already gotten aclimatized to having pacemakers on and in us, as well as mechanical devices serving as hearts (and sometimes just assisting our hearts, such as artificial valves)...and prosthetic devices, some of which take nervous impulses from the spine, joints, and now the brain, and translate it into actions undertaken by the prosthetic itself.

Why would everyone want to stop there?

This reminds me of someone's remark - I think it was Justin Rye, but I'm not sure - that the debate over whether to technologically augment human bodies was decided when people started wearing eyeglasses. However, do all of the devices you mention contain microelectronics? And if they do, are they connected to the Internet?

[zompist]

Having them in our bodies...

Well, we've already gotten aclimatized to having pacemakers on and in us, as well as mechanical devices serving as hearts (and sometimes just assisting our hearts, such as artificial valves)...and prosthetic devices, some of which take nervous impulses from the spine, joints, and now the brain, and translate it into actions undertaken by the prosthetic itself.

Why would everyone want to stop there?

This reminds me of someone's remark - I think it was Justin Rye, but I'm not sure - that the debate over whether to technologically augment human bodies was decided when people started wearing eyeglasses. However, do all of the devices you mention contain microelectronics? And if they do, are they connected to the Internet?

That was me, I'm afraid. (I've said it elsewhere, but that's the ref I can find.)

As that review indicates, I'm not attracted by the notion of being a metal monster myself, and I sympathize with the worry about machine vulnerabilities. But, well, we are already augmented beings— most people have cell phones with them at all times, and though I don't, I'm usually a few feet away from a computer.

But if the alternative is AIs running our lives, or perhaps entirely replacing us primates... I'd rather appropriate their superhuman abilities rather than be ruled by them.

[Travis B.]

The problem with augmentation is when the makers of augmentations decide to abandon them. I recently remember reading about someone who had gotten a $100K exoskeleton suit after becoming paralyzed from the waist down in a horse-riding accident... only to have their exoskeleton suit rendered useless when a wire in the wristwatch controlling it became unsoldered and the maker of the suit refused to fix it because "we don't support devices older than five years". Ultimately someone else was able to repair it, but the lack of a right to repair is a massive problem, especially in the medical device industry. If I had the choice I certainly wouldn't want to rely in such a fashion on a device that the manufacturer would obsolete in a mere five years...

[keenir]

Well, we've already gotten aclimatized to having pacemakers on and in us, as well as mechanical devices serving as hearts (and sometimes just assisting our hearts, such as artificial valves)...and prosthetic devices, some of which take nervous impulses from the spine, joints, and now the brain, and translate it into actions undertaken by the prosthetic itself.

Why would everyone want to stop there?

This reminds me of someone's remark - I think it was Justin Rye, but I'm not sure - that the debate over whether to technologically augment human bodies was decided when people started wearing eyeglasses. However, do all of the devices you mention contain microelectronics?

A lot of modern prosthetics do. Definately the ones that rely on electrical impulses from nerves.

And if they do, are they connected to the Internet?

Offhand, I don't think so...but it probably wouldn't be difficult to accomplish, given that people are already at work to send messages and directions from the skull to the - for example - prosthetic hands or legs, skipping past most of the body.

But if the alternative is AIs running our lives, or perhaps entirely replacing us primates... I'd rather appropriate their superhuman abilities rather than be ruled by them.

If thats the options, then I'd rather have HAL9000 operating my prosthetics, rather than ChatGTP or some hacker.

But its more likely that prosthetics will only get as bright as one of those ant-robots that {in teams} roll balls around, or the Google Search Engine at most.

[keenir]

The problem with augmentation is when the makers of augmentations decide to abandon them.

very true.

. Ultimately someone else was able to repair it, but the lack of a right to repair is a massive problem, especially in the medical device industry. If I had the choice I certainly wouldn't want to rely in such a fashion on a device that the manufacturer would obsolete in a mere five years...

that is a danger with riding too close to the edge of technological ability...rarer to get something like that with longstanding tech like pacemakers. (rarer, not impossible)

[Raphael]

The way this debate here is currently going makes me think of a paragraph from my attempted novel:

Now, bots and security had always been a tricky thing. Centuries ago, when home-, office-, and workshop bots were still a fairly new thing, they were actually connected to the Internet most or all of the time. That worked about as well as anyone should have been able to predict, so after a while, people stopped doing things that way. Later, it became the norm that bots would only be indirectly connected to the outside world by means of thumb drives; if you wanted to transfer a piece of software or information or a configuration or whatever to a bot, or from a bot elsewhere, you moved it by thumb drive. That made hacking into bots more difficult, but not impossible. However, most of the time it was seen as good enough. Except that now, apparently it wasn’t good enough for [our workplace] anymore.

[zompist]

The problem with augmentation is when the makers of augmentations decide to abandon them. I recently remember reading about someone who had gotten a $100K exoskeleton suit after becoming paralyzed from the waist down in a horse-riding accident... only to have their exoskeleton suit rendered useless when a wire in the wristwatch controlling it became unsoldered and the maker of the suit refused to fix it because "we don't support devices older than five years".

This is absolutely a problem now. But you can't extrapolate problems with 10-year-old technology to 10,000 years.

People being baffled by devices they've lived with all their lives is one of my pet peeves in sf. It's like an adult person today getting into a car and wondering out loud how to turn it on and make it go left and right. That bafflement is a characteristic of bleeding-edge technology, not things that have been around for decades, much less millennia.

I'd also note that overcomplicating things is also characteristic of an immature technology. Do your eyeglasses need to be connected to the Internet? I hope not. Sticking a Turing machine into a basic tool is probably, in the long run, a bad move. You can hack a computer, it's harder to hack a mechanical clock.

[Travis B.]

Sticking a Turing machine into a basic tool is probably, in the long run, a bad move. You can hack a computer, it's harder to hack a mechanical clock.

The thing is that it is easier to make things Turing-complete than to make things not Turing-complete ─ things as simple and innocuous as Conway's Game of Life have proven to be Turing-complete.

[Ares Land]

On Dune:

Dune has an interesting take -- the basic idea is that the elite relies on natural human capabilities, expanding them via special training and genetic selection. So you train natural mathematical / strategical genius to the utmost, and you get Mentats. (It's not clear what Mentat do that your ordinary nefarious advisor does, but let that slide for now )

As I recall, in the original books, the problem with thinking machines is not that they're evil robots; it's that relying on AI make human soft or something. Not using computers builds character.

The Bene Gesserit's aims are believable -- complete knowledge of both past and future, that makes sense as the objective of far-future scientists.

I believe it's heavily implied the system is failing, hence the general stagnation and/or regression. And that's why it's about ripe for being taken over by space Chechens.

---

On Dune and space Chechens:
Maybe I mentioned that on the board already? Like most everyone, I was very impressed by the worldbuilding, especially the Fremen culture. A few years ago I read a book on Caucasus culture, Lesley Blanch's The Sabres of Paradise. Turns out a lot about Fremen culture was stolen from that book, entire bits lifted verbatim. Ah well.

---

On AIs... I'm not convinced by AIs taking over.
One huge point is that AI doesn't have agency, or motives of its own. The idea that AIs would somehow 'replace' humans strikes me as anthropomorphism.
Another is that biology seems a lot more efficient when it comes to intelligence. Generative AI is impressive and all that, but human beings don't typically need the energy output of a nuclear power plant to do the same thing!

---

On very advanced civs in SF, one model I like is the aliens in Arthur C. Clarke 2001 and 2010 (books, not movies, also at some point the sequels got really weird, as I recall) -- they're suitably unscrutable and capable of cosmic engineering feats; ultimately though their interest in us and their actions make sense.

[sasasha]

On Dune:

Dune has an interesting take -- the basic idea is that the elite relies on natural human capabilities, expanding them via special training and genetic selection. So you train natural mathematical / strategical genius to the utmost, and you get Mentats. (It's not clear what Mentat do that your ordinary nefarious advisor does, but let that slide for now )

I enjoy this bit of Dune a lot, too, and think it’s one of the better bits of worldbuilding. Also I agree with your comment on the Bene Gesserit. (Their adherence to 21,000 year-old lemmas bothers me though. I get that they’re supposed to know lots about the past, but I would have steered away from almost verbatim Latin and Hebrew terminology in building my ‘scary ancient cult of the future’. It sounds cool, I suppose, and it’s not as bad as Chakobsa...!)

As I recall, in the original books, the problem with thinking machines is not that they're evil robots; it's that relying on AI make human soft or something. Not using computers builds character.

I’d say there’s a little more weight in it than that ‒ according to Wikipedia, God Emperor of Dune (by the OG Herbert) paints the Butlerian Jihad as “a semi-religious social upheaval initiated by humans who felt repulsed by how guided and controlled they had become by machines.”

In the non-original books, there’s a much starker picture of an all-encompassing fight to the death between humans and thinking machines / a human crusade for freedom from enslavement, with a couple of twists and turns.

More: show

Herbert died in 1986, leaving his vision of the events of the Butlerian Jihad unexplored and open to speculation. The Legends of Dune prequel trilogy (2002–2004) by Brian Herbert and Kevin J. Anderson presents the Jihad as a war between humans and the sentient machines they had created, who rise up and nearly destroy humanity. The series explains that humanity had become entirely complacent and dependent upon thinking machines; recognizing this weakness, a group of ambitious, militant humans calling themselves the Titans use this widespread reliance on machine intelligence to seize control of the entire universe. Their reign lasts for a century; eventually they give too much access and power to the AI program Omnius, which usurps control from the Titans themselves. Seeing no value in human life, the thinking machines—now including armies of robot soldiers and other aggressive machines—dominate and enslave nearly all of humanity in the universe for 900 years, until a jihad is ignited. This crusade against the machines lasts for almost a century, with much loss of human life but ultimately ending in human victory.

I do think the Butlerian Jihad must have been pretty bad. What 10,000 year-old prejudices do we still uphold? (I realise that’s a complex question.) The cultural memory of it was sufficiently scary to keep humans for ten millennia away from tools we can barely keep away from for ten seconds.

space Chechens

Right, this is the main issue I have with the franchise. I think it’s interesting how close to a mid-century view of the ‘noble savage’ the Fremen are, and how linguistically and culturally they are obviously descended from that single network of cultural references. But it somewhat makes my skin crawl that they’re often just speaking 20th century Arabic and doing things Muslims do ‒ that’s too little artistic scrutiny, in my opinion, and there’s no viable in-world justification for it. This is twenty-one thousand years in the future.

It’s better than Disney’s Aladdin, but not much. Though I have to register that I hold a lot of love for both ‒ I just think they could have done a lot better on this front.

On AIs... I'm not convinced by AIs taking over.
One huge point is that AI doesn't have agency, or motives of its own. The idea that AIs would somehow 'replace' humans strikes me as anthropomorphism.
Another is that biology seems a lot more efficient when it comes to intelligence. Generative AI is impressive and all that, but human beings don't typically need the energy output of a nuclear power plant to do the same thing!

I agree... for now. What about when (ok, if) advanced computing genuinely is more energy efficient than biological life by every metric?

I don’t think they’ll take over per se ‒ I think they’ll be put in positions of power by us, because at some point it’s going to be obvious that they’ll simply do a better job. Thus they don’t need a motive. And who says they can’t develop agency? We developed it, eventually, from a starting point that was nothing more than a bunch of chemicals accidentally knocking about together in shallow water. (Or whatever.)

Edit: re Dune, frankly, if the universe was run with computer assitance (/rule) for 10,000 years ‒ which generally means, the majority of data is stored by computers ‒ then all the computers were wiped out in a cataclysmic war, then another 10,000 (plus) years passed, why is there any cultural memory of our own era at all? Like archaeologically, materially speaking, where does it come from?

I’d believe it better if it all happened two thousand years from now, which gets to Zomp’s point that this isn’t, really, a far-future scenario.

[Ares Land]

It sounds cool, I suppose, and it’s not as bad as Chakobsa...!)

I really like the idea of "battle languages" -- make your own language so the enemy can't figure out what you're saying. (As a one-off idea, it works; but it's not a very long term technique -- just capture and torture a fluent speaker and you're all set.)

I do think the Butlerian Jihad must have been pretty bad. What 10,000 year-old prejudices do we still uphold? (I realise that’s a complex question.) The cultural memory of it was sufficiently scary to keep humans for ten millennia away from tools we can barely keep away from for ten seconds.

I believe Herbert's original idea is that it's not prejudice -- computers really are a bad idea, and there's no real advantage to using one: it's implied a well-educated person would outperform AI, let alone Mentat. That goes for the first book though, obviously Herbert changed his mind; in the other books there's mention of clandestine devices, so it turns out to be prejudice after all. But the original concept made more sense!

Right, this is the main issue I have with the franchise. I think it’s interesting how close to a mid-century view of the ‘noble savage’ the Fremens are, and how linguistically and culturally they are obviously descended from that single network of cultural references. But it somewhat makes my skin crawl that they’re often just speaking 20th century Arabic and doing things Muslims do ‒ that’s too little artistic scrutiny, in my opinion, and there’s no viable in-world justification for it. This is twenty-one thousand years in the future.

Muslim, yes, Arab maybe not so much -- turns out the original inspiration was Imam Shamil's struggle against Tsarist Russia. The outright borrowing is disappointing... but maybe redeemed by the idea that Islam would turn out to be relevant in the future, impressive for the 60s.

I agree... for now. What about when (ok, if) advanced computing genuinely is more energy efficient than biological life by every metric?

Granted, but that's quite a big when -- or if. I believe that would take a huge technological revolution in the way we build computers... to the point that 'computer' would probably not be the right word. We're talking breakthroughs as big as the transistor and the Turing machine combined. Though possible, of course!

I don’t think they’ll take over per se ‒ I think they’ll be put in positions of power by us, because at some point it’s going to be obvious that they’ll simply do a better job. Thus they don’t need a motive. And who says they can’t develop agency? We developed it, eventually, from a starting point that was nothing more than a bunch of chemicals accidentally knocking about together in shallow water. (Or whatever.)

A possible scenario is creators introducing agency to the superhuman AI, either by accident or on purpose.
As for positions of power? I don't know; lots of philosophical questions here -- what is power anyway? How do we measure being good at exercizing power? Does maximizing intelligence means maximizing efficient use of power?

An interesting thought experiment... suppose we replace the market by AI-powered Cybersyn. Is our AI running human society? Or are we using it to move goods and services around? What's a good test? I have no idea as to what the answer to these questions might be!

[sasasha]

It sounds cool, I suppose, and it’s not as bad as Chakobsa...!)

I really like the idea of "battle languages" -- make your own language so the enemy can't figure out what you're saying. (As a one-off idea, it works; but it's not a very long term technique -- just capture and torture a fluent speaker and you're all set.)

Sure, it’s fun, and I like the sign language that was created for the recent films. In general I think the linguistic work that was done for the reboot was good and has improved the franchise. (Let me say that even louder in case DJP is lurking around here!)

the original concept made more sense!

Agreed!

Muslim, yes, Arab maybe not so much

I mean Arabic, the language (‘Lisan al-Gaib’, ‘Kitab al-Ibar’, ‘Shai-Hulud’ and so on).

The outright borrowing is disappointing... but maybe redeemed by the idea that Islam would turn out to be relevant in the future, impressive for the 60s.

Yes. But I’m not sure it’s so much impressive ‒ rather, lucky. The 60s ‘noble savage’ idea has been mixed in over time with increasingly deep intercultural understanding, and altered geopolitical status, of the cultures on whom the Fremen are based. In this sense, the original depiction of the Fremen has got more interesting over time... like a ripening cheese. When you combine this with the work the Villeneuve reboot did to give Fremen culture more depth (such as, but not limited to, getting a conlanger to try and improve Herbert’s own ‘conlanging’) you get... well... Fremen that are fit for the 21st century. But Herbert’s Fremen of the 60s don’t quite deserve that accolade, IMO. (They’re engaging, don’t get me wrong!! ‒ but as you point out, they’re disappointingly derivative, and as I point out, they’re unfeasibly similar to specific cultures of our own time given how long ago ‘now’ was.)

Granted, but that's quite a big when -- or if. I believe that would take a huge technological revolution in the way we build computers... to the point that 'computer' would probably not be the right word. We're talking breakthroughs as big as the transistor and the Turing machine combined. Though possible, of course!

Sure, sure. I think we’re essentially in agreement here. I’m taking ‘advanced’ to mean ... like, really long in the tooth. (Though I don’t think it is so advanced as to be outside the scope of this thread or zomp’s essay. 10,000 years is a really long time, and AI has developed fast already. The transistor and the Turing machine were invented only 11 years apart... An invention that is ‘as big as the two combined’ is surely a likely invention of the next hundred centuries.)

An interesting thought experiment... suppose we replace the market by AI-powered Cybersyn. Is our AI running human society? Or are we using it to move goods and services around? What's a good test? I have no idea as to what the answer to these questions might be!

Right, I was going to suggest a similar thought experiment. I agree, this is a symbiotic structure. Is the cat the pet, or the human? It doesn’t really matter. The point is, AI will do a lot of the hard work, while biological life forms tend to their biology. AI can be designed entirely around doing the hard work with maximum efficiency; unless we eschew our biology entirely, we can’t.

[TomHChappell]

I really like the idea of "battle languages" -- make your own language so the enemy can't figure out what you're saying. (As a one-off idea, it works; but it's not a very long term technique -- just capture and torture a fluent speaker and you're all set.)

Didn’t the US Army use battle sign-language in World War 2 and in Viet Nam?
IIANM,
It seems the average battle language is useful for about the length of a typical infantryman’s career. (20 years or so.?)
….
I’ve never served, so, I have an even better chance of being wrong this time, than I usually have.