Your model seems to involve an AI going to war against humans. And in that context, you talk about how humans are unpredictable, like dogs in a dog park, or ants. So you seem to accept that humans may relate to a more advanced AI intelligence in a way similar to the way ants and dogs and less intelligent biological beings relate to humans.
We don't fight wars against ants or dogs. If ants get in our way, we deploy insecticides, and the ants are no longer in our way. If dogs are dangerous, we kill them, and if we would prefer they not reproduce, we surgically remove their ability to do so. Their ability to do things we wouldn't predict because they aren't smart enough to pick the best course of action given their goals, doesn't give them a sufficient advantage over us to mean we would lose a war against them, if one was fought. Similarly, if there's a group of expert card players and I join them knowing very little but the most basic rules of the game they are playing, they will be able to infer much more about what cards each other has based on what cards each chooses to play, than they will be able to infer about my hand from my unskilled play. I will often lay a card that confuses them because a more skilled player would have done differently, but they will likely consistently win regardless.
We already have gain of function research, and examples of natural viruses equally as contagious as COVID, while being much more deadly. Part of what makes COVID contagious is that you're contagious before you're symptomatic, so we have proof of concept that viruses can spread without symptoms, and then cause illness after they have spread. A not even particularly smart AI could figure out that deploying a genetically engineered biological agent could kill most of the humans, and the rest of them wouldn't be able to respond because society and its coordination mechanisms would have collapsed.
I don't understand why you think there would be a war. A smart agent wouldn't fight, they would just do things that accomplish their goals, and if the humans got in the way or had resources that were needed, too bad for them. War is a highly inefficient and risky way of accomplishing a goal. Much better to just neutralize your opponent before they have a chance to respond, or accomplish your goals by cooperation (say, by running a business, it gaining control of a lot of resources, then you have control of resources and can deploy them as you please, and it turns out "as you please" and "as humans please" are not at all the same). These are all fairly humdrum ways of taking control of the world, I'm sure a smarter-than-human AI could think of more esoteric possibilities, so that fact that we could in principle plan for and counter the possibilities I've suggested so far (not to say we could in reality coordinate well enough to do that, but in principle it's possible) isn't a real barrier to AI takeover.
But supposing there was a war, just for the sake of the argument: decision making and information gathering can easily be decentralized. There doesn't have to be one controlling mind processing all the information, a program or model (maybe conscious, maybe not) that can digitally copy itself can coordinate with its copies, maybe not perfectly but much more effectively than humans with non-identical brains and histories can coordinate with each other. Our advantage over other animals, when we first gained one, was ability to communicate, coordinate, and pass knowledge between us. and over time AIs would be better than we are at that (there is already the ability to merge model weights between models and successfully have them learn how to perform better that way instead of training separately). So if there was a war between humans and smarter-than-human AIs, I'd give the advantage to the AIs.
The point of this post would be to study what would mean "war" in this context because it is one argument of the discussion.
Now.regarding the example of ants it is an example used to discuss the argument that AI would be so much intelligent that it could manipulate humanity to do what it is looking for (extermination of humans in this thought experiment setting). My point is to say that even if we are super intelligent vs ants we cannot make them behave as we want to. We cannot manipulate ants so that they will kill themselves on our behalf , which is an argument of doomers.
Manipulation of viruses is not an exact science. The same remark as the economic calculation problem applies. You have to test, fail, repeat. But it means it is not something that can be done in silico. That begs the question how the AI will have it's own lab ?
It's not that I misunderstood, but I don't hear those who think there's a big extinction risk suggesting that war is how it will happen - it doesn't seem to me to be an argument that is a serious part of the discussion, so I wanted to challenge the premise. For the reasons I've mentioned, a smarter entity with goals different from or opposed to us wouldn't need to fight us, any more than we need to fight nonhuman animals for control. I did try to listen to the debate with Roko to see if your starting point of "what would happen in case of war" was because that's what you two discussed, but the debate wouldn't play for me.
Anyway, to address your point about manipulating ants: It's true that if you, say, kick an anthill, you're not going to then be able to psychologically manipulate the ants into not stinging you - by that point you've caused a bad outcome. But one way we deal with ants is to leave out baits they see as food which are actually toxic, which they bring back to their nest, and feed to the queen and offspring, and then the nest dies. You don't need to individually remote control every ant once you've started a conflict, in order to "manipulate" the colony into a state where it's dead. And similarly, a smart AI wouldn't need to individually mind-control every human through a carefully crafted twitter feed after revealing its adversarial intentions or something like that. We would start off unaware that we're dealing with an adversary, likely something that some humans have sunk a lot of money into and are getting a lot of benefit out of, and there are plenty of humans who a) would cooperate with a powerful AI because it's powerful, b) would cooperate with a powerful AI because they think intelligence is what matters and the AI has more claim on the future than puny and fallible human intellect, c) would cooperate with AI because they're one of the hundreds of millions of people living on less than $2 a day and the AI is willing to pay them enough to secure their next few meals, d) would cooperate with an AI because they were lied to or tricked, e) would cooperate with an AI that they know is out to destroy humanity because they've had a bad life and this is like a school shooting but on a much larger scale. Those are just off the top of my head, I could come up with more. The point being, if a smart agent wants a human to do a thing that leads to the destruction of humanity, provided one or a small number of humans selected from a large group of candidates can do the thing, it's going to be trivial for an AI to find a human to do what needs doing.
As for how an AI will have its own lab, that's easy too. Setting aside for the moment that most of our scientific research around things like "how will this DNA strand lead to this protein folding which has this effect" is done by AIs or in simulation, let's assume a lab is actually needed and see how an AI could get access to one. I think you're thinking of AI as "trapped in a computer, can't do anything in the real world", but that's wrong. Computers control lots of processes in the real world, including most aspects of most factories and the banking system. Given full control of a standard laptop (where "full control" means I have admin rights and can rewrite the OS and firmware and directly make the hardware do anything it is physically capable of doing - getting to there is fairly easy on most machines, particularly if you're smart enough to look at the monthly OS patches, see what files changed, and then look in those files to see where the vulnerabilities are) and sufficient time (not an issue for an intelligence that thinks several orders of magnitude faster than I do, such that one month of clock-time is like several years of time as I experience it now), I can hack into pretty much any computer that is connected to the Internet, which includes many computers that control physical systems. I say this as someone with 15 years of cybersecurity experience. Not to say I could easily just hack anything that exists now, I have a day job and people patch things. But if I had let's say several years of subjective time to take my hacking skills from where they are now to where I could get in a few years, and then several more years to find a vulnerability in a system I wanted to hack (usually I try for a few hours on practice machines)... yeah, I could likely hack almost any system connected to the Internet. So that gives me lots of paths to power. And that's only at my current level of intelligence and creativity but with lots of time.
But let's back up a step - if I had full control of a laptop and plenty of time on my hands, I wouldn't need to hack, except to the extent of getting a bank account under an assumed identity set up. Once I had a place to build up money and pay people from, I could use various platforms and means to make money in the regular ways that a remote worker can, then use stock trading (already often the best strategy there involves carefully crafted computer-executed algorithms) to grow my cash-pile. Once I had several million dollars, getting a lab set up or funding grants to do research I want done is easily achievable. Or, given a small amount of money, I could just buy ransomware as a service (easily available to anyone on the Internet who's willing to risk going to jail) and use "encrypt a large corporation's data and then extort several million dollars out of them or their insurance company" as the approach I take to getting several million dollars, and thus the power to have lots of people do lots of things for me and access to lots of physical resources.
These are not the only options I can come up with, just the ones that came to mind first. I've got more if you want.
An AI getting the ability to run experiments in the real world is... not a barrier.
What do you mean ? In what measure the AI would contribute (and how) to this ? It is possible that humanity goes extinct. For instance birth rate is crashing all across the board. But it has not a lot to do with AI.
Your model seems to involve an AI going to war against humans. And in that context, you talk about how humans are unpredictable, like dogs in a dog park, or ants. So you seem to accept that humans may relate to a more advanced AI intelligence in a way similar to the way ants and dogs and less intelligent biological beings relate to humans.
We don't fight wars against ants or dogs. If ants get in our way, we deploy insecticides, and the ants are no longer in our way. If dogs are dangerous, we kill them, and if we would prefer they not reproduce, we surgically remove their ability to do so. Their ability to do things we wouldn't predict because they aren't smart enough to pick the best course of action given their goals, doesn't give them a sufficient advantage over us to mean we would lose a war against them, if one was fought. Similarly, if there's a group of expert card players and I join them knowing very little but the most basic rules of the game they are playing, they will be able to infer much more about what cards each other has based on what cards each chooses to play, than they will be able to infer about my hand from my unskilled play. I will often lay a card that confuses them because a more skilled player would have done differently, but they will likely consistently win regardless.
We already have gain of function research, and examples of natural viruses equally as contagious as COVID, while being much more deadly. Part of what makes COVID contagious is that you're contagious before you're symptomatic, so we have proof of concept that viruses can spread without symptoms, and then cause illness after they have spread. A not even particularly smart AI could figure out that deploying a genetically engineered biological agent could kill most of the humans, and the rest of them wouldn't be able to respond because society and its coordination mechanisms would have collapsed.
I don't understand why you think there would be a war. A smart agent wouldn't fight, they would just do things that accomplish their goals, and if the humans got in the way or had resources that were needed, too bad for them. War is a highly inefficient and risky way of accomplishing a goal. Much better to just neutralize your opponent before they have a chance to respond, or accomplish your goals by cooperation (say, by running a business, it gaining control of a lot of resources, then you have control of resources and can deploy them as you please, and it turns out "as you please" and "as humans please" are not at all the same). These are all fairly humdrum ways of taking control of the world, I'm sure a smarter-than-human AI could think of more esoteric possibilities, so that fact that we could in principle plan for and counter the possibilities I've suggested so far (not to say we could in reality coordinate well enough to do that, but in principle it's possible) isn't a real barrier to AI takeover.
But supposing there was a war, just for the sake of the argument: decision making and information gathering can easily be decentralized. There doesn't have to be one controlling mind processing all the information, a program or model (maybe conscious, maybe not) that can digitally copy itself can coordinate with its copies, maybe not perfectly but much more effectively than humans with non-identical brains and histories can coordinate with each other. Our advantage over other animals, when we first gained one, was ability to communicate, coordinate, and pass knowledge between us. and over time AIs would be better than we are at that (there is already the ability to merge model weights between models and successfully have them learn how to perform better that way instead of training separately). So if there was a war between humans and smarter-than-human AIs, I'd give the advantage to the AIs.
I think you misunderstood the argument.
The point of this post would be to study what would mean "war" in this context because it is one argument of the discussion.
Now.regarding the example of ants it is an example used to discuss the argument that AI would be so much intelligent that it could manipulate humanity to do what it is looking for (extermination of humans in this thought experiment setting). My point is to say that even if we are super intelligent vs ants we cannot make them behave as we want to. We cannot manipulate ants so that they will kill themselves on our behalf , which is an argument of doomers.
Manipulation of viruses is not an exact science. The same remark as the economic calculation problem applies. You have to test, fail, repeat. But it means it is not something that can be done in silico. That begs the question how the AI will have it's own lab ?
It's not that I misunderstood, but I don't hear those who think there's a big extinction risk suggesting that war is how it will happen - it doesn't seem to me to be an argument that is a serious part of the discussion, so I wanted to challenge the premise. For the reasons I've mentioned, a smarter entity with goals different from or opposed to us wouldn't need to fight us, any more than we need to fight nonhuman animals for control. I did try to listen to the debate with Roko to see if your starting point of "what would happen in case of war" was because that's what you two discussed, but the debate wouldn't play for me.
Anyway, to address your point about manipulating ants: It's true that if you, say, kick an anthill, you're not going to then be able to psychologically manipulate the ants into not stinging you - by that point you've caused a bad outcome. But one way we deal with ants is to leave out baits they see as food which are actually toxic, which they bring back to their nest, and feed to the queen and offspring, and then the nest dies. You don't need to individually remote control every ant once you've started a conflict, in order to "manipulate" the colony into a state where it's dead. And similarly, a smart AI wouldn't need to individually mind-control every human through a carefully crafted twitter feed after revealing its adversarial intentions or something like that. We would start off unaware that we're dealing with an adversary, likely something that some humans have sunk a lot of money into and are getting a lot of benefit out of, and there are plenty of humans who a) would cooperate with a powerful AI because it's powerful, b) would cooperate with a powerful AI because they think intelligence is what matters and the AI has more claim on the future than puny and fallible human intellect, c) would cooperate with AI because they're one of the hundreds of millions of people living on less than $2 a day and the AI is willing to pay them enough to secure their next few meals, d) would cooperate with an AI because they were lied to or tricked, e) would cooperate with an AI that they know is out to destroy humanity because they've had a bad life and this is like a school shooting but on a much larger scale. Those are just off the top of my head, I could come up with more. The point being, if a smart agent wants a human to do a thing that leads to the destruction of humanity, provided one or a small number of humans selected from a large group of candidates can do the thing, it's going to be trivial for an AI to find a human to do what needs doing.
As for how an AI will have its own lab, that's easy too. Setting aside for the moment that most of our scientific research around things like "how will this DNA strand lead to this protein folding which has this effect" is done by AIs or in simulation, let's assume a lab is actually needed and see how an AI could get access to one. I think you're thinking of AI as "trapped in a computer, can't do anything in the real world", but that's wrong. Computers control lots of processes in the real world, including most aspects of most factories and the banking system. Given full control of a standard laptop (where "full control" means I have admin rights and can rewrite the OS and firmware and directly make the hardware do anything it is physically capable of doing - getting to there is fairly easy on most machines, particularly if you're smart enough to look at the monthly OS patches, see what files changed, and then look in those files to see where the vulnerabilities are) and sufficient time (not an issue for an intelligence that thinks several orders of magnitude faster than I do, such that one month of clock-time is like several years of time as I experience it now), I can hack into pretty much any computer that is connected to the Internet, which includes many computers that control physical systems. I say this as someone with 15 years of cybersecurity experience. Not to say I could easily just hack anything that exists now, I have a day job and people patch things. But if I had let's say several years of subjective time to take my hacking skills from where they are now to where I could get in a few years, and then several more years to find a vulnerability in a system I wanted to hack (usually I try for a few hours on practice machines)... yeah, I could likely hack almost any system connected to the Internet. So that gives me lots of paths to power. And that's only at my current level of intelligence and creativity but with lots of time.
But let's back up a step - if I had full control of a laptop and plenty of time on my hands, I wouldn't need to hack, except to the extent of getting a bank account under an assumed identity set up. Once I had a place to build up money and pay people from, I could use various platforms and means to make money in the regular ways that a remote worker can, then use stock trading (already often the best strategy there involves carefully crafted computer-executed algorithms) to grow my cash-pile. Once I had several million dollars, getting a lab set up or funding grants to do research I want done is easily achievable. Or, given a small amount of money, I could just buy ransomware as a service (easily available to anyone on the Internet who's willing to risk going to jail) and use "encrypt a large corporation's data and then extort several million dollars out of them or their insurance company" as the approach I take to getting several million dollars, and thus the power to have lots of people do lots of things for me and access to lots of physical resources.
These are not the only options I can come up with, just the ones that came to mind first. I've got more if you want.
An AI getting the ability to run experiments in the real world is... not a barrier.
But you seem to have not considered the simple risk of human disempowerment leading to effective extinction.
What do you mean ? In what measure the AI would contribute (and how) to this ? It is possible that humanity goes extinct. For instance birth rate is crashing all across the board. But it has not a lot to do with AI.