Quick take
The Unfortunate Utility of "Can"
Anxiously rereading some things I've written recently, I've found the word "can" to be especially loaded1. Given that humans are horrible at risk assessment and probability in general, I think that having endowed one word with a plethora of probability-laden functions is a rather unfortunate circumstance to find ourselves in.
Consider the following:
"AI can do my homework."
"AI can prove the Riemann Hypothesis."
"AI can take over the world."
"AI can accurately simulate a universe."
There is a whole lot of ambiguity we've imbued into these statements with the word can. AI can do my homework right now, completely and totally. This is a factual statement about the reality I observe myself to be in.
"AI can prove the Riemann Hypothesis" is a bit more subtle. Can it do it right now? The new OpenAI models certainly make a case for it. I don't think there are many people left who think this is totally impossible, even in the near future. "Can" is (informally) implying a "will," a prediction about a imminent future reality we might (or probabilistically, should expect to) find ourselves in.
This probabilistic assignment becomes a bit more dangerous when we move into the next statement: "AI can take over the world." This is hypothetically possible, and depending on who you ask, even probable. However, we're moving away from an indication about the current or expected state of reality. We are explicitly making a proposition about a thing that may potentially occur.
Depending on the relative probability people assign to that statement (or anticipate their counterparty assigning), it takes2 on an entirely different conversational role. If I'm talking to an AI safety person, "AI can take over the world" is an observation that contextualizes a further discussion. If I'm talking to my neighbor, I expect they assign a really low (or even zero) probability to this, and so I'm asserting a claim I need to provide evidence for. "Can" indicates that something warrants discussion; in the past, we might have had "sabertooth tigers can eat us, therefore we should make weapons," or "the winter can wipe out our food supply, so we store things by....."
The next "can" is even worse: stating that one's combination of logic consequence and existing knowledge do not falsify the possibility of the existence of a state of reality. For example "AI can simulate a universe." The added ambiguity is the cherry on top: we have indications that our universe cannot properly simulate our universe, but what does "a universe" mean3? An atom? Something on the scale of our galaxy? Depending on how you define "a universe," I would assign vastly different probabilities to those statements. This additional ability of "can" to accumulate further ambiguity at a fundamental level is truly unfortunate.