All technology can be harmful in the wrong hands, but the raw power of cutting-edge AI may make it one of a few “dual-class” technologies, like nuclear power or biochemistry, which have enough destructive potential that even their peaceful use needs to be controlled and monitored.
The peak of AI concerns is superintelligence, the “Godlike AI” referred to by Musk. Just short of that is “artificial general intelligence” (AGI), a system that can learn and evolve autonomously, generating new knowledge as it goes. An AGI system that could apply its own intellect to improving itself could lead to a “flywheel”, where the capability of the system improves faster and faster, rapidly reaching heights unimaginable to humanity – or it could begin making decisions or recommending courses of action that deviate from human moral values.
Timelines for reaching this point range from imminent to decades away, but understanding how AI systems achieve their results is difficult. This means AGI could be reached quicker than expected.
The End of the World is Nigh!
"How nigh?"
"Oh, maybe before Christmas. We don't really have much of an idea."
Kim