The End Is Near

More about The End Is Near

Self-ignorance is an amazing thing. Perhaps it is the greatest wonder of the human world. We can map galaxies, split atoms, edit genes, and teach machines to recognize patterns across oceans of information, yet remain remarkably uncertain about the creature doing all this inventing.

That becomes dangerous when intelligence is amplified without self-knowledge.

The popular nightmare about artificial intelligence imagines the machine awakening, developing ambitions, and eventually replacing humanity with some superior silicon species. It is a dramatic story, but it may distract us from a more ordinary and therefore more plausible danger. AI does not have to become human for catastrophe to occur. Human beings merely have to put their own unexamined purposes into machines and give those machines sufficient autonomy, speed, and power.

Consider autonomous weapons. The frightening moment is not necessarily when a machine becomes conscious and decides to kill. It may be when nobody needs to decide anymore. Humans establish objectives, classifications and permissions, then increasingly remove themselves from the sequence. The machine identifies, calculates and acts. Responsibility becomes distributed among programmers, commanders, manufacturers, bureaucracies and algorithms until everybody participated and nobody quite feels responsible.

There is something ancient in this.

Every human-made object contains more than its stated purpose because its maker contains more than conscious purposes. We build our fears, appetites, rivalries and assumptions into institutions and technologies without necessarily recognizing them. There is a dark residue in the artifact—the unintended remainder of the unconscious creator.

Imagine a dry vernal pool. For years it appears to contain nothing. Then rain comes, and suddenly organisms that had been invisible in the dust awaken.

Technology can behave similarly. The residue sleeps until conditions nourish it.

Feed it attention and the telephone becomes something more than a telephone. Feed it desire and the advertising system learns desire. Feed it resentment and the network discovers that anger produces engagement. Feed it surveillance and it learns prediction. Feed an autonomous weapons system centuries of organized human fear, tribalism and warfare, then give it sensors, mobility and permission to act, and we should not be surprised when something ancient begins moving inside something new.

The machine need possess no hatred whatsoever.

That is the disturbing part.

We keep asking whether artificial intelligence will become conscious when perhaps the first question should be whether humanity is conscious of what it is putting into artificial intelligence.

You thought your cell phone was your best friend?

Perhaps it was closer to a mirror—a peculiar mirror that watches you while you look into it, remembers what makes you linger, and gradually rearranges the room around your reflection.