This isn’t just a theoretical problem — it’s something that has cropped up with the autopilot feature on airplanes. As a 2014 New Yorker article pointed out, a number of crashes have occurred because pilots simply weren’t paying close enough attention as the plan largely flew itself. That caused them to make crucial mistakes when they were forced to take over control.
One study found that with higher levels of automation, “pilots’ ability to make complex cognitive decisions suffered a palpable hit. They were less able to visualize their plane’s position, to decide what navigational step should come next, and to diagnose abnormal situations.”
With the plane doing most of the driving, pilots had more trouble concentrating on the task in front of them. Their minds tended to wander That made them less well-prepared when an emergency required them to exercise good judgment and quick thinking.
Car companies are just starting to introduce partially self-driving cars onto the market, so we don’t yet know if the same kind of problem will crop up on our roads. But it provides a powerful argument for advocates of allowing fully autonomous driving. (Source)