If you’re driving for the first time, it may not be as hard as you thought, but it would probably be a little overwhelming and frightening since this is the first time you’re operating a vehicle that can injure or kill pedestrians if the momentum of the car is high enough.
If you have already had one or even half a year of experience, driving is easy for most people. Once people get the hang of driving, it kind of blends into a person’s basic skills (ex: walking, eating, drinking).
After all, practice makes perfect.