The average computer user knows how to use Windows exactly. When there's no "Start" button they get confused and decide that it's not worth the trouble to learn how to operate Linux when they already know how to operate Windows, and so they don't research it very much at all. As a result, they don't know anything about Linux and it feels foreign to them and is therefore "hard".
If Linux had gotten as much publicity and stuff as Microsuck did, back when computers were just emerging, it would be entirely conceivable that they switch places from what they are today; with Windows being considered "hard" and Linux the industry standard that everyone uses.
Just my theory; I've never actually talked about Linux with someone who thought that it was "hard".
Everyone I have convinced to try Linux has liked it, and while most still use Windows in one way or another they usually prefer Linux, most common reasons being that "it doesn't make me feel like an idiot", "it's more straightforward", and "it's a lot less buggy and viruses are no problem".
Plus, it's free (well, most of it, and the commercial distro's aren't much better at all than the free ones, as far as I can tell)