The main difficulty with pointers, at least to me, is that I didn't start with C. I started with Java. The whole notion of pointers were really foreign until a couple of classes in college where I was expected to know C. So then I taught myself the very basics of C and how to use pointers in their very basic sense. Even then, every time I find myself reading C code, I have to look up pointer syntax.
So in my very limited experience(1 year real world + 4 in college), pointers confuse me because I've never had to really use it in anything other than a classroom setting. And I can sympathize with the students now starting out CS with JAVA instead of C or C++. As you said, you learned pointers in the 'Neolithic' age and have probably been using it ever since that. To us newer people, the notion of allocating memory and doing pointer arithmetic is really foreign because all these languages have abstracted that away.
P.S.
After reading the Spolsky essay, his description of 'JavaSchools' was nothing like what I went through in college at Cornell ('05-'09). I took the structures and functional programming (sml), operating systems (C), algorithms (pen and paper), and a whole slew of other classes that weren't taught in java. However all the intro classes and electives were all done in java because there's value in not reinventing the wheel when you are trying to do something higher leveled than implementing a hashtable with pointers.
