I am but one man whose only education in programming was a first year university course in C from almost two decades ago (and thus I am liable to completely botch any explanation of CS concepts and/or may just have faulty memories), but I can offer my own opinion.

Most basic programming concepts I was taught had easily understood use cases and produced observable effects. There were a lot of analogous concepts to algebra, and functions like printf did things that were concrete and could be immediately evaluated visually.

Pointers, on the other hand, felt designed purely of and for programming. Instead of directly defining a variable by some real-world concept I was already familiar with, it was a variable defined by a property of another variable, and it took some thinking to even comprehend what that meant. Even reading the Wikipedia page today I'm not sure if I completely understand.

Pointers also didn't appear to have an immediate use case. We had been primarily concerned with using the value of a variable to perform basic tasks, but none of those tasks ever required the location of a variable to complete the calculations. We were never offered any functions that used pointers for anything, either before or after, so including them felt like busywork.

It also didn't help that my professor basically refused to offer any explanation beyond a basic definition. We were just told to arbitrarily include pointers in our work even though they didn't seem to contribute to anything, and I really resented that fact. We were assured that we would eventually understand if we continued to take programming courses, but that wasn't much comfort to first year students who just wanted to pass the introductory class they were already in.

And if what you said is true, that later courses are built on the assumption that one understands the function and usefulness of pointers despite the poor explanations, then its no wonder so many people bounce off of computer science at such a low level.