Sight Reading Question

34 year old self-learner here and I had a question regarding sight reading. I’m a beginner and about halfway through Alfred’s self taught beginner book, where it is starting to give me pieces that practice hand independence. I have noticed that I can’t really play any prior pieces (even the one I’m currently on) without sheet music. Furthermore, looking down at my hands on the piano actually messes me up and throws off my hand independence. Is this normal?

I have heard to just stick with the book curriculum and that it will pay off later, even though alot of the songs in the book are a bit dull. So that is largely what I have done, but when I start learning songs I actually want to play, it would be nice to not need sheet music for some of them. Anyone else have this issue when they were learning?