Why does sex without lube hurt?

As the question states I'm a 23 year old woman. My boyfriend is 31 and every time we've tried to have sex without lube it's hurt. He says it's because of our size difference which it could be but at the same time is there something wrong with me? I'm wet when we go at it yet it feels like it doesn't matter. When we use lube it doesn't hurt at all which makes me feel like the issue lies with me.