Am I insane for thinking caffeine could be causing my severe anxiety?

I’ve been a very anxious person the past couple years. I thought it was because I was getting older and entering the workforce & college grind but I’m starting to suspect caffeine may play a role. I consume 200-500mg a day via energy drinks and green tea.

My doctor says to keep under 400 but doesn’t think it could be causing severe anxiety.