Podcast controls trigger when they shouldn't

Sometime over the last few months, my Apple Watch changed its behavior while listening to podcasts on the Apple podcast app. I play the podcast through my phone, and the controls are mirrored on my watch. The problem is that when I'm walking or driving, the podcast controls activate and the episode or podcast changes. In all these cases, my wrist is down and the watch face is either pointed downward or to the side. It's on my left hand, and I put my hand at about the 7 o'clock position on the steering wheel. It's become extremely annoying, and makes it so I can't even wear my watch while driving.

Has anyone seen this or knows how to fix it?