Calling a woman "ma'am" — yay or nay?

My wife says when talking to a woman you don't know in passing, e.g. a cashier at the grocery store, it's low-key offensive to call her "ma'am" unless the woman is clearly elderly. She says it will make her feel old. I always thought "ma'am" was the proper/polite thing. My wife's from Minnesota and I'm from southern Ohio so it may be a cultural difference. Thoughts?