Awkward scenario

Hi all, I(M37) am in a relationship my girlfriend(F37) of 3 years. Both of us have been married in the past. I have had 2 kids with my ex(F38). My first kid passed away, so every year on his birthday my ex(F38) and her family get together to celebrate the birthday. Last year my girlfriend(F37) attended. She said she felt weird about it all last year and doesn't think she can go again. And feels like she is sharing me. This year I mention about it she(F37) told me that she can't be with me if I do this again with my ex(F38) and her family. I love my girlfriend(F37) a lot, so I told her I won't attend, now she feels I am holding back from what I(M37) want to do. Am I wrong for not attending so we can stay together? And is it awkward that still was doing this with my ex(F38) and her family?