Am I being manipulated?

Met a guy at work. 3 months into dating/talking and he wanted to settle down with me. I refused because it felt too fast. We stopped talking because he saw his ex at a family gathering and would like to "try" with her again (they broke up in good terms before I even knew him, he said he doesn't love her anymore and is just being pressured by his family). After a few weeks of no contact, he realized that he wants me back in his life, so we started talking and decided to figure things out since it was all messy. After a couple of months, he suddenly wanted to stop what was going on between us because I “still didn’t want to commit.” I blocked him. He managed to contact me through work and spoke like nothing happened. Sadly, I fell for him again and now we’re back to square one, but his ex is still around and said he couldn’t just block/avoid her. He reasoned that I’m the one he loves and that his ex just couldn’t move on (I casually get threats from her saying I ruined their relationship and he just tells me to ignore her.)

I’ve been in a few relationships and the chemistry we have is unlike any other. He knows me like the palm of his hand and I’ve never been happier, but it’s all just so fucking confusing.