How do I handle my Ex-husband still in my family's life?

I've been divorced for going on 4 years. He is remarried with children. I am in a relationship. I am the one that requested the divorce. My problem is that my family, (mother, grandmother, sister, and a close aunt/uncle) are still in contact with him. My aunt/uncle (whom, told me they don't ever talk to him) went to his wedding. My mom even told his step-children to call her "Grandma". My sister refuses to talk to me or my partner. And they don't treat my partner of 3 years very well. I'm at a crossroad and I've already talked to my family about how this affects me and my relationship. And I often think they would rather have him than me in the family. So I don't know if I should just pack up and leave and not look back. I know family is very important to me, but they are just pushing it too far. Do you think it's ok for my ex and my family to be in contact when it's posing a problem in my relationship?