Should He Be Around My Daughter?

My boyfriend and I have dated for three years. He is a good provider and a loving father to my daughter and there is no doubt in my mind that he loves me, and I love him with all my heart. But my biggest problem with him is that when we have a disagreement he reminds me of all the things that he has done for me and my child.
As long as I put up with his mental abuse, everything is OK, but to me abuse is abuse and no one should put up with it no matter what the situation is. I work, and I took care of my child and my household long before I met him, so it's not like I am home doing nothing. Recently I got tired of it and called the relationship off, but now he wants to continue to be a part of my daughter’s life, which is OK with me, but I have reservations. I need your advice.