I don't like my mom's boyfriend and am uncomfortable visiting her now. What should I do?

My mom remarried years ago and was with my stepfather for over 10 years. My parents split up when I was 10 and a couple years later my mom married my stepfather. For a lot of this time my mom was in a different state so I didn't get to see her or my stepfather very much. During the time that she was with my stepfather she was really sick and didn't really act like herself. she recently moved back to home to be close to her kids and divorced my stepfather. I liked my stepfather so have been having a difficult time with that. She was single for a bit and started acting normal again. It was so nice to be able to spend time with the mom I knew when I was a child. Anyway recently she has been dating this guy, who I do not like. He is rude to her and talks over her. I recently found out that he is an alcoholic and used to be homeless before my mom found him on the Internet. He practically lives with her now and has left her once saying it was for another woman, he claims to love her but I know he is just using her. Anyway I don't like him at all, he is always around when I visit my mom and now I don't want to go visit her anymore because of it. I want to see her but not him and do to know how to tell her and feel bad not visiting. Any advice would help. Thanks. Btw, she does not know that I know these bad things about him.