Abuse, and neglect, anon, which is also abuse. The fact that you see him very little does not change the fact that when you do interact, he is abusive towards you. If his being ‘rough’ with you is physical, that’s physical abuse on top of emotional abuse. You do not deserve to be treated this way.