Intimidating roommate!?

Hi I'm 22 and living in a house with one roommate. My roommate is like my big brother. When I was younger I had an unstable family and started living at his house and basically became part of his family. I used to share a room with his little brother. I very quicky fell into the role of his little brother as well and he would bully his little brother and me sometimes. Hes a really agressive guy a just likes to ruff house alot. It used to be fine because he sort of dispersed his big brother aggression on both of us, so nobody really got it that bad. But now the little brother is at prep school and we've both moved out of the house and got a place together. Now that hes not around its like I'm the only one who gets bullied. Hes alot bigger than me and its really easy for him to pin me down and mess with me. And it used to be ok cause we were so young. But now I'm an adult and its really humiliating to get bullied so much. On top of that I'm really small. And just recently his little brother got huge and now when he comes to visit they both just team up on me.Now if hes in town and their together I allways have to make sure I dont fall asleep before them or I end up getting taped up or something. It makes me mad cause the little brother was just going through the same thing. But now that hes too big to get bullied hes just turning around and bullying me with his brother. What should I do?