Is it wrong I like my alone time?

I live with a roommate, and people are always dropping by unexpectedly and either hanging out all day with the tv blasting or spending the night. I feel like it invades my privacy a bit especially when I want to do things like use the tv for exercising or occasionally lounging around the house in my pjs. I spoke to my roommate about this and she called me selfish.. am I being selfish?