The stigma is really just an oversimplification. We like to make stupid general social rules like that, which are totally meaningless without context. The only grain of truth I can see is that there should be things in your life that you care about more than video games. (Eg if you rage about video games but are indifferent to more "important" things, that is sort of a problem.) The problem there would be more the lack of strong reaction to the other area though.
That is subjective and situational as hell though, for example progamers SHOULD be highly invested in video game success relative to other things. As long as the passion is in proportion to the importance it's all good. Like I said, generic social rules are pretty meaningless.