Should I even care that 'everyone' seems thinks I'm a 'man '?

Yes! Jumping from women to women, even though not having is still whoring around. It applies for women, it applies for men too. Get used to being called a "manwh*re", man up and stay with one women.