When I do wear out in public, I don't worry about it. If someone notices, they notice, if they don't they don't. No one has ever said anything to me before, and I have had someone walk into my stall mid change once (I forgot the latch) and lets just say, she was trying to get back out of there to really say anything.
People are people, I am me, who cares if they know :-p