I think that is one of the great things about having one that is visible most of the time. Someone else sees that dive flag and knows what it means, you find people that dive that you never would have known were divers otherwise.
As far as "getting in trouble"...I say go for it..lol. It isnt the worst thing you could do. Just dont get it anywhere above the neck!