NSW Treasurer Matt Kean says people spreading gossip on Macquarie Street about the Nazi costume Dominic Perrottet wore to his 21st birthday party need to "come out from the swamp".