Hat/Cap/Whatever.

I went to buy one a few months ago, tried it on and was told to get it off, then I saw Rumi wearing one here and its made me want to find one again and as its already pushing all boundries of what I would usually wear I would go for the plainest one in one of my usual colours (black or any other understated dark colour, brown? khaki?) and hope to pull it of?