I think that if you post anything remotely clever, it all depends on the first vote. If the first vote is an upvote, everybody will look at it as if it might be clever, because it's upvoted. If the first vote is a downvote, everybody expects it to be stupid.

I think that's one of the reasons why Reddit had a feature that would hide votes for a while, because that first vote is so powerful.