What I do is have some general mailboxes then signed addresses on top of that.

So if you email blog@ or kevincox@ you will get a fairly high level of spam filtering. I also have a few other "memorable" addresses that get reduced spam filtering. If you use the unique signed address that I use for signing up to services, newsletters or whatever where the address is private to a specific service then you basically skip spam filtering. Of course if you abuse that privilege then I will outright block the signed address.

Basically by allowing friends and "trusted" services through the spam filter I can crank up the difficulty for unknown senders.