It's really up to you. The non-sacred rares are pretty much useless except to salvage or sell, so if you're after money or materials then it's still worth picking them up. But if you're specifically looking for gear upgrades then there's not much point taking non-sacred items.

I'm at the point in Torment that I no longer bother picking up non-ancestral item drops. Same with gems - I have so many that there's no point picking up yet another bunch of flawless gems.