451
I use Zip Bombs to Protect my Server
(idiallo.com)
This is a most excellent place for technology news and articles.
Anyone who writes a spider that's going to inspect all the content out there is already going to have to have dealt with this, along with about a bazillion other kinds of oddball or bad data.
If you have billions of targets to scan, there's generally no need to handle each and every edge case. Just ignoring what you can't understand easily and jumping on to the next target is an absolutely viable strategy. You will never be able to process everything anyway.
Of course, it changes a bit if some of these targets actually make your bot crash. If it happens to often, you will want to harden your bot against it. Then again, if it just happens every now and then, it's still much easier to just restart and continue with the next target.