738
submitted 1 year ago by BrikoX@vlemmy.net to c/worldnews@lemmy.ml
you are viewing a single comment's thread
view the rest of the comments
[-] RGB3x3@lemmy.world 10 points 1 year ago

So how can a user pretend to be a web crawler?

[-] theMightyMoonWorm@lemmy.ml 21 points 1 year ago* (last edited 1 year ago)

This browser addon can spoof useragents:https://add0n.com/useragent-switcher.html

[-] SketchySeaBeast@lemmy.ca 19 points 1 year ago

You're going to need a special hat.

[-] dangrousperson@vlemmy.net 7 points 1 year ago

Ever heard of https://12ft.io/ ? It allows you to bypass alot of pay walls by basically pretending to be a search engine trying to index a website. For SEO reasons a lot of pay walled sites allow search engines to access the whole article to index. 12ft.io leverages this to show you whole articles behind paywalls. This is something you could also achieve by spoofing the User-Agent. It would probably work for things like Pinterest without an account as well, but that's something I have never tried (since I have no interest in the cancer that is Pinterest).

this post was submitted on 05 Jul 2023
738 points (98.3% liked)

World News

32285 readers
814 users here now

News from around the world!

Rules:

founded 5 years ago
MODERATORS