321
rulebots.txt (lemmy.world)
you are viewing a single comment's thread
view the rest of the comments
[-] morgunkorn@discuss.tchncs.de 43 points 11 months ago

That's the catch, Google made a deal with Reddit and remains the only search engine allowed to access its data for indexing. It cuts off every other search engine

[-] Vorticity@lemmy.world 27 points 11 months ago

Tell me that there is an anti trust suit over this.

[-] GroupNebula563@lemmy.world 26 points 11 months ago

There's a suit over google in general so this may well be part of it

[-] TriflingToad@lemmy.world 3 points 10 months ago

really? ddg will show me reddit links, did they have to make a webscraper or something

[-] morgunkorn@discuss.tchncs.de 4 points 10 months ago

There's a cutoff date, anything indexed before the robots.txt was changed stays in the index

this post was submitted on 21 Aug 2024
321 points (100.0% liked)

196

18073 readers
898 users here now

Be sure to follow the rule before you head out.


Rule: You must post before you leave.



Other rules

Behavior rules:

Posting rules:

NSFW: NSFW content is permitted but it must be tagged and have content warnings. Anything that doesn't adhere to this will be removed. Content warnings should be added like: [penis], [explicit description of sex]. Non-sexualized breasts of any gender are not considered inappropriate and therefore do not need to be blurred/tagged.

If you have any questions, feel free to contact us on our matrix channel or email.

Other 196's:

founded 2 years ago
MODERATORS