Wikipedia going down like that makes me sad, especially since due to ai crawlers, their traffic costs went up significantly.
Why would anyone crawl Wikipedia when you can freely download the complete databases in one go, likely served on a CDN...
But sure, crawlers, go ahead and spend a week doing the same thing in a much more expensive, disruptive and error-prone way...
There are valid reasons for not wanting the whole database e.g. storage constraints, compatibility with ETL pipelines, and incorporating article updates.
What bothers me is that they -- apparently -- crawl instead of just... using the API, like:
I'm guessing they just crawl the whole web and don't bother to add a special case to turn Wikipedia URLs into their API versions.
Vibe coding
My comment was based on a podcast I listened to (Tech won’t save us, I think?). My guess is they also wanna crawl all the edits, discussion etc. which is usually not included in the complete dumps.
Good pod cast
Yes they should really block crawlers or force them to pay. The only way I can think of that they could do is make you have to register an account to access content but that goes against what they originally intended. But these are new times and it's probably for the best. Wouldn't be hard to flag obvious AI scrappers.
It seems there are ways to stop crawlers. Do a web search for "stop ai crawlers" or similar to learn more. I hope it doesn’t escalate into an arms race, but I realise I might be disappointed.
And click through rate is dropping.
Top 10 websites?
Can't reddit just die or something?
Ironically it’s getting more popular but to me it seems it’s getting more popular in the way facebook used to get more popular. At some point your weird uncle is on it and all the good content creators just leave.
Also google makes it the first result for a lot of searches now.
Reddit stopped being fun last November.
I am not engaging with it as much as I used to, and now I am actively annoyed with it and trying to make Lemmy my default again.
What happened last November? I quit when they turned off the API
In fact, I think at least in Brazil Reddit is becoming more and more popular. Go back like 5 years ago and Brazilian subs in Portuguese were small and low-traffic except for one or two (ex: /r/brasil).
Now there are a bunch of different themes and I see new topics being discussed quite often.
I believe the reason is Facebook has enshittified so much that its communities are dying quickly and Brazilians are finding Reddit works better for simple discussions. Also no one posts anything personal on FB anymore, it’s all Instagram style reposts so it lost any purpose.
By whose measurements? [citation needed]
Where's porn?
In 3, 4 & 7 IG
Didn't really support the porn industry since the beginning but I kinda miss the porn magazines and the website made for it.
It's far more dangerous for multiple reasons to view porn on mainstream socials than intended websites, for young specifically... Sadly... One more thing that tend to worsen...
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
