We know the performance of L2-70b to be on par with L3-8b, just to put the difference in perspective. Surely they models continue to improve and we can only hope the same improvements will be found in L4, but I think the point is that models have improved dramatically since this study was run and they have put in way more attention in the fine-tuning and alignment phase of training, specifically for these kinds of tasks. Not saying this means the models would beat the human summaries everytime (very likely not), but at the very least the disparity between them wouldn't be nearly as large. Ultimately, human summaries will always be "ground truth", so it's hard to see how models will beat humans, but they can get close.

Can you confirm that you think with L3, the result would look completely opposite and the summaries of the AI would always beat the human summaries? Because it sounds like you are implying that.

Lemmy users try not to make a strawman argument (impossible challenge)

No, that's not what I said, and not even close to what I was implying. If Llama 2 scored a 47% then 3.1 would score significantly better, easily over 60% at least. No doubt humans can be better at summarizing but A) It needs someone that's very familiar with the work and has great English skills and B) It needs a lot of time and effort.

The claim was never that AI can summarize better than people, it was that it can do it in 10 seconds and the result would be "good enough". People are already doing AI summaries of longer articles without much complaints.