372
submitted 1 month ago by cypherpunks@lemmy.ml to c/technology@lemmy.ml
you are viewing a single comment's thread
view the rest of the comments
[-] Sl00k@programming.dev -2 points 1 month ago

Olympic Arena analysis OpenAI analyses

Compare the GPT increase from their V2 GPT4o model to their reasoning o1 preview model. The jumps from last years GPT 3.5 -> GPT 4 were also quite large. Secondly if you want to take OpenAI's own research into account that's in the second image.

[-] TootSweet@lemmy.world 6 points 1 month ago

if you want to take OpenAI’s own research into account

No thank you.

OlympicArena validation set (text-only)

"Our extensive evaluations reveal that even advanced models like GPT-4o only achieve a 39.97% overall accuracy (28.67% for mathematics and 29.71% for physics)"

  • The OlympicArena analysis that you cited.
[-] Sl00k@programming.dev -2 points 1 month ago

The jump from GPT-4o -> o1 (preview not full release) was a 20% cumulative knowledge jump. If that's not an improvement in accuracy I'm not sure what is.

[-] Aceticon@lemmy.world 3 points 1 month ago* (last edited 1 month ago)

One of the first things they teach you in Experimental Physics is that you can't derive a curve from just 2 data points.

You can just as easilly fit an exponential growth curve to 2 points like that one 20% above the other, as you can a a sinusoidal curve, a linear one, an inverse square curve (that actually grows to a peak and then eventually goes down again) and any of the many curves were growth has ever diminishing returns and can't go beyond a certain point (literally "with a limit")

I think the point that many are making is that LLM growth in precision is the latter kind of curve: growing but ever slower and tending to a limit which is much less than 100%. It might even be like more like the inverse square one (in that it might actually go down) if the output of LLM models ends up poluting the training sets of the models, which is a real risk.

You showing that there was some growth between two versions of GPT (so, 2 data points, a before and an after) doesn't disprove this hypotesis. I doesn't prove it either: as I said, 2 data points aren't enough to derive a curve.

If you do look at the past growth of precision for LLMs, whilst improvement is still happening, the rate of improvement has been going down, which does support the idea that there is a limit to how good they can get.

[-] Sl00k@programming.dev 1 points 1 month ago

which does support the idea that there is a limit to how good they can get.

I absolutely agree, im not necessarily one to say LLMs will become this incredible general intelligence level AIs. I'm really just disagreeing with people's negative sentiment about them becoming worse / scams is not true at the moment.

I doesn't prove it either: as I said, 2 data points aren't enough to derive a curve

Yeah only reason I didn't include more is because it's a pain in the ass pulling together multiple research papers / results over the span of GPT 2, 3, 3.5, 4, 01 etc.

this post was submitted on 16 Sep 2024
372 points (98.2% liked)

Technology

34806 readers
216 users here now

This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.


Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.


Rules:

1: All Lemmy rules apply

2: Do not post low effort posts

3: NEVER post naziped*gore stuff

4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.

5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)

6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist

7: crypto related posts, unless essential, are disallowed

founded 5 years ago
MODERATORS