85
submitted 6 months ago by barsoap@lemm.ee to c/technology@lemmy.world

A new paper suggests diminishing returns from larger and larger generative AI models. Dr Mike Pound discusses.

The Paper (No "Zero-Shot" Without Exponential Data): https://arxiv.org/abs/2404.04125

you are viewing a single comment's thread
view the rest of the comments
[-] lvxferre@mander.xyz 4 points 6 months ago

Not even another info transferring entity would solve it. Be it quantum computers, photonic computers, at the end of the day we'd be simply brute forcing the problem harder, due to increased processing power. But we need something else than brute force due to the diminishing returns.

Just to give you an idea. A human needs around 2400kcal/day to survive, or 100kcal/h = 116W. Only 20% of that is taken by the brain, so ~23W. (I bet that most of that is used for motor control, not reasoning.) We clearly suck as computing machines, and yet our output is considerably better than the junk yielded by LLMs and diffusion models, even if you use a really nice computer and let the model take its time producing its [babble | six fingers "art"]. Those models are clearly doing lots of unnecessary operations, while failing hard at what they're expected to do.

Regarding research, my point is that what's going to fix generative models is likely from outside the field of artificial intelligence. It'll be likely something small and barely related, that happens to have some ML application.

[-] CheesyFox@lemmy.sdf.org 1 points 6 months ago

there's a lot to optimize in LLMs and i never said otherwise. Though, photonic computers if the field would be researched, could consume as much as an LED lamp making it even more effective than our brain. given the total amount of computers in the world, even the slightest power consumption optimization would save colossal amount of energy, and in case of photonics the raw numbers could possibly be unimagineable.

Regarding research...

I bet they simply will find a way to greatly simplify the mathematical apparatus of the neuron interaction. Matrix multiplication is kinda slow and there's lots of it

this post was submitted on 09 May 2024
85 points (85.7% liked)

Technology

59648 readers
1479 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS