Except if this was how it worked, then elves would be babies for like 10 years before being toddlers for like 30 and teenagers for 50-70 years.

A decade as a toddler? Nah.

I think it's fair to say elves reach maturity almost as quickly, just age slower after reaching maturity. Same with dwarves, etc.