As someone who switched from a Walmart bike to a carbon fiber bike I can definitely say there's a huge difference. Everything on the carbon fiber one is much higher quality. The gear ratios is has make sense, whereas I could never find the right ratios on the cheap one. Switching speeds is much faster too as I can be pedaling hard while changing gears and I know it's going to do exactly what I tell it. Finding parts is very easy compared to the Walmart one. Anytime something broke on the old one it was difficult or impossible to find a replacement. With a much nicer bike you can get parts for it easily so I can keep it in top shape for a long time. There are many other benefits as well.

So carbon frames are somewhat softer and provide more enjoyable ride. However carbon is not as resistant to damage as is metal. If it fails, chances are it will fail completely while with aluminium frame you can easily fix a dent and keep moving or not even address the damage at all. My bike for example has best of both worlds, aluminium frame and carbon forks. That way I get some smoothing during ride but most of the bike is aluminium, so a middle ground.

How does it stack up? It's like switching from 50cc scooter to 650cc sport bike. It simply can't compare. Carbon has very little to do with that really, but 50$ bike from a store would assume poorer transmission and general quality which means higher weight. Lower weight goes faster, thinner tires go faster, good transmission means you go faster. In general both bikes will get you from point A to point B, but being able to travel faster than 30km/h on your own strength is a feeling to be experienced. It feels very fast and satisfying.