The main reason for me is consistency. I listen to white noise for sleeping, and when I used a phone for that it would often get interrupted by stuff like the phone randomly restarting for whatever reason phones sometimes do that, or there was a software update that changed something so the volume button on the earbuds activated voice assistant which made a noise that wasn't disabled by do-not-disturb mode, or all the other various quirks and exceptions to how do-not-disturb mode works, or the UI or functionality of whatever audio app I was using suddenly being different in a way I didn't like because apps can just update themselves like that. Transferring files to a phone over usb can be a little involved and sometimes bug out. Then there's that it shares a battery with the phone, which may have gotten low for other reasons, and the trend with phones of not having a headphone jack or sd card slot.

An mp3 player only does one thing, has no network connectivity, does not update its software, and is designed around non-wireless file transfer. that makes it better than a phone for audio from my perspective. You can turn off the phone and use the mp3 player and all of your phone's bullshit goes away.

Size. I use one too when I don't want to carry my large phone.