I did actually start with Persona 4 Golden, then fell deep into a rabbithole, so I played a lot of Megaten games.

Persona 1 was released on the PS1 and then re-released on the PSP. The remake features a different soundtrack to the original, more similar to the style from 3 or 4. However, the original soundtrack is extremely good (by Hidehiko Aoki, who passed away in 2002 unfortunately) -> imo the best way to play it is the PSP version with the original soundtrack patch.

Persona 2 has both PS1 and PSP versions. It is also divided into two parts, even though unfortunately few people play the sequel. The PSP versions are upgrades, but Innocent Sin's gameplay balance has kinda been ruined and the game is very easy. Eternal Punishment's port fixes all gameplay problems and is a worthy successor.

As for other games of the series I'd recommend: Shin Megami Tensei 1 (PS1 version), SMT 2, SMT 3 Nocturne (best played after 1 and 2, since it plays with expectations from that one), Strange Journey (there's an original or remake version - either is fine, but each has their fans and haters), Devil Survivor 1.

And a few I'd recommend with a but: SMT 4 + sequel (great gameplay but mid writing), Megami Tensei 2 (great game, but it's very old and janky), Shin Megami Tensei if... (good if you really like SMT 1 or 2's gameplay - it's light on story and is primarily a dungeon crawler. It's technically a persona prequel, since the canonically female MC shows up in Persona 1 and 2 as a surprisingly relevant character)