I think part of the problem is that there is a lot of anime in the models and when you don't filter that out with negative prompts it can distort the proportions of realistic images (e.g. everybody gets huge breasts unless you negative prompt it away). In general models are always heavily biased towards what they were trained on, and when you use a prompt or LORA that worked well on one model on another, you can get weird results. There is always a lot of nudging involved with keywords and weights to get the images to were you want it.