I did this with a controller for the longest time. Specifically, the thing was not first/third person byt "do I have a visible crosshair or not", as that defined if I am directly moving the camera/head, or if the crosshair is like a laser pointer I move on the screen and the character looks towards it.
I finally had to decide one way or the other with Monster Hunter: World as the sling requires switching between the two rapidly and while you actually can set separate inverts for first and third person, it means you can't "follow" a monster smoothly while switching to the sling, you need to also quickly flick the stick to the other direction. Took me roughly 20 hours of rather chaotic gameplay for it to finally "click" in an instant.
I chose non-inverted as it was easier to imagine a crosshair than it was to ignore one that existed.