"Ninja" were mainly samurai doing spy shit. They might disguise themselves as peasants, but usually they were from the warrior caste acting on behalf of their lord. The whole "sneaky guys in black pajamas" was something that came from plays where people playing ninjas would dress in black to hide in the background of a scene. Then they would enter the stage and it would look like they appeared out of nowhere.