Fish used to be poor people's food. It was plentiful around the sea, but it kept for just a few hours without modern refrigeration, so you couldn't really transport it to the main city market and sell it. It didn't give you much food security or much money, and it wasn't as luxurious as meat, which was the food of choice for the higher classes.

The only fish that was eaten by the higher classes were the ones that could be preserved by salting, drying or smoking, and they were eaten mainly during lent, as a "lean" alternative to meat. It was mostly viewed as a sacrifice. During the late Middle Ages and early modern era, the herring trade started to really flourish, with Holland being a major exporter of herrings, while the Nordic countries like Norway and Sweden exported lots of salted cod and Stockfish (dried cod).

So I'm sure it was at a moment where eating fish was seen as a humiliation, rather than a treat, like it is today. In North America lobster was considered as very poor food, cockroaches of the ocean, fed to those who couldn't afford anything else or to prisoners. Sometimes they were even used as fertilizer for the fields.