A contract just codifies an existing power dynamic, because its terms depend on the negociating powers of the people agreeing to it. It doesn't say anything about the morality of the terms or the context in which it was signed. Very extreme and on-the-nose example: "We have agreed to only allow white people, you have breached that contract ...". This works just fine if your moral system is based on contracts, but it's obvously immoral. There's also the conundrum of people never explicitly agreeing to the social contract they are born into, and even if they did, it's not like they have much of a choice.

Imo pure tolerance is a real paradox, because you cannot tolerate intolerance, and that makes you intolerant yourself. You can't achieve it, but you probably should not want to in the first place. There are certain things we will and certain things we won't tolerate in a modern society, and that is completely fine. The important thing is that we recognize this and make good decisions about which is which.