A couple of reasons:

1. Who would contribute? Banks are highly regulated and sometimes deal with complex products that most Devs don't have a background on. Even most Devs in banks rely on a team of business analysts, designers etc to shape the requirements. Add on top of that the general negative perception of banks, I can't think of a large open-source community forming.
2. Competition. Bank's primarily compete with each other. They all offer very similar products, and any advantage they can gain by developing proprietary software will be explored.
3. Third-party apps. Banks use a TON of third-party apps behind the scenes. A lot of times they will purchase licenses for existing products and then customise on top of that.
4. Outsourcing. Even when they are building the app "in-house" they may have outsourced the development to another company, and will then just maintain the finished product.
5. Banks move slooooowly. As it's a highly regulated industry, every deployment needs to go through a ton of red-tape. An exploit found in public might take weeks to be resolved internally.
6. Reward is not worth risk. It simply isn't a priority and they can't see any benefit for doing it. It's more likely to cause a reputation risk than not.