Okay, but summer is warm and fun and the days literally last longer. Generally people want the nicer weather to stay around, especially when you're a kid and don't have to be in school.

Yeah and winter has snowflakes which are fleeting and delicate and the days are ever so short making them even more fleeting. Spring is usually the season associated with youth. Autumn is harvest time, the time to feel your oats, the time for kids to go out and steal apples and other hijinks. There's justifications like that for every season to have that phrase.
You can look on google trends and see how usage of the phrase corresponds exactly with release of either a GoT book or show. When you sit down and actually think about how that phrase has come to mean what it does, it is asinine to insist it's from anywhere else, when you likewise look at how it shows up in literature and just regular communication. It's from game of thrones. And I'm allowed to be annoyed at a phrase from a piece of fiction I don't enjoy having entered popular parlance AND people insisting it's not from there being annoying to me too. That isn't some horrendous faux pas, it's not some massive fault of mine. I find it annoying, it is grating to me. You trying to explain how it isn't from Game of Thrones, despite the fact it is, doesn't make it less grating.