I think there was even an episode in Enterprise? where an old Klingon lamented that modern Klingon society went too far towards being warriors that they were starting to neglect other parts of society. Hell, Ezri Dax was right that, by the time of the Dominion War, the Klingon Empire was in need of serious reform.

And if anything, the Klingons seemed like they had a larger empire for a while, well before humans even went to warp. It could just be that they went into atrophy for long enough that allowed the Federation to fill the void by Kirk's time.