Three things: Scale, recency and contrition or perceived lack thereof.

The British Empire is the largest empire there has ever been. At its greatest extent, in 1920, it covered about 1/4 of the entire world, long after having lost many holdings like the US. The second largest, the Mongol Empire, reached almost the same size, but hundreds of years earlier.

In the same time period as the British, the Russian empire covered <20% in 1895, but its proportion of colonial lands to their own was much smaller than for the British Empire and the proportion of the current world population living in those areas is also much smaller. The French colonial empire covered less than 1/10th of the world at its peak in 1920, and was by far the other largest recent holding of colonies geographically and culturally outside of the immediate sphere of the holding country.

Spain is rarely brought up, I think, in large part because the Spanish empire reached its peak in the early 1800's and so is "history". Belgium doesn't get discussed at much because 98% of their colonial holdings was Leopold II's personal ownership of the Congo Free State. And then we get to the last bit: Contritition.

Nobody goes around saying the massive scale of gross abuse that happened under Leopold II's rule of the Congo Free State was a good thing. Few people I've met ever defend France's atrocities in Vietnam. Even the defence of their ownership of Algeria, which was special enough to trigger an attempted coup against Charles de Gaulle when he wanted to let it have independence because many saw it as part of France itself, is relatively muted.

But there's still mainstream support for the British Empire in the UK. There are still people who insist the British Empire was awesome for the colonies that were exploited because they got English and rails and British legal systems and that somehow outweighs the mass murder and brutal exploitation and erasure of local cultures.

E.g. this survey from 2019, where 32% were proud of the British Empire, 37% were neutral, and only 19% considered it "more something to be ashamed of". 32% were proud of their country's history of colonialism and oppression. Critically this was significantly higher than for other colonial powers other than the Dutch. At the same time 33% thought it left the colonies better off vs. only 17% who thought they were worse off.

I'm not British, but I've lived in the UK for 23 years, and I've experienced this attitude firsthand from even relatively young British people (ok, so all of them have been Tories) - a refusal to accept that the fact that a substantial number of these former colonies had to take up arms to get rid of British rule might perhaps be a little bit of a hint that the colonial rule was resented and wrong.

No other modern empire has left behind such a substantial proportion of the world population living in countries that have either a historical identity tied up to rebelling against British rule, and/or have relatively recently rebelled against British rule, and/or still have substantial reminders, such as Commonwealth membership or the British monarch as their monarch. When a proportion of the British population then keeps insisting this was great, actually, there you have a big part of it.