A lot of it comes down to the perception of privilege. A lot of people believe that trans women basically have all of the privileges inherent to being an able-bodied cis man so their assumption of participation in a female centric space is to weild their privilege to unfair advantage....

However women's sport leagues fall into different reasons for existing. For some it is because of a perceived difference of physiology... In which case the issue is what original puberty the athlete went through. Technically if an athlete transitions early and never goes through male puberty at all you basically remove that issue.... But then people start being incensed that someone transitioned at 16 with the a-okay of an entire team of doctors, social workers, therapists and the blessings of the legal guardians. It is easier for them to just assume that there is no solution that allows fair inclusion on this front.

Or

The second type of league exists to combat a systematic rejection of women through treating them as novelties or inferiors. Nobody generally wants to do something for fulfillment, bragging rights or fun if it requires them to wade through being treated like a second class citizen. The exclusion of trans women from these communities assume that trans women do not experience the same forces of misogyny that drive cis women from sport and assumes trans women can just re-don all of their male privileges whenever they feel like... Something that doesn't really happen. From personal experience within the community trans women are often REALLY vulnerable to misogyny. They just freeze up and have no idea how to deal with it. They didn't get a training period where elder women assumed they were inexperienced and helped them learn how to deal or strategies how to stay safe or safe places to retreat to where they could find solidarity. A lot of them just stumble into danger with a lot of cis women just leaving them out to dry thinking they are innately strong enough to deal. The exclusion of trans women from these spaces takes a rather callous vein of every one removing all forms of solidarity from all sides of sport. The trans woman looking for a place to do something they are extremely passionate and talented in basically is told that she can either put up with being thrown into the misogyny heavy lions den and treated by all sides as novelty, a reason others can point to to discredit the rest of the needs of the trans community or just as a man (something that for a trans person causes a cascade of negative mental health effects) or she can disappear from the sport entirely...ignoring that similar situations are what caused a segregated woman's league to be established in the first place.

The first assumes male privilege of body fat and muscle distribution. The second assumes an intrinsic male social privilege remains relevant. A lot of people want to assert that a loss of male privilege isn't possible. It's just someone with it trying to angle for an unfair advantage rather than someone who just needs the solidarity of other people who lack privilege in a similar way they do.