I can understand getting anxious about that, but I don't really see how it tracks.
I guess I can understand in a sort of "don't freak out if your kid has some meat when eating at a friend's place" but it doesn't really sound like that's what they're saying. It sounds like they're saying you have a moral failing for being vegan and having a vegan household, and that's just not true.

I just appreciated that your coworker is catholic. That means they definitely gave a catholic guilt button. A lot of antivegan attitude, dressed up as social concern like this or otherwise, is just an expression of people's own guilt about the contradiction between what they know about what they eat and their self image as a Good Person. If you shift the terrain of the discussion away from peer pressure etc and onto more solid ground (I want to raise my child with strong morals, veganism is a strong part of my morals, cruelty to animals is clearly immoral) then you're either going to reach this person or get them to shut the fuck up. This criticism of you is clearly a reflection of their own insecurity, so the more you can turn this around on them the better.

but it would be difficult to navigate accusations like that.

Go for broke, start telling their kids whenever they eat meat an animal is killed for that

read them vegancirclejerk every night and your child will get bullying superpowers. If this person is raising their kids Catholic aren't they also making meatless meals on Friday?

There's no moral argument to raise kids with omni Western pattern diet instead of any other diet, it just has to be healthy and tasty. You can't practically prevent them from eating animals outside the home, but you can make it clear to them why they shouldn't and that you'll be disappointed if they do, like how you'll tell them not to do drugs or steal cars.