They can talk about you behind your back, though I doubt they’re usually aware they’re doing it. Most people don’t know the gory details of federation and don’t pay much attention to which instances are defederated from which.