You could just click on it to hide it, or if you're on mobile you can often just scroll past it to hide

I believe if u vote and then remove your vote that also does the same thing