I should clarify that the bot account option is only for legitimate bots that serve a community-oriented purpose. Bots that do things like auto-archive websites or suggest alternative links that might improve your experience when viewing YouTube videos for example. The bot tagging system is intended solely for developers of legitimate bots, and to make the user experience with legitimate bots as smooth and user-controllable as possible, it is not intended to prevent undesirable bots because as you pointed out, they can easily just not click that checkbox and claim that they're normal accounts. Bots have to operate with a reasonable standard to be allowed, so even if a spam or troll bot labels themselves as such, they intrinsically break the rules so they won't be allowed. It's up to mods and admins to find and remove undesirable bots and it is not an easy task, especially for sites that are pseudoanonymous and intentionally collect minimal information. We could probably stick a reCaptcha on the signup form and solve almost all the bot problems, but that is extremely abusive to the privacy of legitimate users so most instances probably don't want to go that route. Maybe an open source picture captcha that is self hosted, but I'm also not sure how effective those are.

I'm not aware of any instance that permits automated signups, the expectation for a legitimate bot is that a human will create the account and then connect the bot up to it. As Lemmy grows we may eventually need to add more rules to legitimate bots, like they must have an automatic way for users to opt out, provide a way to contact the developer or maintainer should problems with the bot arise, etc, but most instances right now only require that you use the bot tag and have your bot act in a reasonable and actually helpful way. This isn't an explicit rule that I've seen on any instances, but I think there is also an unspoken rule that bots should not vote on any content, not even things they have interacted with and replied to, since the voting system is intended for people, a future update to the API should probably disable voting for bot accounts entirely.