Issue #1 - bots bruteforcing login forms: add a 2FA in form of a TOTP? Simple to setup / create, doesn't depend on 3rd party services and it is less extreme than a Yubikey while providing the same level of security. If you can enable that for all users you can add it straight to the login form after the password, this way bots won't even know if a password they try is correct or not, you can refuse them all with a simple "email, password or 2FA code incorrect".

Issue #2 - bots creating fake accounts: decoy email and password fields on your registration form helps reducing the number of fake accounts. Create your input for email and password with the id / name "email" and "password" and hide them with CSS. Then you create the real inputs with an id like "zipcode" or some other thing that would throw bots off. Server side you set that if the email and password inputs are submitted with anything else than an empty value it should return 401 and/or block the IP address. You can play a lot with this and add checks both client side and server side. To step up the game you can create all those fields dynamically in JS with random IDs based on some algorithm so the backend knows how to identify the real ones.

There are also a few self-hosted captcha options that can be as full featured as google's or simply add a few font awesome icons and ask people to pick the right one.

Updates:

• As many said messing with the input type, name and ID may break password managers and kill accessibility. Depending in your use-case you may or may not want to use those techniques - note they're very effective either way;
• You can also leverage 2FA to avoid fake accounts. Require users to setup 2FA when they're creating an account - bots won't be able to handle that and accounts won't get created. You can also delay the process, like allow people to register as usual and on the first login force the 2FA setup, accounts who don't set it up in, let's say, 5 days get automatically deleted;
• Use the "yahoo trick" to render bots unable to login.