Anytime I ask it about things I'm knowledgeable about, it falls on it's face. It only seems smart on topics I don't know shit about (this red flag has spotlights on it!). Given this, I cannot trust it on anything lest I be led down false paths. Given how LLMs work, I don't see this problem as fixable.

I have found them useful for search term generation. (Using a topic I know things about) If I need a more powerful drill for screwing lag bolts into hardwood and I'm not finding what I need, asking it and and getting back the keyword 'impact driver' is helpful, as it lets me go search for what that is and if it is the solution to my need for a more powerful drill. Note: I do not let it teach me about impact drivers (as it falls on it's face all the fucking time), only use it to get the keyword to then use to search the internet.

To circle back to your question, I'm not scared with how fast they are advancing. I'm scared by how many people think they are good at everything and put them in places they don't belong.