As already mentioned, poetry (python-poetry) is the best thing I know regarding package management for projects. It's quite easy to setup and use, and I use it together with pyenv.

If your problem lies on your locally installed packages, then I think you're basically on your own manually searching for packages you don't use. There's not really a way for pip/python to know which packages are relevant or not (at least not that I know of), and if your problem is just about a single environment, then just delete the environment and start anew.