The point of using the TPM is that it does not unlock the drive unless it has a certain set of software is loaded in a certain sequence on the machine with that specific TPM chip.

So if somebody breaks grub and makes it load a shell, then that results in different software loaded (or at least loaded in a different sequence) and will prevent the TPM to unlock the system. The same is true if somebody boots from a rescue disk (different software loaded) or when you try to unlock the disk in an unexpected phase of the boot process (same software but different sequence of things loaded, e.g. after boot up to send the key to some server on thr network. The key is locked to one TPM, so removing the drive and booting it in a different machine also does not work.

The TPM-locked disk is pretty secure, even more so than that USB idea of yours -- if the system you boot into is secure. It basically stops any attacker from bringing extra tools to help them in their attack. All they have available is what your system has installed. Do not use auto-login or run some root shell in some console somewhere...