I know nothing about growing hemp, but it sounds like what a stoner would say

This. Y'all should checkout Saintdrew's discussion on crops

Hemp and also linen are even harder to grow than cotton, though much of it is due to not as advanced machinery for harvesting and processing. Hemp also sucks as a material for clothing, to make it wearable you have to treat it quite heavily or it's scratchy AF.

Taking production out of the equation linen is the best material of the three: Much better moisture regulation than cotton, only real downside is that it crinkles easily but it also crinkles elegantly so wear it with pride and you'll be fine.

Production-wise the best alternative right now is modal, that is, basically, synthesised cotton, raw material is anything that contains cellulose. Nasty chemicals are involved but in modern processes it's all closed-loop, the nasty stuff all stays within the factory.

Oh, one often overlooked factor: Seams. Modal is better than cotton at being yarn because the cellulose fibres are much longer but nothing compares to the likes of polyester when it comes to not coming apart. I don't think there's an alternative yet, either you use polyester and make the whole garment non-biodegradable or you use modal and live with the reduced durability. Though one idea would be to aggressively get rid of seams, you can knit yarn into any shape whatsoever. Wait: Silica thread is a thing. Usually only used for extreme applications (think firefighter gear), also uses some chemicals to make it usable in sewing machines and it just won't ever hold a knot so when it comes apart it comes apart completely, but it's essentially fancy stone, just like computer chips: Doesn't really biodegrade but it doesn't matter that it doesn't, either.

Another overlooked factor is stretch. There's no natural alternative to elasthan, so no yoga pants or stretch jeans. Tons of stuff nowadays contains elasthan, often just a bit for a tiny bit of stretch simply because it's more comfortable.