It also doesn't account for the common situation where a country could do something more cheaply internally if they had a decade or two to spool shit up.

See China where it's gone from a comparative advantage in coal and not much else, to cheap manufacturing, to high tech manufacturing, to universal comparative advantage.

they do better/cheaper than anyone else, and trade for the rest.

This is absolute advantage. Comparative advantage is when a country can make something relatively cheaper then the other goods. So if a country A can make a coat for 10 tögrög and a bicycle for 20 tögrög, while a country B can make both for 30 tögrög, it means that the country B has comparative advantage in making bicycles and it would be more profitable for both if country B makes bicycles and trades for coats with country A.

What if country A doesn't produce that much at all and instead just prints dollars, then uses it's military to force everyone to trade oil and everything else in dollars enjoying a steady flow of goods and services in exchange for cheap paper? And what if country A then suddenly reverses course (because other countries start to move away from the dollar anyway) and tries to cash in on this flow, while it's still going, with high tariffs?

Yeah you're right because in that case it's not really a difference in the marginal costs that motivates the spreading out of the supply chain (on the contrary, having to transport the goods around adds a lot of overhead) but the colonialist need to keep each place from being self-sufficient.

However, you shouldn't throw out the baby with the bath water. Marxist economists still consider comparative advantage because means of production aren't equal in each country. From Chapter 10 of Towards a New Socialism (from 1993 so this comparison is outdated):

The German clothing industry can probably produce garments with less labor than the Chinese clothes industry. The German car industry can certainly produce cars with less labor than would be needed if they were made by backstreet workshops in Shanghai. In both cases Germany has a productivity advantage over China, but it is nonetheless economical to export Mercedes to China and to import cotton goods to Germany. This arises because of the greater relative productivity of the German car industry. Compared with hand assembly, the highly automated car factories of Mercedes may give, say, a five-fold improvement in productivity; with the rag trade the scope for improvements in productivity is not so dramatic. Although a German clothing firm might be more efficient, the advantage would not be so great as for the car industry. It thus pays Germany to concentrate its labor force in those engineering industries where it has the greatest advantage.

Of course, Cockshott goes on to develop a theory of how this structure actually works when the ratio of demand for the various commodities inevitably doesn't match the productive capacity of both economies. The point is, though, that even if we did away with exploitation we'd still have a situation where it makes more sense to do certain kinds of production in one economy due to unequal means of production that lead to comparative advantage.

also obligatory Cockshott is a TERF disclaimer