The problem is that you only roll once they leave the feywild. Up to that point time between the two planes works in sync. You effectively just time travel when leaving depending on the result of your roll.