Sure. Then you run the election and if anybody else in the area wants to run in the same party as the person who left then the party can put them on the ballot. If not, then no candidate for that party. Don't drop somebody in from elsewhere. Pretty simple. Just pick a time period that a person needs to have lived in a riding (say, one full election cycle) before running to represent the area.

If the person wins the election and moves (other than to government housing in Ottawa so they can do their job) then they give up their seat.

To add on to this, unless we enact political slavery, which while it might sound fun is not likely to make politicians who are subject to it vote in the best interests of the country, then politicians are going to be able to leave their positions at any time anyways, so a by-election is the only reasonable way to ensure the constituents of their riding are properly represented. And if you put some kind of significant delay before people can run in a by-election keep in mind this one is already going to be about 6 months since the last one, and that becomes less of an impediment the richer you are, meaning it gives an advantage to wealthy parachute politicians with no real benefit to citizens.